PII redaction for LLM apps: keep customer data out of your traces

Short answer: PII redaction for LLM apps means masking personal data in your traces before they are stored. In LangWatch a data privacy policy does it at ingestion, replacing each email, phone number or card number with a typed marker, and it can also drop or restrict content per team, so you debug every conversation without keeping your customers' data.

Your users type things into chat boxes that they would never put in a form. Their phone number, their address, a card number "just to check", a full medical history. Every one of those messages ends up in your traces, because a trace records the whole prompt and the whole reply.

Why traces leak customer data

Traces leak personal data because they store the full text of every turn, tool call and retrieved document, and an LLM app's inputs are free text. You can't validate a chat message the way you validate a form field.

The leaks come from more places than the user's message:

1. The user's message. People paste whatever they have, and they don't read your privacy notice first.
2. Tool results. An agent that looks up an order gets back the customer's name, email and delivery address, and that result goes into the trace.
3. Retrieved documents. A RAG app over support tickets or contracts puts chunks full of names into the context, and the context is in the trace.
4. The system prompt. Teams put the customer's profile in the prompt to personalize the reply, so every call carries it.
5. The model's reply. The model repeats the details back, sometimes with a typo that makes them harder to catch.

Then the trace gets copied. It goes into a dataset for an experiment, into an annotation queue for a reviewer, into an export for a bug report. Each copy is one more place your security team has to answer for.

How PII redaction works in LangWatch

In LangWatch one data privacy policy sets what gets stored and who can read it, at the organization, department, team or project level. It runs when a trace arrives, before anything is written, on every way traces come in.

That covers the SDKs and OpenTelemetry, the REST collector, the LangWatch AI Gateway and coding agent logs. You set it once in Settings > Data Privacy and every project under that scope follows it. The data privacy docs cover every control.

PII redaction replaces each personal value with a marker that says what it was, such as [EMAIL_ADDRESS] or [PHONE_NUMBER]. It has four levels:

Level	What it masks
Off	Nothing
Essential (the default)	Emails, phone numbers, card numbers, IP addresses and national ids
Strict	Essential plus names, locations and more, with deeper detection that can add some latency
Custom	Exactly the entities you tick

Card numbers and Brazilian CPF numbers are checked against their checksum before they are masked, so a random 16-digit order id stays readable. If one of your own id formats keeps getting caught, add it as an exception.

Secrets redaction is on by default too. API keys, tokens, private keys and database URLs become [SECRET], which matters for agents with shell or file tools, since they often print config files and environment variables into the trace.

Content controls decide whether text is stored at all, and who reads it. For each of input, output, system instructions and tool calls, you pick one of three:

• Captured stores it and shows it to everyone on the project.
• Restricted stores it and shows it only to the people you pick, such as admins or a custom group. Everyone else sees a placeholder that says who can read it.
• Dropped removes it at ingestion, for good.

A dropped span still keeps its tokens, cost, model, latency and status. So you can drop every input on a sensitive project and still see what each conversation cost and where it was slow.

Stop PII before it reaches the model

Redaction in traces protects what you store. To keep personal data away from the model provider itself, block it on the way in with a guardrail in the LangWatch AI Gateway or in your code.

A gateway guardrail runs an evaluator on the request before it goes to the provider. Put a PII detection evaluator on the request direction, and a prompt with a card number in it never leaves your network and costs you no tokens. You can run the same check on the response, or on each streamed chunk.

If you don't route through the gateway, call the PII evaluator as a guardrail from your code and decide there whether to block the call or clean the text first.

Redaction and evaluation also work together. When a value was already masked, the PII evaluator still counts the marker as a detection. So your PII monitor keeps showing how often users share personal data, even though none of it is stored.

What to redact and what to keep for debugging

Mask the values and keep the rest of the conversation. You almost never need to see the actual phone number to fix a bug, and you almost always need to see that the user gave one and what the agent did next.

• Start on Essential everywhere. Emails, phones, cards and ids are the values that do real harm when they leak, and they are detected as the data arrives.
• Turn on Strict where names matter. Health, legal, HR and finance agents talk about people by name. Strict catches names and locations, and it can add some latency at ingestion, so use it on those projects instead of the whole organization.
• Restrict instead of drop when you can. A small on-call group can still read the full conversation when a customer escalates. Drop the content only when your policy says it may not be stored at all.
• Keep the markers. A conversation that reads "my number is [PHONE_NUMBER]" still tells you the agent was asked to call someone back, and that's usually what you need to reproduce the bug.
• Add exceptions for your own ids. An order number that looks like a phone number makes traces hard to read. Exceptions are regular expressions, and LangWatch refuses one that would also let real emails or card numbers through.

One limit to know: the policy covers traces. A row you already copied into a dataset or an export is a separate copy, so clean the data before you share it.

How well PII detection works

Detection quality varies a lot, and a regex is not enough. We ran 58 PII detectors on six tests and published the results in our PII detection benchmark.

• A plain regex left almost half the personal data in on our policy test. Formats like emails are easy, and names in free text are where it falls apart.
• Open models you can host yourself did far better, and the best of them fit on a single GPU. So detection doesn't have to mean sending your data to another API.
• Results change a lot by kind of text. Traces, business documents, code and logs, and other languages all rank the detectors differently, so check the test that looks like your traffic.

Regulated industries: where the data lives

For banks, insurers and health companies, where the data sits matters as much as what is masked. LangWatch runs in four ways, and in two of them your trace data never leaves your own infrastructure.

Setup	Who runs what	Where trace data lives
LangWatch Cloud	We run everything	Our cloud
Cloud Enterprise	We run it on dedicated single-tenant instances	The region you choose
Hybrid	We run the app, you run ClickHouse and S3	Your VPC
Self-hosted	You run the full stack on AWS, Azure, GCP or bare metal	Your infrastructure, air-gapped if you need it

The self-hosting docs cover each setup. LangWatch is open source, and the self-hosted edition runs the same software as our cloud.

Add single sign-on and a retention period per team, and you can answer the first things your security team will ask: what is masked, who can read the rest, and where it all lives.