You cannot debug an LLM application without logging prompts, and you cannot keep those prompts if they are full of names, emails, and account numbers. Users type personally identifiable information straight into the box, so the log that helps you understand a bad response is also a growing store of regulated data sitting in CloudWatch or S3 with the wrong retention and the wrong access controls. The fix is not to stop logging. It is to redact before storage and to put a retention policy on what remains, so the debugging value survives and the liability does not.

The reframing: a prompt log is a data-collection surface, not a debug convenience. The moment you write a raw prompt to disk you have collected whatever the user put in it, under whatever regulation covers that data. Treat the log pipeline as a place where PII gets removed on the way in, not a place you clean up later after an audit asks.

Redact on the way in, not on the way out

The only reliable place to remove PII is before it is written. Redacting after storage means the raw data already existed at rest, already replicated, already in backups, and "we delete it later" is not a control an auditor accepts. Put the redaction step between the request and the log sink, so the value that lands in storage was never sensitive in the first place.

Amazon Comprehend gives you the detection primitive. DetectPiiEntities inspects text in real time and returns each PII entity it finds, its type, its character offsets, and a confidence score. You use those offsets to replace the spans before you log. Comprehend supports two masking modes: replace each entity with its type, so "Jane Doe" becomes [NAME], which keeps the log readable, or mask the characters with a fixed symbol. Replacing with the type is usually right, because it preserves the shape of the prompt for debugging while removing the identity.

entities = comprehend.detect_pii_entities(Text=prompt, LanguageCode="en")
redacted = prompt
for e in sorted(entities["Entities"], key=lambda x: x["BeginOffset"], reverse=True):
    redacted = redacted[:e["BeginOffset"]] + f"[{e['Type']}]" + redacted[e["EndOffset"]:]
log.write(redacted)   # only the redacted form is ever persisted

Splice from the end backward so earlier offsets stay valid as you rewrite the string. Detection is a real-time call; for large historical batches Comprehend also offers asynchronous redaction jobs, but the live path is what protects new writes.

Detection is probabilistic, so design for misses

Comprehend is a machine learning detector, which means it has a recall below 100 percent and a confidence score you have to set a floor on. A low threshold redacts aggressively and can mangle legitimate text; a high one lets edge-case PII through. Two habits make this safe:

  • Layer a deterministic pass. For structured identifiers with fixed formats, credit card numbers, national IDs, some account formats, a regex catches what a probabilistic model may miss, and catches it the same way every time. Run both.
  • Redact toward false positives. In a debug log, over-redacting costs you a little readability. Under-redacting costs you a data-protection incident. Bias the threshold toward removing too much.

Amazon Bedrock Guardrails can also filter sensitive information at the model boundary, which is complementary: Guardrails protects the request and response path, Comprehend protects what you write to your own logs. Use the one that sits where your risk is.

Retention is the other half of the control

Redaction reduces what a log contains; retention limits how long even the redacted form lives. A redacted prompt is lower risk, not zero risk, and a debug log has a natural useful life measured in weeks, not years. Set retention explicitly at the sink:

  • Expire on a schedule. A CloudWatch log group retention setting or an S3 lifecycle rule that deletes after a defined window means old logs age out without anyone remembering to clean them.
  • Match retention to purpose. Operational debugging rarely needs more than 30 to 90 days. If a longer window is required for a specific reason, name the reason and scope that retention to the logs that need it.
  • Lock down access. Even redacted logs deserve scoped IAM. The set of people who can read raw prompt logs should be small and named.

The takeaway

Prompt logging and PII protection are not in conflict, they are two steps of the same pipeline. Redact before the write with Comprehend, layering a deterministic pass for fixed-format identifiers and biasing the threshold toward over-redaction. Then put a retention policy on what remains so the redacted logs expire on a schedule instead of accumulating forever behind loose access. Do both and you keep the debugging signal you need while the regulated data never lands at rest. The log that helps you fix the app should not be the one that shows up in a breach report.

Read this next

For the platform side of log pipelines, retention, and access control, the cloud field notes live at ercan.cloud, and the hub is at ercanermis.com.