All Posts
AI Engineering

OpenTelemetry Traces Are Not an Audit Trail for Your Agent

Anthropic's October OpenTelemetry export is best-effort and sampled. If an agent acts on customer data, your own database has to hold the audit record.

9 min read

Anthropic shipped OpenTelemetry export for the Messages API and Managed Agents on 14 October, and it is good observability. It is not an audit trail, and the documentation says why: delivery is best effort, a span can arrive twice, a caller can switch tracing off per request, and prompt content is opt-in and truncated at a size limit. If an agent can move money, send email or change records in your product, the record of what it did has to be written by your own code, in the same database transaction as the action. An audit trail is an append-only record, written in the same transaction as the action it describes, that lets you reconstruct what an actor did and what the system believed when it did it. A trace tells you how a request went. An audit record is evidence.

Why this matters now

The Claude Platform release notes for 14 October describe the new export as a way to trace Messages API requests and Managed Agents sessions in your observability backend, with SDK spans in seven languages. In the same fortnight Anthropic added Compliance API betas that cover chats and saved instructions in its own apps. That is a different thing from what your product's agent did to your customers' data, and it is easy to hear "tracing" and "compliance" in one release stream and conclude the vendor has the record covered. It does not, and it was never positioned to.

There is a regulatory backdrop too, with a softer deadline than it had in the spring. The EU's AI Omnibus, in force since 27 July, moves the stand-alone high-risk obligations from 2 August 2026 to 2 December 2027. When they apply, Article 12 requires high-risk systems to technically allow automatic recording of events over their lifetime. Whether your agent is high-risk is a legal question and I am not a lawyer. The engineering point is that the cheapest moment to design an event record is before the agent has acted on a year of customer data.

What the vendor's own documentation says the export is

The OpenTelemetry page and the API destination page are candid. Four statements matter for anyone tempted to treat spans as a ledger:

  • Delivery is best effort and at least once. Exported spans are not guaranteed to be a complete record of your API usage, a span can be lost, and a span can arrive more than once, so you deduplicate by trace ID and span ID.
  • The caller controls sampling. A request can leave out its spans by turning off the sampled flag in its traceparent header, and the API follows it.
  • Content is opt-in. The default mode is metadata only; prompts and responses appear only if you choose Full content, and spans are truncated at 0.75 MB of content by default, flagged with anthropic.content.truncated.
  • Retention belongs to someone else. Your observability platform stores the content under its own access and retention settings, and a destination is disabled after 24 hours of failed deliveries.

None of that is a flaw. It is what good telemetry looks like: cheap, sampled, lossy and tolerant of a slow backend. An audit record has the opposite requirements. It must be complete, it must not be switchable by the caller, it must hold what you decide it should hold, and it must outlive your observability vendor contract. Those are properties of a table you own, not of a pipe you subscribe to.

The record I would write instead

The shape is one append-only table of agent events, written inside the same transaction as the side effect it describes. Each row carries the tenant, the run, the actor, the action, a hash of the input, the version of the policy or prompt that was in force, and the trace and span IDs so you can join to the vendor's telemetry when you want the detail. Two timestamps matter. The first is when the system recorded the event. The second is when the event was effective in the real world. They differ whenever you correct something later, and keeping both is what makes the table bitemporal: you can answer both "what did we know on the 3rd" and "what was true on the 3rd". A correction is a new row that points at the one it supersedes. Nothing is updated, and the application role gets INSERT and SELECT only.

// agent-events.ts: record the action and its audit row in one transaction (sketch, not run against production load)
import { Pool, PoolClient } from "pg"
import { createHash } from "node:crypto"
import { trace } from "@opentelemetry/api"

const pool = new Pool()

type Act = {
  tenantId: string
  runId: string
  actor: string          // "agent:invoice-triage@v7" or "user:42"
  action: string         // "invoice.mark_paid"
  input: unknown
  policyVersion: string  // prompt + tool-policy version in force
  effectiveAt?: Date     // valid time; defaults to now
  supersedes?: string    // id of the event this one corrects
}

export async function actAndRecord<T>(a: Act, effect: (tx: PoolClient) => Promise<T>): Promise<T> {
  const ctx = trace.getActiveSpan()?.spanContext()
  const tx = await pool.connect()
  try {
    await tx.query("BEGIN")
    const result = await effect(tx) // the real change, in the same transaction
    await tx.query(
      `INSERT INTO agent_events
         (tenant_id, run_id, actor, action, input_sha256, policy_version,
          trace_id, span_id, effective_at, supersedes_id)
       VALUES ($1,$2,$3,$4,$5,$6,$7,$8,COALESCE($9, now()),$10)`,
      [a.tenantId, a.runId, a.actor, a.action,
       createHash("sha256").update(JSON.stringify(a.input)).digest("hex"),
       a.policyVersion, ctx?.traceId ?? null, ctx?.spanId ?? null,
       a.effectiveAt ?? null, a.supersedes ?? null],
    ) // recorded_at is a column default of now(): system time, never supplied by the caller
    await tx.query("COMMIT")
    return result
  } catch (e) {
    await tx.query("ROLLBACK")
    throw e
  } finally {
    tx.release()
  }
}

Two details keep this honest. The row stores a hash of the input, not the input, so the table stays small and you decide separately where the full prompt lives and for how long. And actions that leave your database, like sending an email, cannot share a transaction with it. For those I write an intent row inside the transaction, perform the external call after commit, and append an outcome row when it returns. A crashed worker then leaves a visible intent with no outcome, which is exactly the thing an audit should surface.

What a founder or CTO does with this

The practical test is a question a customer's security team will eventually ask: show me everything the AI did to our data last quarter, and prove nothing is missing. With vendor traces alone you cannot answer the second half, because the vendor has told you in writing that spans can be dropped and that callers can opt out. With an owned event table the answer is a query. The cost is modest: one table, one wrapper, one grant, and a retention decision per field. The cost of not having it is discovered at the worst time, during a dispute or a questionnaire, when the only available answer is a sampled trace from a system you no longer pay for.

A four-step pattern

  1. Classify every tool the agent can call as read, reversible write or irreversible write. Only the last two need audit rows on day one.
  2. Write the event in the same transaction as the change, with intent and outcome rows for anything external.
  3. Store the trace and span IDs as join keys, and treat the vendor's telemetry as the debugging layer on top of your record, never the record itself.
  4. Reconcile monthly. Compare your event counts and token totals against the vendor's Usage API, which the documentation notes also counts requests that have no spans, and investigate any gap.

My perspective

I would not let a vendor's telemetry be the system of record for anything an agent does to a customer, however good the vendor's export becomes. Export formats improve every quarter and that is fine, but the incentives never line up: the vendor optimises for low overhead and tolerance of failure, and you need the reverse. I hold this view from the compliance-data work I describe in the Klimado case study and in The Regulation Is Not Your Schema, where the lesson was that anything auditors may ask about later has to be versioned data you own. My position has a price. You carry a table, a retention policy and a reconciliation job, and you will write some rows nobody ever reads. I accept that, because the alternative is a record you cannot vouch for. For the guardrail side of the same problem, see why agent safety belongs in code, not prompts.

What to do this quarter

List the actions your agents can take today and mark the ones that write to customer data. Add the event table and the wrapper for the two or three highest-risk actions first, revoke UPDATE and DELETE on it, and turn on the vendor's OpenTelemetry export in metadata-only mode as the debugging layer. Run the monthly reconciliation once by hand to learn what your gap is. If you want a second opinion on where the record should sit in your stack, my services page describes how I work as a fractional CTO on agent and compliance architecture.

Find the gap before a customer does

If you are not sure whether your agents leave a record you could defend, book a time and we will map it together.

Book a Time
#AI Agents#Audit Trail#OpenTelemetry#Postgres#Fractional CTO

Keep reading