OpenTelemetry, both directions: the monitor speaks OTLP in and out

Manouk and I recently wrote about merging an LLM trace and a cache decision into a single waterfall with zero custom glue, because BetterDB and LangWatch both happened to speak OpenTelemetry. In her follow-up from the observability seat, she made the point that stuck with me: most observability platforms are OTel-native in one direction. They'll happily ingest your spans, then hand you a proprietary SDK to produce them. The bet that pays off, in her words, is to be open in both directions - ingest is open, and emit is open.

That post was about the chat app and the cache libraries. This one is about the monitor - the thing you run next to your Valkey - and what happened when we took "both directions" literally. The short version: because every edge of BetterDB now speaks plain OTLP, it drops into whatever slot your pipeline has open. Already have a collector? We feed it. Don't have one? We are one. Want spans out of the cache? They're standard spans. Already have a trace? Send it, and we'll tell you why a given cache hit or missed, using the live database state the trace itself can't see. There's no BetterDB-shaped hole you have to cut in your stack to make room for us.

The monitor already had a language. It just wasn't the shared one.

The monitor has spoken Prometheus for a long time. Scrape /metrics and you get a few hundred betterdb_* series: memory, keyspace hits and misses, replication offset, cluster slot health, per-command latency, ACL denials, anomaly counters, vector-index docs. It's a good format and a lot of people already run it.

But Prometheus is a pull-based metrics format. It isn't spans, it isn't logs, and it isn't the thing an OpenTelemetry collector expects on the other end of an OTLP exporter. So if your stack is standardizing on OTLP - one collector, one pipeline, one place everything lands - the monitor was a special case you had to wire up separately. That's exactly the "open in one direction" trap, just pointed the other way: we emitted, but only in our own dialect.

So we added two things, in the two directions.

Direction one, outbound: mirror what we already know over OTLP

The nice part about this one is how little new code it needed, and I mean that as a compliment to the standard, not to us.

The monitor already builds a full prom-client registry every poll. Instead of teaching every metric site a second way to emit, the OTLP mirror reads that same registry - the exact snapshot the /metrics scrape serves - and replays each data point into an OpenTelemetry instrument on an interval. Point it at a collector and the same numbers you'd scrape start arriving as OTLP metrics.

A few deliberate choices fell out of doing it this way:

• Names carry over verbatim. betterdb_memory_used_bytes in Prometheus is an instrument literally named betterdb_memory_used_bytes in OTLP. Prometheus labels (connection, db, slot, severity) become OTLP attributes. Your existing dashboards and alerts translate one-to-one, because we refused to invent a second naming scheme.
• Units get inferred, not renamed. A _bytes suffix becomes the UCUM unit By, _seconds becomes s, _ratio becomes 1. The unit lands in OTLP metadata where it belongs, and the metric name stays what you already know.
• Histograms are honestly skipped. prom-client only exposes pre-aggregated buckets, and the OTel observable API wants individual values, so a histogram like betterdb_poll_duration_seconds can't be mirrored losslessly. Rather than fabricate points, the mirror leaves it out. Gauges and counters - which is nearly everything - come through.

There's a second outbound stream that isn't metrics. The discrete events the monitor already sends to webhooks - an instance going down or back up, a cluster failover, a compliance alert when memory crosses a threshold under noeviction - are mirrored as OTLP log records, decoupled from the webhook gate. If you run an OTLP-only deployment with no webhooks configured at all, those events still show up in your collector.

Turning it on is one variable:

# Emit betterdb_* metrics and monitoring events over OTLP.
# No-op until the endpoint is set - off by default, nothing leaks.
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
# optional; defaults to 15s
OTEL_METRICS_EXPORT_INTERVAL_MS=15000

Metrics post to ${endpoint}/v1/metrics, events to ${endpoint}/v1/logs, both as OTLP over HTTP/protobuf - the OTel SDK default, no config flag to flip. Leave the endpoint unset and the whole thing stays inert.

Direction two, inbound: ingest OTLP spans, then explain them

The other direction is the one from the last post, now living inside the monitor. The @betterdb/* cache and memory libraries already instrument themselves against the global OpenTelemetry tracer - semantic_cache.check, agent_cache.llm.check, agent_memory.recall, all carrying cache.hit, cache.key, TTLs, and the model. Any OTel backend can read them. Now the monitor is one of those backends.

Your app points its OTLP exporter at the monitor:

OTEL_EXPORTER_OTLP_ENDPOINT=http://<monitor-host>/v1/traces
# JSON works too, if you'd rather not send protobuf:
# OTEL_EXPORTER_OTLP_PROTOCOL=http/json

POST /v1/traces is the standard OTLP/HTTP path - we deliberately kept it off the monitor's /api prefix so a stock exporter hits it with no rewriting. It accepts both encodings the ecosystem actually uses: application/json, and application/x-protobuf, which is what an unconfigured OTel SDK sends by default. For protobuf we decode against a minimal inline .proto with just the fields we read, and we wire a Long backend on purpose so nanosecond timestamps above 2^53 don't quietly round.

So how much do we actually ingest?

"We support OTLP ingestion" can mean almost anything. Some tools swallow the whole firehose. Others take their own spans and quietly drop the rest. So here's exactly what ours does.

We accept everything. /v1/traces is a real OTLP collector endpoint - it takes a full ExportTraceServiceRequest from any instrumented service, not just BetterDB. Your web framework, your AI SDK, your own hand-rolled spans, all of it. Every span in the payload is parsed. Point a general-purpose OTel exporter at it and nothing bounces.

We store selectively, and tell you the difference. Out of everything received, the monitor persists the @betterdb/* cache and memory spans plus each trace's root - the parentless span like chat.turn that frames the request. The intermediate spans from other libraries are parsed, counted, and dropped from storage. The ingest returns both numbers - spans received versus spans stored - so it's never silently swallowing part of your trace; it's telling you it kept the eight cache spans and the one root out of the forty you sent.

That selectivity is a decision, not a limitation. The monitor isn't trying to be your general-purpose trace store - you almost certainly already have one of those, and it speaks OTLP too, so send your traces there in parallel. What the monitor is uniquely positioned to be is the cache-and-memory lens on those traces: send it the whole request, it keeps the spans it can say something authoritative about, and it hangs them off the root so the waterfall still reads as one turn instead of a pile of disconnected cache checks. You get the focused view here and the full trace wherever your backend already lives, from the same exporter, with no forked configuration.

From there it stamps a per-trace summary - span count, how many are BetterDB, duration, error state, service - and renders an AI Traces waterfall: the cache and memory spans inside each request, the BetterDB ones tinted so they stand out from the app's own root. On self-hosted, traces age out on a seven-day local window (whole traces at a time, so a long request never loses its early spans while later ones survive); nothing you have to schedule.

That much is a competent OTLP trace viewer. Here's where sitting on the database changes what the view can say.

The part a pure trace tool can't do

A trace is a recording of what happened. It's honest, but it's frozen: the span says cache.hit=false and that's all it will ever say. The question you actually have at 3am is why, and the answer usually lives in the database's state right now, not in the span from four minutes ago.

The monitor is already holding a live connection to that Valkey. So for every BetterDB span in a trace, it can join the recorded outcome against the current state - run EXISTS and TTL on the cache key the span acted on, read the instance's similarity or recall threshold, check the vector index's build state - and turn that into a sentence. A few of the cases it distinguishes:

• The cold miss that self-healed. The span reported a miss, but the key exists now with a live TTL: "it was populated after this request (cold miss; later calls hit)." The trace alone can't tell you the miss was temporary. The join can.
• The hit that won't repeat. The span reported a hit, but the key is absent now: "Hit at request time, but the key has since expired or been evicted." That's the difference between "the cache is working" and "the cache worked once."
• The genuine miss, with a reason. A semantic or memory miss has no matched key, so it surfaces the instance context instead: "nothing matched above the recall/similarity threshold (threshold 0.82)." Now you can see whether the threshold is the thing to move.
• The degraded index. If the vector index isn't ready, it appends: "Index state is 'building' - recall may be degraded." Which reframes a run of misses from "the cache is bad" to "the index hadn't finished."

None of these explanations exist in the span. They exist in the join between the span and the live database, and only something already sitting on that database can turn a frozen cache.hit=false back into a why.

Where this lets BetterDB sit

Put the two directions together and the point of the whole exercise comes into focus. There isn't one integration; there are four positions BetterDB can occupy in an OTLP pipeline, and you pick whichever your stack leaves open:

• Feed the collector you already run. Set OTEL_EXPORTER_OTLP_ENDPOINT and every betterdb_* metric and monitoring event lands in your Grafana/Tempo/Datadog/Honeycomb/OTel-Collector pipeline next to everything else - no Prometheus scrape target to add, no exporter sidecar, no separate dashboard to babysit.
• Be the collector, when you don't have one. The monitor's own /v1/traces endpoint accepts anyone's spans, so a small team with no observability platform can point their app straight at the monitor and get a real trace view. You don't need to stand up a stack to start seeing traces.
• Produce spans anywhere. The @betterdb/* cache and memory libraries emit standard spans against the global tracer. Whatever backend you already trust - ours or a competitor's - reads them, because there's nothing proprietary on the wire.
• Consume spans and add to them. Send the monitor a trace and it doesn't just store it; it correlates the cache and memory spans against live Valkey state and hands back the why. That's the one position on this list nobody without a live database connection can fill.

Notice none of these require the others. You can take the emit side and never ingest a span. You can take the ingest side and never touch our metrics. You can use our libraries with someone else's backend, or someone else's libraries with our endpoint. Every edge is a standard OTLP boundary, which means every edge is optional - and that's exactly what "fits in any part of the stack" has to mean if it's going to mean anything. Not one blessed integration path, but a component that composes wherever there's a socket shaped like OTLP, which is increasingly everywhere.

Open, and safe by default

Emitting and ingesting telemetry only helps if it's also safe, and the defaults lean that way.

The privacy posture starts one layer up, at the emitter. The @betterdb/* libraries default their spans to metadata only - hit or miss, similarity, latency, cost - and attach prompts, responses, and recalled documents only when content capture is explicitly turned on. The monitor stores whatever attributes it's handed, so keeping payloads out is a decision you make at the emitter, not something the monitor does for you. What it does enforce is access. Ingestion can be gated behind a bearer token (OTEL_INGEST_TOKEN), and in cloud mode - where the endpoint sits past session auth - it fails closed: with no token configured it refuses spans rather than accepting anonymous ones into a tenant's store. Both the outbound mirror and the whole monitor are self-hostable, so if you'd rather nothing leaves your network, nothing does.

The leaner image is on Docker Hub. Point an OTLP exporter at /v1/traces, set OTEL_EXPORTER_OTLP_ENDPOINT to push your metrics the other way, and open the AI Traces page. If a span you care about isn't getting persisted, open an issue - that's exactly the kind of gap we want to hear about.