Bringing Advanced Sampling to the OpenTelemetry Collector
Honeycomb is donating its adaptive tail sampling processor to OpenTelemetry, moving beyond rigid static rules toward dynamic, fingerprint-aware sampling that keeps rare traffic visible while capping costs. The key insight: trace fingerprinting plus logarithmic rate normalization lets you hit a throughput or percentage budget across heterogeneous traffic patterns without losing coverage of low-volume journeys.
Transcript
Edmund Okay, so Honeycomb just donated their adaptive tail sampling processor to OpenTelemetry, and I think this is actually a really clean product move—it's the kind of thing that sounds boring until you realize it solves a genuinely painful tradeoff that every big observability shop has been living with.
Geffen Right.
Edmund The problem is: you've got three ways to sample traces—head sampling, probabilistic, tail sampling—and all of them are rigid. You pick a rule, it applies everywhere, and you're stuck choosing between cost and coverage. Tail sampling lets you be smarter, but the rules are static, so if you say 'keep every error' or 'keep 10% of traffic,' you're either blowing your budget or missing the rare stuff.
Geffen Yeah, and the rare stuff is often the most interesting—a low-volume tenant, a checkout flow that only happens because someone applied a discount. You want to see those, but you can't afford to keep everything.
Edmund Exactly. So Honeycomb's move is: instead of static rules, you use trace fingerprinting—you define what makes two traces similar, like the set of services they hit plus the HTTP route plus the tenant—and then you apply adaptive sample rates based on a budget you set.
Geffen So you're grouping traces by fingerprint, then distributing your sampling budget across those groups?
Edmund Yeah, and here's the clever part: they use logarithmic analysis to normalize the volumes. So if your homepage generates a hundred times more traffic than your settings page, the log function lets you give each fingerprint a relative sample rate. The homepage gets sampled harder, but the settings page still gets representation. You hit your budget—either a percentage of traffic or a throughput target, like keep 1000 spans per second—and you don't starve the quiet paths.
Geffen Okay, that's genuinely smart.
Edmund Right? And the rates recalculate every 15 seconds by default, so as your traffic changes, the budget adapts. A DDoS on your homepage doesn't blow your observability budget for checkout.
Geffen That's the real win. Cost control that's actually reliable. Most sampling strategies, you set them and hope, or you end up over-provisioning because you're scared of surprises.
Edmund Exactly. And they're also doing sample rate attribution—they carry the sampling decision downstream in the tracestate so your backend can extrapolate accurately. You know what each trace represents now.
Geffen Hm, so you're not losing fidelity because you can normalize the traces you kept?
Edmund Yeah, the backend can say 'this trace was sampled at one in ten, so this metric is worth ten times what I see.' You get accurate extrapolation.
Geffen That's assuming the backend actually implements it.
Edmund True. But Honeycomb's open-sourcing this as a processor for the OpenTelemetry Collector, so it's not locked to them. You can run it on any collector, any backend.
Geffen And they're bundling it in the Honeycomb Collector Distribution as a drop-in, so you can try it today without waiting for it to land in the official Collector?
Edmund Yeah. It's working its way toward alpha upstream, but you can pull the image right now and run it locally or on Kubernetes with basically a config file and a couple of helm commands.
Geffen Okay, so the mechanism is solid—fingerprinting plus log normalization is a real answer to the coverage-cost tradeoff. But I'm curious: how much does the log function actually matter? Like, if you just did linear normalization, what breaks?
Edmund Mm.
Geffen If your homepage is a hundred times the volume of your settings page, linear normalization would give it a hundred times the sample rate, and the settings page would barely show up. Log flattens that curve, so you get representation across the range.
Edmund Right, and that's the actual insight—you're not trying to keep the same proportion of each fingerprint, you're trying to keep enough of each one to see patterns. Log does that.
Geffen It's a neat trick. I don't think it's magic, but it solves a real problem that static rules don't touch.
Edmund Yeah, and the fact that they're donating it instead of gatekeeping it behind a Refinery subscription is the move that makes this actually land. This was locked to Honeycomb customers before, and now any Collector operator gets it.
Geffen That's the product play right there. The technology is solid, but the distribution is what makes it real.
Edmund Exactly. So if you're running observability at scale and you're tired of choosing between cost and coverage, this is the thing to try today.
Geffen Docker pull, pass a config, and you're running adaptive sampling. Pretty clean.
Edmund Yeah, Geffen, I think we're done here.