NVIDIA's Nemotron 3.5 Super VL with post training beats Claude Opus 5, GPT 6.1 Sol on specialized task benchmarks
Edmund and Geffen dig into NVIDIA’s Nemotron 3.5 Super VL launch, with Geffen skeptical of benchmark-cherry-picked claims and Edmund focused on the product reality: a multimodal model that can be post-trained on enterprise docs, tools, and policies, plus local Bolt variants for smaller deployments. They land on the idea that the real story is not “it beats Claude or GPT” in the abstract, but that NVIDIA is selling an adaptable stack with deployment control, while the public benchmark chart still needs independent verification.
Transcript
Edmund Okay, the thing that jumps out is NVIDIA basically saying, “trust us, this custom post-trained model beats Claude and GPT on the task.” That’s either a real enterprise wedge or a very expensive benchmark screenshot.
Geffen Yeah, and my first reaction is: which task, which dataset, which protocol? They show five bars, but not the task definitions or sample sizes, so the headline is carrying way more weight than the evidence.
Edmund Sure, but the user story is the part that matters. If you’re an enterprise agent reading PDFs, charts, tables, web pages, and your own policy docs, the win is not some abstract leaderboard crown. It’s whether one model can do the whole ugly chain without you stitching together three systems and praying.
Geffen Mm-hm.
Edmund And that’s why Super VL is more interesting than the usual ‘new model, bigger number’ thing. NVIDIA is saying you can post-train it on your own tools and rules, then choose whether it lives on Blackwell, Hopper, or something local. That’s a product move, not just a model move.
Geffen Right, but I don’t want to let them smuggle in the conclusion. The chart shows specialized versions on specialized tasks. It does not show the base model beating Claude Opus 5 or GPT-6.1 Sol generally. That distinction matters, because otherwise everyone hears “open model wins” and stops reading.
Edmund No, that’s fair. But if you’re the buyer, the base model is almost the wrong unit anyway. You’re buying the adapted system, the training recipe, the deployment path, the security story. That’s exactly the boring-but-works lane we keep ending up in. Very Exploring Next sentence of them, honestly.
Geffen Okay, that phrase is cursed, but yeah, you’re right. The annoying part is that NVIDIA also didn’t give the evaluation details, so you can’t tell whether the gains came from real generalization or from tuning to the shape of the benchmark.
Edmund And there’s the actual test. If a claims-review workflow or credit-agreement workflow gets better after post-training, nobody cares whether the raw base model was prettier on paper. They care that the model extracts the right facts and follows the rules. That’s the adoption path.
Geffen Exactly. The one thing I do like is the family structure. Super VL for the big multimodal job, Bolt 74B for local agent work, Bolt E4B for tiny device stuff. That is a sane menu. It’s closer to procurement than to vibes.
Edmund And the local angle is real. If Bolt 74B actually fits in twenty-four gigabytes with that 2-bit compression, that matters for teams that do not want every code check or research pass leaving the machine. Not sexy, but it ships.
Geffen Mm-hm.
Edmund The speed chart is where I get a little side-eye, though. Tokens per second at one hundred thousand input tokens is useful, but it’s not the same as task latency. Tool calls, retries, reading the output, all of that still lives in the workflow.
Geffen Yeah, that’s the key correction. A fast generator can still be a slow agent if it spends half its life waiting on retrieval, browser steps, or validation. So the five-times-faster claim is real in the narrow sense, but it’s not the whole product story.
Edmund Still, if the model is fast enough and good enough, that changes what people are willing to automate. You stop treating each multimodal step like a precious event. You just run more of them. That’s where the market pull comes from.
Geffen I buy that more than I buy the benchmark theater. And I do think NVIDIA’s doing the right kind of boring packaging here: open weights, training recipes, NIM, Hugging Face, OpenRouter, the whole distribution stack. That’s the part other teams can actually pick up.
Edmund Yeah. It’s not a victory lap, it’s a menu. Which is why the real question is whether anyone outside NVIDIA’s chart reproduces the gains on their own mess. If they do, this is useful. If not, it’s a very polished weekend of benchmark cosplay.
Geffen That’s the whole thing. I’m not ضد the model. I’m against pretending a custom scorecard is the same as a broadly better foundation model. Give me the receipts, then we can talk.
Edmund Come on, Geffen, you only sound this happy when something is suspicious. Which, okay, is most weeks on this show.
Geffen It keeps you honest. And this one does need honesty.
Edmund Yeah. Fine. My bet is somebody outside NVIDIA has a public replication or a serious teardown of those task gains within a few weeks, and the real story stays the enterprise adaptation path, not the headline model duel.
Geffen I’d take the other side on the replication pace, a little. Give it a month before anyone produces a clean comparison with the same tasks and enough detail to trust it.
Edmund That feels right enough to argue about later, which is basically our brand. Also, if Bolt E4B lands on October twenty-third the way they say, I’m going to pretend that was the whole plot all along.
Geffen Of course you are. That’s such an Edmund move.
Edmund Guilty. Anyway, let’s see if the boring stack wins again. I’m weirdly optimistic, which probably means you should be worried.