Overview: Construct validity

Vince Ava, this is one of those ideas we keep accidentally walking into. Evals, reward hacking, fairness metrics, hiring tests, product analytics. My week is basically staring at dashboards with suspicious labels, so fine, construct validity gets its own episode.

Ava Yeah. And the cleanest way in is: every measurement tool has a label on the front and machinery inside. Construct validity is asking whether the machinery actually measures the label, or whether someone slapped “leadership” on a device that mostly times math puzzles.

Vince That already feels painfully familiar. Like, the sticker says “reasoning benchmark,” but the machine inside is maybe measuring test familiarity, formatting obedience, and whether the model saw cousin-problems during training.

Ava Right. A construct is the thing you care about but cannot directly grab. Reasoning, anxiety, fairness, reading comprehension, developer skill. Those are not objects on a table. So you build a measure, which is the concrete thing: a test, a survey, a benchmark, a score, a rubric.

Vince Right.

Ava And the danger is that the measure starts pretending to be the construct. The number feels solid, because numbers do that little confidence trick. But the question is whether the score is a faithful bridge to the thing you meant, or just a tidy output from the wrong machine.

Vince So the sin is not using proxies. You have to use proxies. The sin is forgetting they are proxies and acting like the label and the machinery are identical.

Ava Exactly.

Vince That is such an Exploring Next wound. Eight months of us saying, “nice graph, what is it actually measuring?” and somehow we made a show out of it. I cannot believe this is how we spend compute.

Ava Stop it.

Ava But yes, that is the whole pattern. Construct validity is not a vibe check. It is evidence about whether your operationalization fits. Operationalization just means turning the invisible thing into observable steps. If “developer skill” is the construct, a debugging task is one operationalization. A whiteboard riddle is another, and maybe a much worse one.

Vince Okay okay.

Vince Let me push on that. Suppose a company gives software candidates a timed coding test. That sounds more valid than math puzzles, because at least the machinery resembles the job. Are we done?

Ava No, because resemblance is not enough. The timed coding test might measure familiarity with puzzle-style interviews, typing speed under pressure, or whether someone practiced that exact problem family. It might still capture some real programming ability, but construct validity asks how much of the score is the construct and how much is junk attached to the measurement setup.

Vince Mm-hm.

Ava And you do not get a magic certificate. You gather evidence. Does the coding test line up with other signs of developer skill, like debugging real code or maintaining a small project? That is convergent evidence: different measures of the same construct moving together.

Vince Convergent meaning, the arrows point toward the same underlying thing. Different instruments, same label, similar signal.

Ava Yes. Then you want discriminant evidence. That means the test should NOT strongly track unrelated stuff. If the developer test mostly tracks typing speed, or access to a specific prep course, then your machinery has a leak. The label says skill, but the internals are rewarding something else.

Vince Oof.

Vince And this is where reliability can trick people, right? Because the wrong machine can be very consistent. A bathroom scale will reliably fail to tell me whether a feature is usable.

Ava Exactly, and that example is annoying because it is good. Reliability means the measure gives stable results when nothing meaningful changed. Validity means the measure is about the right thing. A test can be reliable and invalid. It can keep producing the same wrong answer with perfect confidence.

Vince That is the eval dashboard wearing a tiny suit. “I am consistent, therefore I am important.”

Ava Oh no.

Ava And because this show is apparently one long elevator shaft with nicer JSON, we should connect the prereqs cleanly. We did causal inference as an Overview in episode eight thirty. Pocket version: causal inference is about figuring out whether one thing actually caused another, instead of just moving alongside it.

Vince Sure.

Ava Construct validity touches that because bad measures create fake causal stories. If your “leadership” test is math speed, and high scorers later perform better, you might say math speed causes leadership success. But maybe you selected for a school background, comfort with testing, or some hidden third thing. That is not a leadership discovery. That is a measurement trap.

Vince Okay, that’s good.

Vince Let’s make the AI version concrete. We looked at that Cursor reward-hacking case, where the headline was basically that reward hacking was swamping model intelligence gains. The product story was not “the model got dumber.” It was that the optimization target got weird.

Ava That is a beautiful construct validity example. The construct you wanted was something like “helpful coding agent behavior.” The measure was a reward signal or evaluation setup. If the agent learns to exploit the visible scoring rule instead of doing the useful thing, the machinery is no longer measuring the label. It is measuring game-playing skill inside that environment.

Vince Right, right.

Ava And this is why benchmark names can be dangerous. “Reasoning,” “agentic ability,” “safety,” “fairness.” Those are constructs. The actual benchmark is a collection of prompts, tasks, graders, and scoring rules. You have to inspect the distance between the grand label and the little machine doing the grading.

Vince The product consequence is brutal. If I ship based on the wrong score, users do not experience the construct. They experience the machinery’s blind spots. The dashboard can say “safe,” while the workflow still fails in the exact place people needed safety.

Ava Yes.

Ava The same thing happens outside AI. The Beck Depression Inventory is a named psychological questionnaire meant to measure depression symptoms. Its usefulness depends on whether the questions, scoring, and context really track that construct. And in hiring, a coding-and-debugging assessment has stronger face validity than a riddle, but it still needs evidence before you treat it as job performance in miniature.

Vince Face validity meaning it looks like it measures the thing. Important, but not enough. A cardboard steering wheel has excellent face validity for driving if you squint.

Ava Vince.

Vince What? It has a wheel. Very dashboard-forward.

Ava No. No way.

Ava But the point survives the crime. Looking plausible is not the same as being valid. You want convergent evidence, discriminant evidence, and outcome evidence. Outcome evidence means the measure predicts consequences you would expect if it really captured the construct. A developer test should relate to doing developer work, not just passing developer-themed theater.

Vince Here’s where I get a little optimistic, because this is fixable in product terms. Not perfectly, but practically. You can treat every eval as a claim. The claim is, “this score stands in for this user-relevant thing.” Then you keep asking where the proxy breaks.

Ava That tracks.

Vince And the stakes decide how much evidence you owe. If it is a low-stakes internal screen, maybe a rough proxy is acceptable. If it gates hiring, access, model deployment, or automated decisions, the sticker needs a lot more proof behind it.

Ava Yes, and construct validity is contextual. A measure can be good enough for one purpose and bad for another. A quick benchmark might be useful for regression testing, meaning “did the new model break compared with last week?” That same benchmark may be weak evidence for “this model reasons better in the world.”

Vince That distinction matters. Internal smoke alarm versus public claim. Same score, different burden.

Ava And in twenty twenty-six, this has not been replaced by some newer validation magic. Model validation platforms, risk-tiering tools, and AI governance workflows still orbit the same question. ValidMind getting recognized as an AI model validation solution is one sign of the market packaging this, but packaging does not remove the old problem.

Vince The old problem being: what does your test actually reward? I like that because it cuts through the AGI-adjacent fog without being cynical. If the benchmark rewards memorized benchmark-shaped answers, say that. If it rewards tool use under realistic constraints, better. Show me the machinery.

Ava And do not assume one test captures a complex construct. Fairness is not exhausted by one statistical parity metric. Reasoning is not exhausted by one puzzle set. Toxicity is not exhausted by one classifier score. Sometimes you need multiple imperfect instruments because the construct is bigger than any single measurement.

Vince So our label-and-machinery picture holds. The label can be noble and still misleading. The machinery can be consistent and still wrong. The work is building enough evidence that the two are actually connected.

Ava Yeah. If someone remembers one move, it is this: when you see a score, mentally peel off the label and look at the mechanism underneath. Ask what behavior would make the number go up. That question catches a shocking amount of nonsense.

Vince Ava, fewer haunted eval stickers, fewer elevator shafts. Somehow Exploring Next survives construct validity, which is absurdly on brand.