Overview: Confounding Variables

Jessica Cathy, we keep throwing around “confounder” like everyone was born inside a stats textbook. People have been asking us to slow down, so fine. Episode eight twenty-eight gets the hidden-variable couch treatment.

Cathy Good, because this is one of those words that sounds fancy, but the basic idea is almost annoyingly practical. If you misunderstand it, you start believing every chart with two lines going up.

Jessica Tiny status update before we touch the stove: my week is a spreadsheet that learned to glare at me. Which is thematically perfect, because confounding is basically when the spreadsheet lies with confidence.

Cathy Oh no.

Cathy Picture two dials on a wall. One says coffee drinking, one says lifespan. You watch both rise together and think, okay, coffee must be pushing lifespan. But behind the wall there may be a hidden knob turning both. Causal inference is figuring out what actually causes what, instead of treating visible patterns as causal.

Cathy That hidden knob is the confounding variable. It is a factor you did not measure, or did not control, that influences both the supposed cause and the outcome.

Jessica So the trap is not just “two things are correlated.” It is “two things are correlated, and I am telling a story where one caused the other, while the real driver is behind a curtain eating chips.”

Cathy Yes. Correlation just means two things move together. Confounding means a third thing helps explain why. The third thing can create a fake link, or hide a real link by making it look weaker, stronger, or weirdly backwards.

Cathy The classic kid example is shoe size and reading ability. Bigger shoes seem linked to better reading. But age is the hidden knob. Older children tend to have bigger feet, and older children tend to read better. The shoes are not teaching phonics.

Jessica And this is where my product brain gets nervous, because a dashboard can make the wrong thing look like the magic lever. You launch a study app, the students using it get better scores, and suddenly everyone wants a celebratory landing page.

Cathy Jessica, that example is perfect because the app users might be more motivated. Or they may already study more hours. Or they may have tutors. Motivation or tutoring is turning both dials: app usage and grades.

Cathy If you do not account for that, the app gets credit it has not earned. And the reverse can happen too. A genuinely useful app might look mediocre if the group using it started off with harder classes.

Jessica Okay, but slow it down one notch. When you say “account for it,” what is the actual move? Because that phrase can hide a lot of hand-waving.

Cathy There are two broad moves. You can measure the possible confounder and adjust for it, or you can design the experiment so the confounder is less able to sneak in. Adjustment means comparing like with like. Randomization means assigning people by chance, so hidden knobs are less likely to pile into one group.

Jessica So with the app, instead of letting the most motivated students choose it, you randomly assign some students to use it and some not to use it. Then motivation should be spread around more evenly, at least in theory.

Cathy Yes. Not magically perfect, but much better. Randomization breaks the link between the hidden knob and who gets the treatment. Treatment just means the thing being tested, like the app, the feature, the policy, the new model, whatever.

Jessica And if you cannot randomize, because real life is annoying and users keep behaving like users?

Cathy Then you are in observational data land. You are watching what happened rather than controlling who got what. It is often all you have. But it is where confounders flourish, because the world assigned the groups for reasons you may not know.

Cathy This is where conditional probability matters. The quick version is: it asks how likely one thing is after you know another fact. So instead of asking, “do app users score higher,” you ask, “among students with similar study hours, do app users score higher?”

Jessica That feels like turning on the lights behind the wall. You are not removing every hidden knob, but you are saying, wait, this one knob is visible now, so stop letting it impersonate the app.

Cathy Say a hiring model sees historical data where people from a certain school were hired more often. It might learn the school name as a signal of future success. But the hidden knob could be past recruiters already favored that school, so the data contains a selection pattern, not proof.

Jessica That is the part that makes this more than a stats-class warning label. The model is not sitting there reasoning, “I found a confounder.” It just sees a useful-looking pattern and grabs it.

Cathy Exactly. No soul, just math, like we are calming down a raccoon again. If the confounded shortcut helps on that data, the model has no built-in moral objection to using it.

Jessica There it is. The raccoon is back.

Cathy And this connects to that CausalMix work we went through, where data mixture for language model training gets treated as a causal inference problem. The interesting part was asking which data sources are actually helping the target behavior, instead of just riding along with correlated factors.

Jessica So in our hidden-knob picture, the training mixture has a bunch of dials. Maybe code data, math data, web text, instruction data. If performance improves, you cannot automatically crown the loudest dial.

Cathy Yes. You have to ask whether a data source caused the improvement, or whether it came bundled with another property that did the work. This is the same caution behind confounder-free continual learning ideas, where the goal is to stop a model from clinging to features that predict the target in one phase but fail later.

Cathy And there are methods like rank-to-group scoring that try to discover hierarchical confounder effects in raw data or model-derived data. The method details are not the point here. The point is that modern systems still need ways to find hidden knobs, not just hope bigger models sort it out.

Jessica Let me put that a better way for my side of the house. If a team ships an A I feature based on a confounded eval, they may optimize the wrong thing. They might think the new prompt helped, when really the test set got easier.

Cathy Yes, and that is the product version of the same statistical mistake. Power users can be the hidden knob. They adopt the feature earlier, know the workflow better, and get better outcomes. Then the feature looks brilliant even if it mostly selected for people already good.

Jessica This is where I hear your skepticism sharpening knives in the kitchen.

Cathy A little. But this is not pessimism. It is hygiene. If the claim is “users who touched the agent saved time,” I want to know who those users were, what tasks they chose, and whether the comparison group had the same task mix.

Jessica No, that is fair. Annoying, but fair.

Cathy And there is a nasty limit. You can only control for confounders you identify and measure well enough. If motivation matters but you measure it with a sloppy survey, the hidden knob is only partly visible. If you never measure it, it can keep fooling you.

Jessica So randomization is the cleaner move, but not a force field. It spreads the known and unknown weirdness around, assuming the assignment really is random and the groups are big enough.

Cathy Right. And in messy systems, randomization can break after assignment. People drop out. They ignore the treatment. The measurement changes. The groups get different side effects. So even experiments need judgment, not just a victory stamp.

Jessica Where does confounding stand now, then? Because some concepts we cover are old terms buried under newer machinery. This one feels… not buried.

Cathy Not buried. Still completely live. Causal graphs, proxy-variable methods, randomized trials, adjustment techniques, all of that is tooling around the same core problem. The field has not moved past confounding. It keeps rediscovering how expensive causal claims from messy data are.

Jessica And in A I, the stakes got sharper because models scale the mistake. A person misreads one chart. A model can absorb a confounded pattern and apply it across a product, quietly, over and over.

Cathy Yes. That is why I do not treat confounding as academic trivia. It is one of the load-bearing reasons evals, training data choices, product analytics, and governance dashboards can all look more certain than they are.

Jessica Okay, the one thing I want to stick is the hidden knob. If A and B move together, do not immediately draw an arrow from A to B. Ask what C might be turning both.

Cathy That is it. Not every pattern is fake, and not every third factor matters. But the habit is to look behind the wall before you build the product, publish the claim, or fine-tune the model.

Jessica Okay, Cathy, episode eight twenty-eight: I fear the knob. That is the whole emotional arc.