Where Causal Structure Actually Appears

01 The Question

A reasonable question from anyone evaluating a causal AI practice: if causal structure is so important, why isn't it already inside the models?

The short answer is that it is not, and the reasons are worth understanding rather than dismissing. There is no Bayesian network anywhere in the training stack of a frontier language model. There is no elicited DAG, no conditional probability table, no do-operator.

But the underlying machinery of graphical models, and the logic of Pearl's interventions, both appear in serious places. They appear in different places, and the field routinely confuses the two. What follows separates them.

02 The Word Trap

Start here, because it derails more conversations than anything else on this page.

The standard term for a GPT-style model is a causal language model. The mechanism that makes it work is called a causal attention mask. Both terms are ubiquitous in the literature and in every major framework's documentation.

Neither has anything to do with causation. "Causal" here means temporal: the model may attend to earlier tokens but not later ones. It is a masking convention1 borrowed from the language of causal order in time series. It carries no claim about mechanism, intervention, or counterfactuals.

The practical consequence: executives arrive having read that their model is already causal, because the documentation says so. It is a naming collision, not a capability.

03 Graphical Models in Disguise

Several modern generative architectures are graphical models with neural conditional distributions. The graphs are simple and the math is inherited directly from the probabilistic modeling tradition.

The pattern across all four: the graph carries almost no structure. It is a chain or a clique. All the modeling capacity sits in the neural conditionals. That is close to the opposite of a causal model, where the structure is the asset and the parameters are secondary.

04 Where Pearl Landed

This is the strongest genuine intersection, and it is now standard practice at the frontier labs.

Mechanistic interpretability treats a trained network as a structural causal model in its own right. Activations are the variables. The computation graph is the DAG. Researchers then do exactly what Pearl prescribes: intervene and measure.6

Activation patching replaces an activation with a value from a different run and measures the downstream effect on the output. Path patching isolates specific routes through the network.10 Causal mediation analysis has been used to localize where bias and factual recall live.7 Model-editing methods locate a factual association and overwrite it.8,9 Causal abstraction formalizes the question of whether a human-readable high-level causal story is a faithful abstraction of the low-level computation.11,12

There is a structural reason this works so well. A neural network is a deterministic SCM with no unobserved confounding and full interventional access. You can set any variable to any value and rerun. The hard problem of causal inference, identifying effects from observational data when you cannot experiment, simply does not arise.

What remains hard is different: the variable space is enormous, and the abstraction problem is unsolved. Which is worth stating plainly to technical audiences who assume causal methods are only for data-poor settings. Here they are used in the most data-rich setting that exists.

05 Data and Training

Causal framing also shows up in what goes into a model and how the objective is shaped. The record here is mixed, and honesty about that is more useful than advocacy.

Counterfactually augmented data: minimal edits to an input that flip its label, is used for robustness and debiasing.13 It works, and the reasoning behind it is properly causal: you are constructing pairs that differ on the factor of interest and nothing else.

Invariant risk minimization and related invariance methods borrow the idea that causal relationships remain stable across environments while spurious ones do not.14 The idea is sound. The empirical results have underdelivered relative to the initial reception, and that is now reasonably well established.15,16,17

Backdoor-adjustment-flavored debiasing appears throughout the vision and question-answering literature. Quality varies widely. A meaningful fraction of these papers are reweighting schemes decorated with do-calculus notation, where the causal graph is asserted rather than defended.

Causal representation learning: recovering latent variables that correspond to genuine causal factors rather than arbitrary directions, is the ambitious version of this program.18 It is real research and largely pre-production.

06 Adjacent, Not the Same

Two families that get pulled into this conversation and should be kept out of it.

Bayesian deep learning: weight priors, Laplace approximations, Monte Carlo dropout, deep ensembles, is about uncertainty over parameters. It is Bayesian. It is not causal, and it is not structural. Conflating the two muddies every subsequent claim, because a well-calibrated uncertainty estimate over a correlational prediction is still a correlational prediction.

Message passing is a genuine lineage rather than a confusion. Graph neural networks descend directly from belief propagation, and there is a line of work that unrolls belief propagation into learned architectures for factor-graph inference.19 Related: neural amortized inference, where a network is trained to approximate posteriors for simulator-based models. Real, useful, and about computing inference efficiently, not about establishing causal direction.

07 At Inference Time

The remaining category is the one that matters commercially: causal models and language models operating together at decision time, neither one absorbed into the other.

The language model handles what it is good at, reading documents, parsing an executive's question, translating between natural language and formal structure. The causal model handles what it is good at, holding the mechanism, answering interventional and counterfactual queries, and stating its assumptions in a form a regulator can audit.

There is also a research direction using language models as priors for causal discovery: proposing candidate edges or orienting them from domain literature. Reported performance on pairwise orientation benchmarks has been surprisingly strong.20 It carries an unresolved objection, many benchmark graphs appear in the training corpus, so it is difficult to separate reasoning from recall.21

08 The Line Worth Holding

Pulling it together:

Causal structure is not used to build frontier models. The architectures borrow graphical-model math, but with graphs so trivial that they encode no causal content.

Causal structure is used to understand them after training. Interpretability is real applied causal inference, and it is the field's most successful adoption of Pearl's framework by a wide margin.

Causal reasoning is used to curate what goes into them, with genuine but uneven results.

And causal models sit alongside them at inference, which is where the enterprise value is.

Anyone claiming that today's models have causal reasoning built in is describing a research aspiration, not a shipped capability. That is a defensible position, and it happens to be the one that makes the architectural argument rather than undermining it.

09 References

1. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. Attention Is All You Need. NeurIPS, 2017. Origin of the masked self-attention mechanism later termed "causal."
2. Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., & Ganguli, S. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. ICML, 2015.
3. Ho, J., Jain, A., & Abbeel, P. Denoising Diffusion Probabilistic Models. NeurIPS, 2020.
4. Kingma, D. P., & Welling, M. Auto-Encoding Variational Bayes. ICLR, 2014.
5. Rezende, D. J., & Mohamed, S. Variational Inference with Normalizing Flows. ICML, 2015.
6. Pearl, J. Causality: Models, Reasoning, and Inference. 2nd ed., Cambridge University Press, 2009.
7. Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., & Shieber, S. Investigating Gender Bias in Language Models Using Causal Mediation Analysis. NeurIPS, 2020.
8. Meng, K., Bau, D., Andonian, A., & Belinkov, Y. Locating and Editing Factual Associations in GPT. NeurIPS, 2022. arXiv:2202.05262. Introduces causal tracing and ROME.
9. Meng, K., Sen Sharma, A., Andonian, A., Belinkov, Y., & Bau, D. Mass-Editing Memory in a Transformer. ICLR, 2023. arXiv:2210.07229. MEMIT.
10. Wang, K. R., Variengien, A., Conmy, A., Shlegeris, B., & Steinhardt, J. Interpretability in the Wild: A Circuit for Indirect Object Identification in GPT-2 Small. ICLR, 2023.
11. Geiger, A., Lu, H., Icard, T., & Potts, C. Causal Abstractions of Neural Networks. NeurIPS, 2021.
12. Geiger, A., Ibeling, D., Zur, A., Chaudhary, M., Chauhan, S., Huang, J., Arora, A., Wu, Z., Goodman, N., Potts, C., & Icard, T. Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability. Journal of Machine Learning Research 26(83), 2025. arXiv:2301.04709.
13. Kaushik, D., Hovy, E., & Lipton, Z. C. Learning the Difference that Makes a Difference with Counterfactually-Augmented Data. ICLR, 2020. arXiv:1909.12434.
14. Arjovsky, M., Bottou, L., Gulrajani, I., & Lopez-Paz, D. Invariant Risk Minimization. 2019. arXiv:1907.02893.
15. Rosenfeld, E., Ravikumar, P., & Risteski, A. The Risks of Invariant Risk Minimization. ICLR, 2021. arXiv:2010.05761.
16. Gulrajani, I., & Lopez-Paz, D. In Search of Lost Domain Generalization. ICLR, 2021. arXiv:2007.01434. Finds standard empirical risk minimization competitive with invariance-based methods under fair model selection.
17. Kamath, P., Tangella, A., Sutherland, D. J., & Srebro, N. Does Invariant Risk Minimization Capture Invariance? AISTATS, 2021.
18. Schölkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, A., & Bengio, Y. Toward Causal Representation Learning. Proceedings of the IEEE 109(5), 612–634, 2021.
19. Yoon, K., Liao, R., Xiong, Y., Zhang, L., Fetaya, E., Urtasun, R., Zemel, R., & Pitkow, X. Inference in Probabilistic Graphical Models by Graph Neural Networks. 2018. arXiv:1803.07710.
20. Kıcıman, E., Ness, R., Sharma, A., & Tan, C. Causal Reasoning and Large Language Models: Opening a New Frontier for Causality. Transactions on Machine Learning Research, 2024. arXiv:2305.00050.
21. Zečević, M., Willig, M., Dhami, D. S., & Kersting, K. Causal Parrots: Large Language Models May Talk Causality But Are Not Causal. Transactions on Machine Learning Research, 2023. arXiv:2308.13067.