Against Black Boxes
For an Executive ReaderA model whose parameters cannot be read is a model whose assumptions cannot be contested. That is not a property you want in a system that makes consequential decisions.
01 The claim
Accuracy without explainability is not a complete answer.
The standard defence of black-box models is accuracy. A neural network with 200 million parameters outperforms a logistic regression on almost every benchmark. That is true. It is also beside the point.
The question is not whether a black box can be accurate. The question is whether an accurate black box is appropriate for a decision that must be explained, contested, or held up under audit. In most regulated enterprise contexts, the answer is no, not because accuracy doesn’t matter, but because accuracy without explainability is not a complete answer to the question the decision requires.
An accurate model that cannot explain itself cannot be challenged. If you cannot say which variables drove a decision and why, you cannot identify when the model is wrong in a specific, correctable way. You can only retrain it and hope.
02 What a black box is
Parameters you cannot read. Assumptions you cannot challenge. Errors you cannot locate.
A black box is a model whose internal representation is not human-readable. The parameters, weights, embeddings, attention matrices, encode relationships between variables, but in a form that cannot be directly inspected, interpreted, or challenged by a domain expert.
Implicit representation. A black box encodes what it learned from the training data. If the training data contains a confound, a variable that correlates with the target for reasons unrelated to the mechanism of interest, the black box will encode that confound as signal. It has no way to distinguish a confound from a cause, because it has no causal structure.
Post-hoc explainability is not the same as explainability. SHAP values, LIME, and attention maps are approximations of a black box’s behaviour, not representations of its actual reasoning. They tell you which inputs the model’s output is sensitive to, not which inputs caused the output. That distinction is Pearl’s Ladder again: sensitivity is Rung 1; causation is Rung 2.
Retraining is not correction. When a black box is wrong, the response is to retrain on more data or adjust the architecture. There is no mechanism to point at the specific assumption that was wrong and fix it. The error is not locatable because the representation is not readable.
03 The auditability gap
The inputs can be listed. The reasoning cannot be verified.
Auditability requires that a decision can be explained in terms of the inputs, the reasoning, and the assumptions that connected them, and that each of those can be independently verified. A black box fails on the second and third counts.
The inputs can be listed but the reasoning cannot be verified. A SHAP value says that feature X contributed 0.3 to the output. It does not say why X is relevant, whether X is a cause or a correlate, or whether the model’s reliance on X would hold under a different data distribution. There is nothing to point a regulator at except the number itself.
The assumptions are implicit. Every model embeds assumptions about which variables matter and how they relate. In a black box, those assumptions are encoded in the weights and are not separately inspectable. In an explicit causal model, they are named edges in a graph, each one a stated, falsifiable claim about the world.
For decisions made under regulatory scrutiny, credit, clinical, insurance, employment, the implicit assumption is not a minor inconvenience. It is a liability. When a decision is challenged, the question is not whether the model was accurate on average. It is whether the specific decision was made for defensible reasons. A black box cannot answer that question.
04 The falsifiability argument
A black box cannot be wrong in a specific, correctable way. It can only be retrained.
Gary Marcus’s critique of deep learning is relevant here: a system that encodes knowledge implicitly, in weights adjusted by gradient descent, cannot be wrong in a specific, correctable way. It can only be retrained. That is a problem for science; it is also a problem for enterprise governance.
An elicited causal model is more falsifiable than a black box, not less. Every edge in the graph is a stated claim: X causes Y. That claim can be tested against domain knowledge, against historical data, against natural experiments. When it is wrong, you can find it and fix it. The model becomes more trustworthy over time in a way that is legible to the people responsible for it.
Named edges are not a constraint on intelligence. The causal graph does not prevent the model from representing complex relationships. It requires that those relationships be stated explicitly, in a form that a domain expert can inspect and contest. That requirement is what makes the model auditable, and auditable is what a regulated decision system needs to be.
The objection that explicit models are less accurate than black boxes is true in the same sense that a contract is less flexible than a handshake. The explicitness is the point. A model that a regulator can read and challenge is a different kind of deliverable than one they cannot, and in regulated contexts, it is the only kind that is fit for purpose.
05 The alternative
An explicit model. Named edges. Stated mechanisms. An audit trail that is a first-class artifact.
The alternative to a black box is not a simpler model. It is an explicit model, one in which the relationships between variables are named, directional, and elicited from domain experts.
A structural causal model encodes what causes what. Each edge is a stated mechanism. Each parameter is a quantified claim. Each assumption is visible. When the model is wrong, the error is locatable, a specific edge that should not be there, a parameter that is miscalibrated, a confound that was not encoded. The fix is also locatable.
The explanation is not generated. It is read off the model. Pattern analysis and impact analysis (ARCH2-Pattern) produce explanations in the model’s own variables, not post-hoc approximations of a black box’s behaviour. The explanation is the model’s reasoning, not a reconstruction of it.
The audit trail is a first-class artifact. Every query produces an AuditRecord: the estimand, the identification certificate, the evidence set, the assumptions, the posterior. This is not logging. It is a typed, versioned, contestable record of how the decision was reached, something a regulator can inspect and a court can evaluate.
06 In practice
The feedback loop is slow, opaque, and impossible to direct. An explicit model closes it.
The practical consequence of the auditability gap is that organisations deploy systems they cannot explain when challenged, correct only by retraining, and improve only by accumulating more data of the same kind. The feedback loop is slow, opaque, and impossible to direct.
An explicit causal model closes that loop. When a decision is challenged, the model can say which variables drove it and why. When the model is wrong, the error is findable and fixable. When the world changes, the mechanism that changed can be identified and re-elicited, rather than waiting for the retrained model’s accuracy to recover on the new distribution.
The explicit model is not perfect. Its edges can be wrong. Its parameters can be miscalibrated. Its scope can be too narrow. But it is wrong in specific, correctable ways, and that is the property that makes it appropriate for decisions that matter.