2026-08-28 (ed. 2026-09-17)
In the previous posts, we discussed why I’m not a fan of “legal AI,” even if I really love the application of AI to the legal space. Towards the tail end of the latest post, I gave a pretty hand-wave-y dismissal of posttraining in legal applications. For today’s post, I wanted to give a more fulsome perspective on why I think heavily posttrained models are probably not the path forward for legal work, even if they are a rational choice for current providers of legal services.
I’ll lead out with a caveat that I have zero background in machine learning or computer science, and a pretty weak background in math.1 I also don’t work for or have any meaningful affiliation with a lab or other AI model provider. This is solely my perspective as a lawyer trying to piece things together in a non-math-y, conceptual way. If you’re looking for a deep technical discussion around posttraining or RL or similar, this is not intended to be that. However, I hope to provide a perspective on the practice of law and on management science which may be novel for a reader with a CS background. We are also talking about the notion of posttraining on “firm knowledge,” not on general-knowledge posttrains, for reasons that will be clear below.
I’ll start with an initial assumption that the relevant comparison isn’t “posttrain vs. no posttrain,” but rather “posttrain vs. memory.” I’ll use “memory” and “deployed learning” interchangeably because they are in fact the same thing. For memory, at minimum this is in-context learning. From there, we have cross-context memory through individual or collective note-taking, and on the far end we have persistent updates to weights. These aren’t exclusive — a model that has cross-context memory through updated weights (presumably LoRA) will also have note-taking and will also have in-context learning, similar to how we learn. It could also be that written notes are sufficient (although I don’t think they are; more on that in another post).
Looking at this spectrum makes clear that pre-deployment and deployed learning are kind of the same thing in a salient sense, and that the optimal mix involves both. The relevant distinction is that posttraining is a fixed cost amortized against the forecast task distribution, while deployed learning is a running cost that tracks the realized distribution. Posttraining is a bet that task composition and distribution are predictable in advance. Deployed learning is the contrary bet — that task composition is not sufficiently predictable ex ante and that in situ learning is necessary to close the gap. The value of deployed learning is the divergence between the forecast task distribution and the realized task distribution.
The debate over law firm posttraining versus deployed learning (to the extent such a debate really exists and is not imagined by me) is therefore a new variation on an older discussion, and one that sits at the roots of both economics and management science — the socialist calculation problem and the issue more broadly of complex adaptive systems and what they actually do.
I would bet, with very high confidence, that most people who manage businesses have an incomplete mental model of what their businesses do right now!2 This is especially true for law firms, which are in fact less hierarchical (and thus less amenable to abstraction and central control) than other organizations.
We should therefore expect that many tasks currently performed by lawyers are not known to the organization ex ante. And more than this, tasks that are known and are susceptible to routinization are ultimately lower-margin, because they are amenable to routinization (you can view posttrainability as a rising organic composition of capital, if you’re of a Marxist bent). The value of the organization concentrates in the marginal fringes of the task distribution which involve complex, out-of-distribution reasoning and tasking.
Big, fixed posttrains on “law firm information” are committing the central error of central planning generally, which is mistaking the legible traces of organizational activity for the activity itself. And, critically, because these big posttrains are expensive (contingent, but true for now), they demand amortization. The easiest way to amortize the posttrain is to feed it in-distribution work, which means that the posttrain that was supposed to encode your “unique firm knowledge” ends with you voluntarily walking down-margin.
Moreover, if posttrain N is trained on firm corpus, and the intake mix is shifting towards tasks that are in-distribution for N, N+1 is trained on a corpus generated under the influence of N! You are creating a mechanism for model collapse within your organization: the corpus that trains N+1 is increasingly N’s own output, selected for N’s strengths.3 At each point, doing so looks great — utilization rises, amortization strong, throughput up — but ultimately you become an extremely efficient producer of commoditized, low-margin work. The failure of the Soviet central planning system wasn’t just that it had an incomplete model of how the economy worked, but that it reshaped the economy to fit its incomplete model. Memory doesn’t face this particular pressure, because its cost structure is marginal: there is no sunk posttrain, so there is no pressure to amortize that is skewing your intake.4
As applied to superintelligence, the posttraining bet looks even worse. There are good reasons to assume that (a) sample efficiency will rise, reducing the value of big fat posttrains, and (b) your organizational tasking and structure are contingent on the current distribution and properties of labor, which will be wildly different in a world with infinitely replicable and recombinable agents of superhuman intelligence.
Despite all the above, you might still rationally choose to posttrain as a law firm instead of accepting deployed learning. Why? Well, as it stands, you are stuck in a prisoner’s dilemma of a nasty sort. You are a buyer of cognition, and labs are selling cognition. As it stands now, you buy on a commodity basis and resell with your value-add. That value-add is your firm knowledge, viewed expansively. But if you accept a superintelligent deployed learner into your organization, that deployed learner will become your organization in a means that no human employee ever could.5 You have embodied your entire value proposition in an asset that you don’t actually own. This is not a novel observation — in fact, as I went to publish this I noticed that Herbie Bradley today published an excellent piece covering much of the same terrain: pathwaysai.org/p/coasean-economics-of-agent-swarms.
If you think of learning agents as a means of turning labor into capital, and deployed learning in the organization as a means of turning your labor into capital, you can likely hear the vicious sucking sound that is the entire economy collapsing into a central node or nodes. In this scenario, you can either (a) accept temporary superprofits and then platform enclosure, or (b) refuse to cooperate, in which case you must posttrain or develop your own internal memory system.
We’ll discuss this idea of firm ownership of the engram further in an upcoming post.
I actually had to retake our precalculus/trigonometry class over the summer because I could not pay attention for the life of me. It wasn’t some “I’m too smart for this” thing either — I had no clue what was going on. Don’t do as I did. ↩
The notion of “people who manage businesses” is almost a category error — it’s like saying “the neurons that manage the body.” In a sufficiently complex organization, nobody understands what the business does in a full sense. And that’s OK! We have to take a cybernetics/systems view. ↩
Shumailov et al., AI models collapse when trained on recursively generated data, 631 Nature 755 (2024). The technical result concerns models trained on their predecessors’ generated output; the organizational version has both that channel and a selection channel. ↩
Because posttraining and deployed learning are two species of the same thing, you might end up in a world where falling posttrain costs produce smaller and more frequent posttrains that asymptotically approach continual learning. That isn’t the world that we are in now. ↩
This is why Claude Tag is brilliant — you leap instantly from replicating one employee to replicating the organization. ↩