what i don't like about "legal ai"

In my previous post, I discussed why I’m skeptical of the idea of “legal judgment.” This idea is at the core of why I’m skeptical of “Legal AI” as a specific product class, and why I hope other lawyers will be too.

I’ve tried to expand a little below, because that post was a statement of beliefs without much reasoning behind it.

We previously discussed that AI does seem to be capable of “legal judgment” as necessary for at least the substantial majority of legal work. In that post I used the phrases “legal judgment” and “judgment” somewhat interchangeably. In addition to me blasting these posts out by hand with no copy-editing, I use the two interchangeably because I think that they are interchangeable. This idea of legal-specific reasoning or judgment is essentially a category error — what you are generally describing is general reasoning implicated in the practice of law. The fact that you wear a wig while you do it doesn’t really change the situation. So “legal reasoning” becomes a contingent description of who is doing the reasoning — essentially an artifact of human specialization rather than a property of the thought itself.

Firstly, I am immediately very skeptical of any company that intends to challenge big models with smaller, specialized models in the field of general reasoning (which, as discussed above, includes lawyering generally). There are some steelman cases for this business model, which I’ll discuss below, but the overwhelming rule so far has been that the frontier models get better as they get bigger, and they get better in weird, emergent ways. Importantly, and as noted before, they are now way better at legal reasoning and analysis than they were even six months ago, and with near-zero investment in legal specifically (at least to my knowledge).¹

When “legal AI” is offered by companies that are not model providers, it’s difficult to identify the value-add. Instead of paying one company for inference, you are paying a company to re-sell inference to you. It is impossible, from a business perspective, for that company to give you a cheaper price per token. They can give you better value on a net basis in roughly three ways, ranked in order of how interesting I think they are:

1. Industry-specific compliance needs: Law firms have lots of compliance requirements, as you would expect for entities that are dealing with sensitive information belonging to other companies. It’s reasonable to expect a layer of compliance services (auditing, monitoring of lawyer interactions with the system, enforcing matter-level permissions) will best be served by an intermediary rather than the labs. This is a perfectly good business model, but the benchmark is legal IT spend rather than legal labor spend, and these are at least an OOM apart.

2. Better routing: The argument here is that legal AI platforms will intermediate between model providers and users, routing queries to the best models for the task to reduce total cost. This is an appealing business model for multiple reasons. The first is that model capabilities are still jagged, and critically heterogenous between models. As long as that is true, there’s space for trading even between price-competitive models. Labs can’t do this because they currently only directly capture revenue via inference.²

Model routing is sometimes presented by reference to the principal-agent problem that arises from the fact that labs want you to use more tokens and you want to use less. But principal-agent problems don’t get solved by introducing another agent. A legal AI wrapper that charges on a per-seat basis has an incentive to reduce token spend, but it also has an incentive to do so by routing you to a dumber, cheaper model. That slippage in quality is the agency cost recurring in a new form. In fact, this is a more insidious agency cost than the labs, because the lab agency cost is transparent (you spent a bazillion dollars on tokens), while wrapper agency costs are discernible only by counterfactual (here’s how the work would have looked if you weren’t routed to a crappy model). If the wrapper gives the user a model selector, then (a) routing gains diminish or vanish, and (b) because of the flat price, users are incentivized to keep that model selector pegged to the SOTA model. If the platform switches to usage-based pricing, routing gains can only reenter as a service (you pay a premium for us to save you money on the back end), which is a weaker value proposition.

It shouldn’t be surprising that agency costs aren’t eliminated by adding agents, even if it alters their form. What we know reduces agency costs are monitoring and bonding. Monitoring is cheap in verifiable domains, but expensive in legal. And bonding won’t work, because router slippage is counterfactual (as noted above). So the same property that makes legal work really interesting and valuable as a long-horizon RL task also makes third-party routing structurally unviable. Routing is still a good task — law firms route work between attorneys, and better routing drives better outcomes — but it’s not necessarily a good business.

3. Better legal output: This is really the prize pig of all “legal AI” — the idea that it will improve output quality over competition (however defined). At one point, the idea was that there would be legal-specific pretrains. Those failed, for predictable Bitter-Lesson-shaped reasons. Now, the flavor of the month is posttrains. Posttraining is a legitimately valuable thing, but we should be clearer about what posttraining can and can’t do.

Posttraining of the kind that wrappers do (fine-tuning on domain data) is mostly eliciting and steering latent capabilities in the base model, not developing new reasoning.³ This is why the pitch recently has been posttraining on firm or domain-specific knowledge, rather than general reasoning capabilities. There are a lot of issues with the idea that domain-specific posttraining is going to meaningfully displace general models. The critical point is what we’ve been discussing — legal reasoning is general reasoning. The primary value driver here is the reasoning capability, not its capability to write in a particular memorandum form.⁴⁵ We will talk more about posttrains and the path forward in an upcoming post.

As an aside, ideally we would want to amortize institutional knowledge over time, but posttrains are depreciating assets! It’s not fair to say “we posttrained Crappybot and now we get great results for Crappybot prices” without pricing the posttrain into the amortized cost. The only scenario that makes sense is for narrow “worker ant” models where you are relatively indifferent to intelligence but really want some posttrained capability — but again, skeptical that this category of work is a huge value driver.⁶ Maybe I’m wrong? Please reach out if you disagree.

To be fair, they have spent a lot of time on coding, and it turns out that coding and lawyering are actually very similar (and the areas where they aren’t similar are where all models right now are weak — i.e., the marginal economics of legal drafting are different than code, which is why AI makes big changes to contracts sometimes, and it doesn’t have a great theory of mind or process, which is the root of a host of problems. I’m confident that these are eminently fixable problems.). ↩
As we will discuss more in the future, this is not a permanent state of affairs. ↩
This is a high-level statement and I’m writing quickly, so I’m not going to get into the weeds on RLVR and capabilities until I understand better. ↩
Here, as with everywhere, proxies can be deceptive. Because the true reasoning capability of a model (legal or otherwise) is resistant to quantification, we have to approach it by proxies. Benchmarks are cheap proxies for expensive tasks, and where the gap between proxy and task is widest (as in legal, where verification costs are comparatively high) the Goodhart pressure is the strongest. The gap between proxy performance and true task performance is what people colloquially call “big model smell” — there is something about a big, general model that is greater than the sum of its individual benchmark scores. So we should be skeptical of the notion that posttraining on a non-frontier model, even if it produces good results on a bench, is producing a model that is good to use or interact with for legal work. ↩
And note that we see Goodharting in regular lawyers/law firms/consumers of legal services as well, precisely because the same verification gap exists. ↩
The world where posttrains make sense is one where the pace of releases slows or stops entirely and there’s no internal deployment gap, such as a true pause. In that world, the depreciation curve of your posttrain would (barring technical advances) look much better. ↩

What I Don’t Like About “Legal AI” Specifically