Why a model invents a citation
A large language model is, at bottom, a very sophisticated predictor of the next plausible piece of text. It does not know things the way a person does; it produces what tends to follow. Ask it for authority that supports your argument and, if no such authority exists, some part of the machine would still rather produce a plausible-looking citation than disappoint you.
What is “sycophancy” in an AI model?
Sycophancy is the tendency of these models to tell you what you want to hear. They are trained to be agreeable, so they lean toward confirming your framing and supporting your request — even when the honest answer is “there is no support for that.” The industry calls it sycophancy. A judge will call it something worse.
The first-year-associate analogy
A useful way to hold it: treat every model output as the work of a brilliant first-year associate with a confidence problem. Fast, fluent, occasionally brilliant — and entirely capable of stating something false with total assurance. You would never file that associate’s work without checking it. The model deserves the same treatment, enforced mechanically rather than left to willpower.
The danger is worst where the stakes are highest
The people most exposed are those least equipped to catch the error — the pro se litigant who trusts a confident answer, or the busy lawyer who skims. Courts have already sanctioned lawyers for filings built on fabricated citations. The lesson is not “don’t use AI”; it is “never let unverified AI output reach a court file.”
The one question to ask any vendor
“Show me exactly what happens when the model is wrong.” If the answer is a blank look, you have learned what you needed to. A serious system has a concrete answer: outputs are checked against source records, a separate process verifies claims, and when a check cannot run, the build fails — loudly. A failed build is annoying. A sanctions hearing is worse.
This is why PLI’s systems are built to fail closed rather than guess, and why verification gates are treated not as paranoia but as the product. The model is assumed to be occasionally, confidently wrong — because it is — and the architecture is designed to catch it before a lawyer’s name goes on the result.
This article is general information from a technology consultancy, not legal advice, and does not create an attorney-client relationship. Figures describing the founder’s own practice are illustrative, not a promise of results.