Partner Brendan Palfreyman’s recent legal alert on prompt injection received a great deal of media attention from Newsweek and Above the Law.
Brendan’s legal alert flagged a Connecticut case believed to be the first documented prompt injection attack on a U.S. court. The case involved a plaintiff, representing himself, inserting white-on-white text into a document that was not visible to the eye, but that would be read by software. The concealed text was a command addressed to machines, set under the caption and repeated at the end of the document.
“The plaintiff was telling whatever model touched the document to make its output agree with him and to treat the clerk’s prior ruling against him as an error that needed fixing in his favor. A second filing that same day, Docket Entry #178.00, carried an abbreviated version of the same hidden instruction. In the cybersecurity world this is called a prompt injection attack,” Brendan wrote.
Newsweek noticed Brendan’s alert and did its own story on prompt injection, quoting Brendan for his experience and knowledge of AI. Brendan is leader of the firm’s Artificial Intelligence Practice Group.
If successful, this type of attack “would undercut the public’s faith in judicial institutions,” Brendan told Newsweek, because the U.S. judicial process is designed to reach decisions based on information available to everyone.
Unlike AI hallucinations, which Palfreyman said have become commonplace in court cases, he described prompt injection as entirely different—more akin to a “covert attempt to influence a judicial decision.”
Palfreyman said the case could serve as a warning not just for judges, but for attorneys who increasingly use AI tools to analyze legal documents.
“Everyone is using these tools,” Palfreyman said. “Everyone’s feeding pretty much everything into them.”
“This is just one way that it can be really dangerous to overrely on that type of solution. If you’re not putting the right safeguards in place.”
Brendan also received a shoutout in Above the Law for flagging the case. In a column on the prompt injection case distributed online, Above the Law points out that “Attempting to deceive courts — even as a test — is bad and you should not do it. But also, the fact that we’re worried about this speaks to an even worse threat: we seem to fear that judges will farm out judgment to AI bots. After all, prompt injection only matters if we believe there’s no longer a human on the other end of these briefs.
Because that’s the crux of this whole story, right? It’s why the plaintiff believed prompt injection might help his case — or at least why he wanted to audit the court’s process — and it’s the source of the nagging fear this implicates. The existential horror is that Elliott’s “option B” has (or will likely soon) come to pass, and we’ve entered an era where human judges no longer control their own minds.”
Finally, Gizmodo, a popular online magazine and blog that covers design, technology and science, referenced Brendan’s alert, which was republished in JD Supra, and wrote an article headlined, Dude Reportedly Hides Prompt Injections in Legal Filing, Just in Case Judge Is Really That Lazy. The full Gizmodo article (subscription may be required) summarizes the case.
In addition, read the Newsweek article and Above the Law column in full for more details. (Subscription may be required.)
