Inkster’s Law

Inkster’s Law is a term coined, this past week, on LinkedIn by Gav Ward.

It all started with a LinkedIn post by Dan Hunter. Dan starts the post with:

Ok, here is my first pass at full syllabus for a comprehensive Legal GenAI subject…

(What have I missed? Will post explanations, teaching notes, and references for each seminar shortly.)

He then goes on to give us details of 11 seminars on Legal Gen AI. Two of these cover prompt engineering. I couldn’t resist telling Dan what he had missed:

If you are giving two full seminars to Prompt Engineering you need at least four to being a Legal Hallucinatory Detectorist.

Gav Ward then chimed in with:

The 2:1 ratio of time needed to check for accuracy of AI generated material – Inkster’s Law?

To which I retorted:

I hope a Suss Jnr is quoting Inkster’s Law in years to come 🤣

So, there we have it, Inkster’s Law: The 2:1 ratio of time needed to check for accuracy of AI generated material.

Reactions on Inkster’s Law

In addition to the comments made on this blog post itself (see below), on LinkedIn the following comments have been made:-

Sally Inkster – Brand You (will work with you to create a Memorable Personal Brand / Bespoke Brand To You):

How could I not comment on this! A family name law!

Me:

Indeed Sally!

-+-+-+-+-+-

Rob Marrs (I work out how lawyers become lawyers… and help make sure the profession is diverse):

Welcome to the club!

A judge once said ‘’ The Marrs Test’’ was: if a managing partner could walk to the front door of the office and look his or her staff in the eye having accidentally emailed pay data to all staff earlier that afternoon. This was a test i’d set out in an article and she thought a good one regarding gender pay

I have to say I was disappointed this didn’t take off… could have got on the global speaking shill!

Gav Ward (Helping Ambitious Lawyers & Law Firms Prosper w/ AI-Enhanced Marketing & Legal Expertise | Projects Director @ Leading Agency MLT Digital | Ex-Lawyer, Consultant, Connector & Blogger @ WardblawG, Five Fantastic Lawyers):

Rob Marrs will bear that one in mind, Rob. There’s also Millar’s Law after Douglas Millar – more of a rule than a maxim – which is “Don’t do that which you can’t undo”. A general rule we’ve had at MLT for years that we apply to backing up and restoring parts of websites.

Me:

Clearly a good Club to be in! And the Marrs Test deserves wider recognition.

-+-+-+-+-+-

Gav Ward:

Here it is in a development / coding sense – slightly higher ratio.

 

 

 

 

 

 

 

Me:

Inkster’s Law may change in legal based on testing when lawyers actually really start using it 😉

 

 

 

 

 

 

 

 

 

 

 

-+-+-+-+-+-

Nancy Myrland (LinkedIn™️ Coach For Lawyers | Legal Marketing & Business Development | Content, Social Media, Podcasting, Video & Virtual Presentation Consultant | Individual & Group | Speaker, Trainer & Advisor for Lawyers & Law Firms):

This is funny, Brian and Gav. Now I’m kind of jealous and want a Nancy’s Law!

Me:

You might find that Gav Ward can coin one for you in time 🙂

Gav Ward:

This might well be my new thing 😉 Although I would note it’s the first time it’s happened – it really was a one of a kind connection between Brian’s insight during the week and the many comments from others I’ve seen about how lawyers reviewing AI generated material (when not properly prompted) seems to add onto their billable hours. Any profound statements you’ve made before, Nancy, or excellent blog posts I can look at? This plus identifying AI bullying screenshots – “look everyone, ChatGPT or Gemini said this or that” – think I’m going to line up a blog post on that. #leavechatgptalone

-+-+-+-+-+-

Kyle Bahr (Legal Ops & Tech Innovator | Former Fortune 200 Legal Operations, Commercial Disputes & Contracts Attorney, Federal Law Clerk, and International Law Firm Attorney & Paralegal):

We may have something here, Brian and Gav, although I’d like to see some qualifying words sprinkled in like “proportional”, “expertise”, and perhaps “b.s. detector” because it ain’t always 2:1!

Depending on how one uses these tools, and for what purpose, a human can know in an instant whether it’s an hallucination, or actually pretty dang spot on!

I remember back in January 2023 the first time I tried getting case law citations from ChatGPT. I wanted some Georgia breach of commercial contract cases. It gave me case names like “Lee v. Lee,” with a nice looking bluebook citation to the Ga. reporter. I was immediately skeptical, because that’s a typical family law case name, and a very *atypical* breach of contract caption. Sure enough, tossing them into Lexis, the bluebook citations were completely fabricated. I was into it.

Turns out, though, that you *can* get accurate cites for main cases (Iqbal and Twombly, for example) from LLMs that lack Internet search capabilities. Toss in Internet searchability (e.g., Bing Chat/Copilot, ChatGPT-4) or document upload (e.g., briefs, datasets, etc.), and reliability increases because you can actually see the source material.

Finally, per Nicola Shaver, 2024 will herald accurate Legal LLMs.

Me:

Perhaps wishful thinking on your part and Nicola Shaver’s part. We can revisit the question of accurate LLMs this time next year!

Kyle Bahr:

You got it, Brian, it’s a date!

By the way, what’s our definition of hallucination? At this point, I think of it as a “convincing falsehood.” For example, a case citation that looks perfectly formatted, but doesn’t actually exist, or a description of a case that seems accurate on its face, but upon digging in is wholly made up or wrong in an important respect.

Which of course humans do, either intentionally or unwittingly, more often than we should. We’re kind of used to it with people, and learn to navigate the murky waters of trustworthiness. Maybe we apply those same skills to Gen AI?

Me:

“Hallucination” is a bad term coined probably to make bad ChatGPT output look batter than it is. Microsoft’s version “usefully wrong” is along the same lines. Why do we not just call it what it is: a “mistake” or an “error”. Or as one Judge put it “gibberish“.

Some of the “falsehoods” that GenAI produces are not “convincing” but blatantly wrong. I appreciate one person’s “blatantly wrong” may be another person’s “convincing falsehood” depending upon the knowledge held on the output provided.

In law most lawyers would not create “convincing falsehoods” when producing case law to cite in a court case. They would do their homework properly and diligently and obtain copies of the actual cases of relevance and read each one in its entirety (AI summarisations have no place in that process in my opinion).

There should not be “murky waters of trustworthiness” where lawyers and the law are concerned. Unfortunately, AI appears to be muddying those once clear waters!

-+-+-+-+-+-

Jane Clemetson (IP & business affairs lawyer working with media and creative businesses | commercial, tech and data protection | European Woman of Legal Tech | Bit of a #girlyswot | Resourceful | FRSA):

Definitely questions on matters of fact – although I think a 2:1 ratio may be a bit optimistic tbh

Me:

Inkster’s Law is flexible 🙂 Will always take you at least twice as long to check but could and often will be much longer 😉

Jane Clemetson:

Shrieks!!

-+-+-+-+-+-

Mitch Kowalski (European/Canadian, Head of Legal and Legal Operations Advisor – Eligible to work in Canada/US/Europe without sponsorship):

Beautiful.

-+-+-+-+-+-

Amit Sharma (High Speed Solicitor acting for u/hnw clients who value their time and want 100% attention on their prime deals):

Provenance has entered that chat.

-+-+-+-+-+-

Stephen Gold (Principal at Stephen Gold Consulting: Helping Lawyers Build Great Firms):

Brian, Gav, funny. May I propose Inkster’s Second Law, or possibly the First Exception to Inkster’s Law, that, in any case where the 2:1 ratio proves insufficient and results in error, it will occur in the one matter you absolutely can’t afford to go wrong.

Me:

The odds of that happening are probably high!

… but that is Sod’s Law!

Stephen Gold:

Good point!