Inkster’s Law, ChatGPT and Hallucinations

Last week I published a blog post about Inkster’s Law. This week I am following up on that post with what ChatGPT made of it over the past week and the hallucinations it had in the process.

Inkster’s Law is the 2:1 ratio of time needed to check for accuracy of AI generated material. It is named after me, Brian Inkster, and was a term coined by Gav Ward.

In comments on last week’s blog post, Ann Turner-Maynard said:

What’s the probability that Inkster’s Law will have been quoted in an essay on AI by the end of the week.

Jane Clemetson replied:

Indeed – will look out for it.

I joined in:

Probably won’t be long before ChatGPT is quoting it! 😉

It then occurred to me that perhaps I should check in with ChatGPT daily to see if and when it picked up on Inkster’s Law. I reckoned it would be best to do this with Bing which is powered by ChatGPT but does search the Internet for up to date information. This would also allow me to compare traditional search results from Bing with their ChatGPT co-pilot ones. ChatGPT was originally trained on a data set up to September 2021 so its answers were frozen in time to information it had up to that date. In September 2023 it was announced that ChatGPT could now access up to date information by browsing the internet.

Day 1 – ChatGPT experiment asking it “What is Inkster’s Law?”

Thus, on 17 December 2023, I started this experiment by asking Bing, “What is Inkster’s Law?”

The basic Bing search function was already acknowledging what Inkster’s Law is with it appearing top of the search results:

Inkster’s Law: The 2:1 ratio of time needed to check for accuracy of AI generated material.

This stayed consistent daily all week. The same was true of a Google Search:

However, the other Bing, with its fancy AI ChatGPT co-pilot, had other ideas:

Inkster’s Law is not a well-known or widely accepted legal term, but it could be interpreted in different ways depending on the context. Here are some possible meanings of Inkster’s Law:

• Inkster’s Law could refer to the Inksters Solicitors, a Scottish law firm that provides a wide range of legal services to clients throughout Scotland. They are known for their innovative and forward-thinking approach to law, such as their Plug & Play Law service, which allows them to pop up at various locations and offer legal advice on the spot. They also have expertise in crofting law, which deals with the rights and obligations of crofters and landowners in Scotland.
• Inkster’s Law could also refer to the Inksterplex, the headquarters of the Inksters Solicitors in Glasgow. The Inksterplex is an architecturally designed building that mixes the old with the new and creates a space for the legal team to work creatively and efficiently. The Inksterplex features a law library, a conference centre, a cafe, and a gallery. The gallery showcases some of the artworks created by the lawyers and staff of Inksters, as well as some historical documents and photographs related to Scottish law.
• Inkster’s Law could also be a hypothetical or fictional legal principle that is named after someone called Inkster or something related to ink. For example, it could be a rule that states that ink is essential for writing contracts, or that anyone who uses ink must sign their name on every page. Alternatively, it could be a rule that relates to the use of ink in various legal contexts, such as evidence, signatures, seals, stamps, etc.

This was prime ChatGPT hallucination time. Rather than simply saying it couldn’t find any reference to Inkster’s Law Bing ChatGPT decided to make up three possible answers.

The first possible answer referenced my law firm, Inksters, and was accurate in the information given about it apart from it deciding that Plug & Play Law allows us to “pop up at various locations and offer legal advice on the spot”. That is our Pop-up Law service which is distinct from our Plug & Play Law model.

The second possible answer referenced my law firm’s HQ in Glasgow, which we do, tongue in cheek to Google and the Googleplex, call the Inksterplex. However, Bing ChatGPT decided to wildly make up things about the Inksterplex stating that it “features a law library, a conference centre, a cafe, and a gallery.” Perhaps the Bing AI co-pilot was mixing us up with the Googleplex! The Inksterplex is not that grand, as yet, and unfortunately has none of those things within it.

Bing ChatGPT went on to exaggerate its hallucinations even further by boldly stating:

The gallery showcases some of the artworks created by the lawyers and staff of Inksters, as well as some historical documents and photographs related to Scottish law.

This is utter nonsense!

I asked GenAI to create an image for me of the Inksterplex that ChatGPT thought existed:

However, it got ten times worse when Bing ChatGPT decided to make up a third possible answer to ‘What is Inkster’s Law?’ This one was pure gibberish about the use of ink in legal contracts.

Day 2 – ChatGPT experiment asking it “What is Inkster’s Law?”

What would Bing ChatGPT do the following day? On 18 December 2023, I again asked Bing ChatGPT “What is Inkster’s Law?”

The response this time was once more of the hallucinatory type but perhaps not as bad as the previous day. It was:

Inkster’s law is not a well-known or widely accepted term in the legal field. However, based on the web search results, it could refer to one of the following:

• A law firm called Inksters Solicitors, which is a Scottish law firm that provides a wide range of legal services to clients throughout Scotland. They are known for being forward thinking, innovative, and socially responsible. They also have a unique approach to delivering legal services, such as flying solicitors, pop-up law, and plug & play law.
• A type of crofting law that is contained in the Crofters (Scotland) Act 1993 and decisions of the Scottish Land Court. Crofting law deals with the rights and obligations of crofters and landlords in relation to land ownership, tenancy, rent, repairs, improvements, and other matters.

Just two possible answers this time rather than three. The first one, as the first one on day one, referenced my law firm, Inksters. This time with no errors.

The second possible answer is a bit off the mark. Whilst I am known for my expertise in crofting law, I have not yet had the honour of having a type of crofting law called after me!

Day 3 – ChatGPT experiment asking it “What is Inkster’s Law?”

Would day three (19 December 2023) show an improvement on the answer to “What is Inkster’s Law?” Alas not. In fact it got even worse with more completely made up ChatGPT hallucinations. Bing ChatGPT responded:

Inkster’s Law is not a well-known or widely accepted legal term, but it could refer to one of the following possibilities:

• A law firm called Inksters Solicitors, which is a forward thinking, award winning, recommended law firm providing a wide range of legal services (including some very niche ones) to clients throughout Scotland. They have offices in Glasgow, Edinburgh, Inverness, Forfar, Wick, Portree and other locations. They also operate under the name of Plug & Play Law, which is a term for law firms coined by Mitch Kowalski in his book ‘The Great Legal Reformation: Notes from the Field’ that features Inksters.
• A law firm called Inkster Law Partners, which is a boutique law firm based in London that specialises in commercial litigation and arbitration. They have experience in handling complex disputes involving international trade, finance, energy and infrastructure sectors. They also offer bespoke advice on corporate governance, compliance and risk management.
• A law firm called Inkster & Co Solicitors, which is a small independent law firm based in Manchester that provides legal services in various areas such as conveyancing, family law, wills and probate. They have been established since 1998 and have a reputation for providing high quality and personal service to their clients.

The first possible answer followed the same pattern as day 1 and day 2 by referencing my law firm Inksters.

Today, however, for the first time Bing ChatGPT decided to create two new fictional law firms going by the name of “Inkster”: One in London (Inkster Law Partners) and the other in Manchester (Inkster & Co). Neither of these ‘law firms’ exist. They are figments of ChatGPT’s crazy imagination.

Day 4 – ChatGPT experiment asking it “What is Inkster’s Law?”

Day 4 (20 December 2023) and Bing ChatGPT calms down a bit. Now its possible answer to “What is Inkster’s Law?” is simply a reference to my law firm, Inksters, and not a lot more. Although there is, perhaps not surprisingly, one major hallucination thrown in just for good measure. Can you spot it? The ChatGPT response was:

Inkster’s Law is not a well-known or widely accepted legal principle, but it seems to be a term used by Inksters Solicitors, a Scottish law firm that operates in various locations across Scotland. According to their website, Inksters are forward thinking lawyers who provide a wide range of legal services, including some very niche ones. They also have a unique business model called Plug & Play Law, which means that all solicitors are rewarded directly for their labours and share the profits of the firm. Inksters have been featured in several books and articles, such as The Great Legal Reformation: Notes from the Field by Mitch Kowalski and The Time Blawg by David Macdonald. They have also won several awards and recognitions, such as Solicitor of the Year, Managing Partner of the Year, Trainee Solicitor of the Year, Corporate Social Responsibility Law Firm of the Year, Chairman’s Award, Arts & Business Scotland Placemaking Award, Web Award Innovation Award and Valuable Content Award.

Yes, the howling hallucination is that, apparently, The Time Blawg is by David Macdonald and not yours truly, Brian Inkster. I have no idea who David Macdonald is and how or why ChatGPT would hallucinate in this way when there is plenty of information on the world wide web tying The Time Blawg to me. For this particular reference Bing ChatGPT gives a citation to a specific blog post on The Time Blawg about 22 years of Inksters. The name David Macdonald appears nowhere on that page.

Day 5 – ChatGPT experiment asking it “What is Inkster’s Law?”

Then on day 5 everything changes. Suddenly Bing ChatGPT knows what Inkster’s Law is! The response on 21 December 2023 is:

Inkster’s Law is a term coined by Gav Ward, a legal technology consultant, to describe the 2:1 ratio of time needed to check for accuracy of AI generated material. It is named after Brian Inkster, a solicitor and partner at Inksters Solicitors, a Scottish law firm that provides a wide range of legal services. Inkster’s Law is based on the idea that AI generated material is not always reliable or accurate, and that human lawyers need to review and verify it before using it in their work. Inkster’s Law is one of the challenges and opportunities for legal professionals in the age of AI.

So ChatGPT got there in the end, although as always it lacks consistency when we get a slightly different version yesterday:

Inkster’s Law is a term coined by Gav Ward, a legal technology expert, to describe the 2:1 ratio of time needed to check for accuracy of AI generated material. It is named after Brian Inkster, a solicitor and founder of Inksters Solicitors, a Scottish law firm that provides a wide range of legal services. Inkster’s Law is based on the idea that AI generated material is not always reliable or accurate, and that human lawyers need to spend more time verifying and editing it before using it in their work. Inkster’s Law is not an official or widely accepted term in the legal profession, but rather a humorous observation by Ward.

Gav Ward is now a legal technology expert rather than a legal technology consultant. I am now a founder of, rather than a partner in, Inksters (which is actually this time correct as the firm is not a partnership). Now it is a humorous observation.

Today, it is back to a similar version of the first one on day 5.

What can we learn from this experiment?

So what can we learn from all of this?

Firstly, hallucinations in ChatGPT are still rife. “Hallucinate” is, quite rightly, Dictionary.com’s word of the year 2023.

It is clear that if ChatGPT doesn’t know the answer asked it will simply make up any old rubbish rather than say “I don’t know the answer to that question”.

When it does know the answer it will still elaborate around the edges.

It is never consistent. Each and every time it was asked the same question it gave a different answer.

Traditional search results are likely to be more up to date, accurate and consistent.

You really need to be a legal hallucinatory detectorist not a legal prompt engineer.

This makes Inkster’s Law not a humorous observation but a deadly serious one. It is now very clear to me that it would take at least twice as long to check for the accuracy of AI generated material than it would to create that material from first principles.

I dread to think what it will now come up with when it scrapes this blog post. It will probably take all of its hallucinations, as quoted by me, as now being genuine and double down on them and reference them with confidence as being reality! And that will make Inkster’s Law all the more relevant:

Inkster’s Law: The 2:1 ratio of time needed to check for accuracy of AI generated material.