Legal Geek 2023: The one with Withnail and GenAI
Legal Geek 2023 was the one with Withnail and GenAI.
Legal Geek 2023: Withnail
No big surprise that GenAI was being overcooked at Legal Geek 2023, but what was Withnail doing there?
Withnail’s alter ego, Richard E. Grant, was giving a keynote discussion on ‘grief and its impact’. Perhaps to prepare the GenAI evangelists for the inevitable? 😉
But, being serious, Richard E. Grant was at Legal Geek 2023 to speak about the impact of grief following the death of his wife, Joan Washington, in 2021. Nothing to do with Legal Tech. But perhaps a welcome interlude to the relentless and generally rather pointless talks on GenAI.
We’ve all either experienced grief or will do at some point in our lives. I’ve had my fair share of grief already and seen more than I would care for in the past year alone. Thus, Withnail held my attention more than the potential experiences of GenAI that were covered in other talks.
As this is a blog about the past, present and future practice of law I’m not going to dwell long on Richard E. Grant’s talk other to recommend that you buy and read his book ‘A Pocketful of Happiness‘. I will be doing so. The book title is what Richard’s late wife, Joan, challenged him to find in every day after she was gone.
Legal Geek 2023: GenAI
Now we can move from Withnail to GenAI. Legal Geek 2023 was packed with it. I counted 14 talks on day two alone that were specifically about AI. That does not include the countless product pitches that day, some of which no doubt included AI too.
I didn’t go to all of those Legal Geek 2023 talks. My sanity had to be retained! I was interested primarily in the holy grail of GenAI in legal: finding the use cases. A topic I previously blogged about: LLMs in Law: Hype v Magic
I sat through Legal Geek 2023 talks entitled:
- Step into the light – mastering the force of AI in legal practice
- Cutting through the GenAI noise: how to balance new initiatives without turning down the volume on core legal tech
I came away from both with one note on my reMarkable: “Where are the use cases?”
Surely, I hoped, the next panel session would have all the answers:
- Painkillers, not vitamins: Real use cases for generative AI
Real use cases for generative AI
Unlike the earlier talks, that were only 10 minutes long each, this was a full blown 45 minutes. Surely the panel couldn’t talk for 45 minutes without giving us some use cases, could they?
This Legal Geek 2023 panel discussion was chaired by Michael Grupp (CEO – Bryter). The panel consisted of:
- Elliot White (Head of Innovation & Legal Tech Operations – Addleshaw Goddard)
- Tara Waters (Chief Digital Officer and Head of Ashurst Advance Digital – Ashurst)
- Daniel Ang (Legal Technology Manager – Anglo American)
- Stefanie Briefs (senior Legal Counsel – Bertelsmann)
It promised:
With Generative AI, you can now build any digital solution you want, quickly and easily. So, what next? What are the use cases for Generative AI that solve real problems rather than ‘nice-to-haves’? Which areas are overhyped? Join the session to hear the full range of market perspectives on how legal departments are using AI.
The Legal Geek 2023 workshop room was ram packed for it. Standing room only when I arrived. Clearly a lot of other delegates were in search of the holy grail of GenAI in legal too. They were about to be sorely disappointed. Alternatively, perhaps, mightily relieved depending on your viewpoint!
What was your GenAI moment?
Michael kicked off proceedings by asking this Legal Geek 2023 panel: “What was your GenAI moment?”
Stefanie told us she had exclaimed “wow” when she saw AI manipulated images.
Daniel described the power now at your fingertips. He recalled the old days of IBM Watson. Back then (doesn’t seem that long ago to me!) it was hard to construct a chatbot and conversation. Now it is “mind-blowing”. Daniel had a moment of ecstasy with the response received via GenAI for zero effort. He must be easily satisfied!
Tara referred to the senior partner at her law firm revealing that he had bought a premium licence to ChatGPT and wanted to use it. She said as soon as you see it in action you start to imagine what it can do. Lawyers are asking about it: “what can we do with it?”
Michael referred to redlining has gone and what needs to go into a contract has gone. His response was of the imagining magic kind: The second of the seven deadly sins of Legal Tech predictions.
What Disappointments have you experienced?
Michael then turned the discussion towards disappointments that the Legal Geek 2023 panel had experienced with GenAI.
Not as good as expected
Elliot revealed his disappointment to be things not working out. GenAI was not as good as expected. It was not perfect. Michael asked Eliot for examples of what it couldn’t do and Elliot decided to defer on that. The list is clearly so long that Elliot didn’t know where to begin 😉
Chatbots can suck
Tara referred to chatbots being disappointing. Often giving out garbage. Chatbots often suck. They don’t work out of the box perfectly. A lot goes into getting accurate results. I did a series of chats with legal chatbots before Chat-GPT (BCGPT) and it appears nothing has really changed that much after Chat-GPT (ACGPT): Chatbots
Hallucinations
Daniel told us that hallucinations was a real problem but that there were strategies around that. I have suggested the need for legal hallucinatory detectorists to be as great if not greater than the need for legal prompt engineers.
Expense
The cost of GenAI was also referenced by Daniel. He said that no one was really speaking about that. You need to weigh up the cost with use case benefit.
I noticed just this past week that cost is an issue that analysts predict will result in ‘Overhyped’ generative AI getting a ‘cold shower’ in 2024.
Time and Effort
Stefanie revealed that her disappointment was the amount of work you need to put into it. You need to train/prompt it to not get rubbish out. Tell me about it! I spent at least an hour trying to produce an image for this blog post using DALL-E 3. I wanted GenAI to mimic the image used for much of the PR (including on DVDs and posters) of the film ‘Withnail and I’:
My plan was to replace ‘I’ with a robot. However, despite telling DALL-E 3 in detail where I wanted the robot to sit compared to where Withnail was sitting, and that the robot (not Withnail) had to be holding and reading the newspaper, it did the exact opposite. It took 15 different prompts and the production of 58 images to get the one below, which was only one of two out of those 58 that had Withnail and the robot sitting in the correct positions and the robot reading the newspaper.
Blindly following the data
Was bias at play? Humans should come first? Robots do not read newspapers but humans do? Or am I just crap at prompting?
It appears that GenAI is “just blindly following the data“. It was recently discovered that if you ask DALL-E 3 to produce a clock/watch it will invariably (even when you specifically ask it not to do so) provide an image that has the clock/watch hands at 10 minutes to 2 (otherwise 10 minutes past 10) because almost all of the product images advertising timepieces use that time as it is more visually appealing (i.e. it looks like a smiling face).
Withnail having a drink with a robot was an easier prompt to produce a good image from and I decided to use that one (which I liked better) as the main featured image. And if anyone can tell me how I could have made Richard E. Grant appear in the GenAI images as Withnail (like the Pope in a puffer jacket) then do let me know!
What are the use cases for GenAI in law firms?
Now we came to the part we were all waiting for. Was the search for the holy grail of GenAI in legal over? What are the use cases?
Q&A chatbot
Daniel told us he was experimenting with Q&A chatbot. This was on a particular playbook/contract, say HR. Questions lawyers get you can then use GenAI to replace lawyers having to answer them. This is using Retrieval Augmented Generation (RAG). He pointed out that you do need quality content first. I recalled Parker! You can, of course, have FAQ without a chatbot being involved. Is a chatbot the best solution for this use case? And at the moment Daniel was just experimenting. He had not deployed his chatbot yet. I look forward to having a chat with it when he does 😉
No business value
Tara admitted, very frankly, that she was seeing no business value yet from GenAI. They were, at Ashurst, still testing and experimenting. She referred to summarisation being a use case but added that she had seen good and less good examples of this. They were looking at some vendors who are developing solutions that use GenAI. She concluded that people in law firms saying they are using GenAI are lying to you!
Due diligence
Elliot referred to due diligence being a potential use case. Loads of documents go into a data room. At Addleshaw Goddard they are looking at how they can use an AI system to triage those documents. How do they find information quickly. I wondered whether that was not just e- discovery? Has AI not been used for years already just for this purpose?
Building your own LLM
Elliot thought that building your own LLM seemed like a lot of effort. They were looking at summarisation at Addleshaw Goddard. But it was still early days. You need to think about what the best thing is to use GenAI for because it is expensive.
GenAI: Ideas, Changes and Impact
Michael asked the Legal Geek 2023 panel what (a) ideas are there of what they could build?; (b) changes are there in business and innovation?; and (c) impact have they seen?
Questions from lawyers
Elliot said there were lots of questions from their lawyers. There had been some good uptake. They were freeing up the minds of lawyers by letting them use Chat-GPT. I was a bit confused by this giving his earlier comments but maybe any use by lawyers was all part and parcel of them looking at or experimenting with GenAI?
From Blockchain to GenAI!
Tara told us that she was the front for GenAI at her firm. She did that previously with Blockchain! I was wondering how and why Big Law had the resources to waste on these shiny new toys as soon as they appeared before them. Tara didn’t mention whether she had once been the front for the metaverse at Ashurst! Tara revealed that her lawyers haven’t become coders and are not using no code Apps. She told us no one wants to sit and chat with a chatbot all day (I could appreciate that after only very short periods chatting with chatbots!). Tara told us no one is going to ask a chatbot to take a deep breath before it responds. A reference to a recent report suggesting that was a good prompt to give ChatGPT in order to optimise its output.
Lawyers are too busy
Daniel referred to lawyers being too busy to stop what they are doing. He thinks that legal work will continually increase. Augmented help might be worth looking at – how could it help you.
The Beatles
Stefanie told us there were different types of lawyers. Like different members of the Beatles. I lost the thread of this particular argument. I guess (but may be wrong) that it was something along the lines of some will be interested in GenAI and some won’t be. I’m guessing there will be a lot more of the latter.
Questions and Answers from the Legal Geek 2023 floor on GenAI
It was now time for the Legal Geek 2023 delegates to ask the panel questions. There was, however, only time for two questions.
Question 1 was on fine tuning. We were told that big banks are spending a lot of money on fine tuning. Are law firms considering it?
I had to ask Bing ChatGPT what fine tuning was. I was informed:
Fine-tuning is a technique in which pre-trained models are customised to perform specific tasks or behaviours. It involves taking an existing model that has already been trained and adapting it to a narrower subject or a more focused goal. Fine-tuning lets you get more out of the models available through the API by providing higher quality results than prompting, ability to train on more examples than can fit in a prompt, token savings due to shorter prompts, and lower latency requests.
The process of fine-tuning involves updating pre-trained models with new information or data to help them adapt to specific tasks or domains. During the process of fine-tuning, the model is trained on a specific set of data to customize it to a particular use case.
Elliot said they had considered it but it was too early. He was aware of it but had enough areas to explore before that.
However, after Legal Geek 2023 I discovered, via VentureBeat, that with regard to fine-tuning:
A recent study by Princeton University, Virginia Tech, and IBM Research reveals a concerning downside to this practice. The researchers discovered that fine-tuning LLMs can inadvertently weaken the safety measures designed to prevent the models from generating harmful content, potentially undermining the very goals of fine-tuning the models in the first place.
Question 2 was if we ask “is there a use case?” we struggle with it. If we then find one how long will it take to create and implement a solution?
Tara said that the risk in not testing and experimenting. We need to put something in front of lawyers that is easy to use. For example not needing complex prompts. That is why Ashurst are taking time to look at it.
Elliot suggested that you could template prompts for the lawyers. Will there not be an answer to each of those templates? Do you not therefore simply supply the ready made answers to the lawyers. Will the lawyers not need a search function to find the correct template prompt? How do they prompt that? Why bother? What is the use case for even doing this?!
Legal Geek 2023: Summary on Gen AI and use cases
I asked ChatGPT to summarise the foregoing. Its attempt was pathetic. I wouldn’t even waste time using it as a basis for drafting a proper summary. Tara was correct: there are less good examples of summarisation.
Anyway, I will now, myself, summarise the outcome of Gen AI and use cases at Legal Geek 2023:
- There are no actual use cases of GenAI in real live practice of law taking place in law firms at the moment.
- If anyone tells you there are, then they are lying.
- Some law firms (probably just big law firms) are “looking at” and “experimenting with” GenAI.
- Chat GPT takes a lot of time and effort.
- Lawyers are too busy for that time and effort.
- No one (especially lawyers?) want to sit and chat with a chatbot.
- Chatbots are disappointing in any event.
- GenAI is expensive. Is it worth it?
Given the foregoing you really do have to wonder why law firms are bothering with it!
Legal Geek 2023: Document Automation
Something law firms should definitely be bothering with is document automation (sans GenAI).
Catherine Bamford of Bam Legal took the stage to tell us about ‘Legal Documentation – Where did it all go wrong?’
Catherine is very well equipped to discuss this topic being described as:
the world’s leading document automation expert, Catherine has helped many law firms successfully roll out and scale their use of document automation.
We were told by Catherine that although document automation has been around for some 20 years (I’m sure I can remember systems from circa 30 years ago although I have only actively been using such systems for about 15 years) it is still not widely used. This is because document automation is “really hard”. A lot of hard thinking and drafting is involved in a document automation project. Lawyers don’t usually have time to do it themselves. The most successful projects are outsourced or have dedicated in-house legal engineers. I can vouch for that having introduced the legal world to the job role of “legal process engineer” when we employed our first one at Inksters in 2014.
Catherine has now published a transcript of her Legal Geek 2023 talk on LinkedIn which is well worth a read.
I do wonder if in 20 years’ time there will be talks at Legal Technology conferences about how GenAI is still not widely used by lawyers because it is really hard to implement in law firms!
Bring Back Boring
Marcus Ramtohul, our Legal Geek 2023 host, told us, when introducing Catherine, that last year’s Legal Geek had lots of talks about document automation. This year Catherine’s talk was a drop of practical reality and truth in a sea of GenAI hype and gibberish.
I hope next year Jimmy Vestbirk does a #bringbackboring with more on document automation and the actual real benefits it can give lawyers. I am sure that for most lawyers a bit of #hardyards on document automation will pay more dividends than similar time spent on GenAI. As Catherine told us: “it is hard, but it is worth it”. Again, I can vouch for that having seen what it has done for my own law firm over the past 15 or so years.
Pre-mortems: How Imagining Failure Can Avoid Project Disaster
Another highlight of Legal Geek 2023 that I unfortunately missed in real life but caught up on after the event was ‘Pre-mortems: How Imagining Failure Can Avoid Project Disaster‘ by Peter Duffy.
It is all about what makes a problem-preventer at the start of a delivery project.
I was tempted to stay for the talk when Peter told me there was no GenAI in it! But I couldn’t at the time so was pleased to catch it when Peter kindly published it for all to see. Again, Like Catherine Bamford’s talk, it is well worth a read.
Legal Geek 2023: The People
As I always say about legal conferences, it is often as much about the people that you meet there than the content programmed.
Legal Geek 2023 was no exception to that. Indeed on day one of the conference I concentrated on meeting and chatting with people rather than taking in the content (which I did more of on day two).
The drinks reception this year was in the main exhibition hall which was a big improvement on having it in JuJu’s. Where historically they had to limit numbers getting in and where in 2019 I suffered an injury and ended up in a sling for a week after being pushed over in the tight crowd!
I was invited, along with a group of other Legal Geekers, by Uwais Iqbal of Simplexico, to an evening dining in a Punjabi restaurant. A very pleasant evening of good food and company it was indeed. Finishing up with drinks outside (it was still weather for it) at a nearby pub.
Over the two days old acquaintances were rekindled and new ones were made. On the whole it was more Withnail than GenAI. Yesterday I was in a wine bar where the WiFi password was “Uncle Monty” and I immediately thought of Legal Geek 2023!
Image credits ©: DALL-E 3/Brian Inkster – Withnail and GenAI; Brian Inkster – Richard E. Grant, GenAI use case panel, and Catherine Bamford at Legal Geek 2023; Helen Burness – Brian Inkster and Alex Hamilton; Uwais Iqbal – Punjabi restaurant; and Handmade Films – Withnail and I.
Reactions on Legal Geek 2023: The one with Withnail and GenAI
On LinkedIn the following comments have been made:-
Helen Burness (Legal marketing: simply built different ⚡️ LinkedIn for Legal ⚡️ Lawbreaker ⚡️ SEND parent ⚡️Smashing narratives one at a time 👊):
Ha ha love my photography credit! And also I know this image took you forever on Dall-E which I can never ever figure out, but it is also extremely cool.
Bonus points for not ending up in a sling this year. And yes – I’d love a #BringBackBoring! Show where the really hard and meaningful work is going on with regards to all all things automation.
Me:
Thanks Helen. You are always there with the camera! A much quicker, easier and more authentic way to get images than using GenAI!
Helen Burness:
And a digital legacy nightmare for my children 👍
Me:
Indeed!
-+-+-+-+-+-
Graeme Johnston (Software to map work – before that a lawyer):
Enjoyable to read and useful, as ever, Brian.
The refs to the holy grail inevitably conjured images —
Me:
Thanks Graeme. Legal Tech and Monty Python (when it is not Uncle Monty) often go hand in hand 😉 Was good to see and chat with you over a beer there.
Graeme Johnston:
Likewise. I also smiled at “it was still weather for it” (implied term – esp. for people living a few degrees further north)!
Me:
Indeed!
-+-+-+-+-+-
Dr Nick White (Making the intangible tangible! – IPM Consultant and Patent Attorney -Tangible IP):
My thoughts?
Brian the Bubble Burster.
Having attended many of these and others over a long period of time the consistent feeling these days is…meh.
No revolution on the immediate horizon.
Me:
Thanks Dr Nick White.
Those whose bubbles are being burst remain noticeably silent!
“Meh” is a very good word to sum up the feelings these days!
-+-+-+-+-+-
Sean Hocking (Consultant Legal Publishing Content Knowledge & Biz Development. Owner Cannabis Law Report, House of Butter, MP Records):
love this – the first commonsense article i’ve read on Legal AI this year .. rather than people endlessly going. “wow” and endlessly repeating the mindless mantra word “generative”
Me:
Thanks Sean. You’ll find a series of such articles by me linked to from here: https://thetimeblawg.com/chatbots/
-+-+-+-+-+-
Douglas Amber (Lawyer):
Brian, your Legal Geek 2023 post on the Time Blawg is brilliant. I cited and linked to it, in my post today. I hope you don’t mind.
Me:
Thanks Douglas. Feel free to cite and link as you please.
Douglas Amber:
You’re welcome, Brian. Thanks for reacting to my post.
-+-+-+-+-+-
Jamie Giani (Attorney | Audiobook Narrator | Test Prep Coach | Grief Awareness Advocate | Empathetic Human):
I did not attend Legal Geek, so thanks for your summary! I am surprised to learn that there are no real use cases GenAI yet in practice. Though it sounds from your commentary that I shouldn’t be. Very interesting considering all the hype.
Me:
Use cases are very elusive in real life. I am giving out magic wands to those that find them: https://thetimeblawg.com/2023/09/03/llms-in-law-hype-v-magic/
Jamie Giani:
Well now I’m motivated!
-+-+-+-+-+-
And on Twitter (now sometimes called X) the following comments were made:
Dave Anderson (@Fritteratti):
Strathclyde Uni did a wills drafting project with Bird Semple using CAPS Author in 1990: I was press-ganged into some code maintenance in 1992. Couldn’t believe my luck when hi-tech(!) HotDocs v.1 turned up the same year. #bringbackboring
Me:
Thought we were nearer the 30 year mark than the 20 year one. You were clearly a very early #LegalTech #documentautomation pioneer! cc
@BamLegal
-+-+-+-+-+-
Ron Friedmann (@ronfriedmann):
An In-House Revolution: Early Adopters of AI Reduce ‘Rote Tasks,’ Spend More Time on Strategy | http://Law.com
A long and worthwhile read for a sense of the state of play of #LLM, with discussion of some specific use cases.
Me:
Long but woolly. Usual summarising and brainstorming but weak on actual legal tasks. At #LegalGeek the use case panel would have called them all liars!
Peter Duffy (@PDffy):
Agreed, not much reference to legal tasks. A key challenge with many legal tasks is the tolerance for errors is so low, and rightly so!
So far I think LLMs are unsuitable for many legal use cases, but often incredible for use cases where we tolerate and can easily spot errors.
I’m sure they are suitable for some legal use cases, but we need to get really specific on the nuances of the use case before confirming suitability!
Me:
A magic wand goes to anyone who can actually achieve that!
