Chats with Legal Chatbots ~ Episode 1 : The Global Legal Hackathon and LawDroid

This is a series I have had in mind for a little while since I experimented, a few months ago, having chats with some legal chatbots. Apparently chatbots are the next big thing.

In my Legal Technology/IT Predictions for 2018 I blogged:-

The big thing in 2018 will be for law firms to start introducing chatbots onto their websites never mind getting actual content on there first for the chatbot to reference.

The fact that introducing a basic search function to their website might be more effective and useful than a chatbot that gets confused with most queries put to it (unless there is a human being operating it) will be lost on most lawyers.

However, if it has not been done already, a legal chatbot will in 2018 be able to order your Uber for you. This will be major news in legal publications but not really a giant leap for the legal profession.

So I thought I would test out a chatbot or two. First up is LawDroid, the chatbot used by the Global Legal Hackathon. I tried to have a conversation with it at the time the Global Legal Hackathon was in full swing and before writing my blog posts linked to or about that event: Lawyers and coding; Hack the Law to Reinvent the Wheel? and London Legal Hackathon bend the rules?

For the uninitiated it appears to be the norm and perhaps expected (at least in the UK) at a hackathon that the participants will be supplied with pizza. So I thought I would ask LawDroid about pizza.

This is how our chat went:-

LawDroid: Hi [hand symbol]! I’m LawDroid.

What’s your name?

Me: Bond, James Bond.

LawDroid: Hi James Bond!

I’m programmed to provide helpful information for Global Legal Hackathon hosts and attendees.

If you ever get stuck, don’t worry, you can type in the word ‘restart’ to start over. Or, type ‘end’ to stop.

Shall we get started?

Me: Yes

Law Droid: Would you like to learn more about GLH’s Organizers?

Me: No.

LawDroid: Will you be attending or hosting the Global Legal Hackathon in your location?

[presented with Attending or Hosting option]

I was doing neither but selected Attending.

LawDroid: Here are some FAQs for attendees:

What is the Global Legal Hackathon? What is a hackathon? How do I form a team? etc

Would you like to try another option?

Host Info – Attendee Info – Ask a Question

Me: Ask a Question

LawDroid: What issue do you need help with?

Me: Will there be pizza?

LawDroid: So we can follow up with you about your concern, I’ll need a couple of pieces of information.

What’s your email address?

Me: I don’t want you to e-mail me I want you to chat.

LawDroid: Sorry, I didn’t quite get that

Me: Can we continue our chat?

LawDroid: Sorry, I didn’t quite get that

Me: You have not been programmed well have you?

LawDroid: Sorry, I didn’t quite get that

Me: You don’t get very much do you?

LawDroid: Sorry, I’m still not getting that.

Me: Will we start again?

LawDroid: Hmm. I didn’t get that. Sorry.

Me: Goodnight.

LawDroid: Hmm. I didn’t get that. Sorry.

I gave up. A lot of time and questions to not get the answer I was looking for.

The Global Legal Hackathon website didn’t help much on the pizza or food front either. All I could find under a “food” search (a “pizza” search returned no results) was a recommendation to bring non-alcoholic beverages with you.

But the London Legal Hackathon had their own website with a FAQ page which reveals to the question “Will there be food?”:-

We are catering for around 100 participants comprising of 8 to 12 teams. There will be a plentiful supply of tea, coffee, water and meals (kindly provided by our sponsors). We’ll add more details here soon (and we’ll cater for non-meat eaters too, but please warn us if you have any special dietary requirements).

We will provide lunch and an evening meal, but not breakfast, so please grab a bite on your way Saturday and Sunday please.

You are welcome to bring your own food and refreshments if you want.

No mention specifically of pizza but at least it tells you there will be food.

Although confirmation, if needed, that there was pizza (at least in London and Sydney) could be found on Twitter:-

The essentials of any hackathon are announced.#GlobalLegalHack #GLH2018 #DevOps #tech pic.twitter.com/LMH0ntbeMd

— Disruptive LIVE (@disruptivelive) February 23, 2018

He wasn’t wrong pic.twitter.com/Uu6IomT3bB

— Disruptive LIVE (@disruptivelive) February 23, 2018

Pitches start in less than 2 hours!
But first…pizza #GLH2018 #hackathon #GLHSydney #LegalTech @_23Legal @Legal_Forecast pic.twitter.com/8cSR2EcgYP

— Ivy Shi (@ivy_shi1703) February 25, 2018

So a quick glance through a FAQ page on the London site (and this might apply to other location sites on the global circuit) was much quicker and more helpful than using the chatbot on the main site. So far, on the whole, that has been my experience of using legal chatbots. Do you agree? Or have you had good experiences of using legal chatbots? If you want to do a guest post of your conversation with a legal chatbot on this blog, for a future episode in this series, I will be happy to consider any submissions. Just e-mail it to me.

Reactions on Social Media

There have been reactions to this post on LinkedIn. To keep these together with the post itself I have copied the comments here:-

Alex G Smith:

I broke a GDPR one in a minute by asking it what privacy by design was. It then had a chat with me about how my day was and was having a good time. I only tried it because the hype said AI had landed … I never expected the Spanish Inquisition on how I was feeling about life in general. Can’t wait for part 2.

Graham Laing:

Create something even less responsive than the lawyers they ultimately seek to replace. Only in legal ;-P

Garth Watson:

If you think about the phrase “privacy by design” it’s no wonder it was confused.

Surely, “well designed privacy policies”, as opposed to privacy itself by a thing called “design” would be a better way of putting it.

Most lawyers would also struggle to answer what privacy by design was. Most obvious answer is locking the door.

Totally get that these thing have a long way to go, but baby steps come first.

Graham Laing:

That is an interesting question in itself – can ‘baby steps’ be permissible in a highly regulated environment? My view is not. Something works or it doesn’t.

Alex G Smith:

Garth Watson, Privacy by design is a core principle that GDPR pushes forward and given that these bots are “hand programmed” not self learnt I felt it not a hard question to ask. I don’t mind if the narrow decision tree it no doubt was following didn’t have it but if that’s all it is why the whole bot thing, an FAQ or a nice blog could have done the job. I’ll give your questions (or search terms) a go and see what happens … my guess is I’ll end up in the bizarre banter on “how I am” and “am I well” as last time.

-+-+-+-+-+-

Nir Golan:

At least you checked one of their GLH requirements box- legal chatbot.. check. it doesn’t matter if it does not solve the problem. That’s not relevant to the GLH.

-+-+-+-+-+-

Jonathan Maskew:

Well worth having a chat with Billy Bot

Brian Inkster:

Oh I have! We didn’t chat about pizza, although I understand he (unlike LawDroid) has been trained to deal with pizza queries. I would expect Billy to feature in a future episode of ‘Chats with Legal Chatbots’ 🙂

-+-+-+-+-+-

Enia Titova:

I don’t understand how an industry that puts either “this is not legal advice” disclaimers or “PRIVILEGED & CONFIDENTIAL, ATTORNEY-CLIENT COMMUNICATION” on literally everything that comes out of its mouth thought this was a good idea.

-+-+-+-+-+-

Alastair Ross:

Good to see someone using the good Lean principle of ‘Go see’! Look forward to Episode 2 Brian – hoping there will be pizza….

-+-+-+-+-+-

David Martin:

Managing the expectation from Chatbots is a challenge.
It is where the Chatbots are used for “Transactions” that typically generates the integration to RPA.

-+-+-+-+-+-

Thomas G. Martin (Founder at LawDroid, Bot Development and Consulting for Legal Industry):

Hi Brian – Thanks for checking out LawDroid as deployed on the Global Legal Hackathon website.

In the lead up to GLH, LawDroid helped provide thousands of attendees and hosts with helpful information to prepare. However, it wasn’t programmed to answer venue specific questions. We did build in an opportunity for a person to ask a question that wasn’t answered by the bot and pass it on to a GLH volunteer for assistance. This automated + human approach worked very well for many, many people.

LawDroid gave you this opportunity to ask such a question which it would email to a volunteer for follow up, but you didn’t do that. It also provided instructions for how to restart the conversation but you didn’t do that either. So, yes, you managed to break it.

I would have enjoyed speaking with you to discuss our approach to designing the bot, what problem we were attempting to solve,  what results we achieved, and what we learned, but you never reached out to me to discuss.

Brian Inkster:

Tom, I didn’t think of it as a venue specific question as pizza and hackathons seem to go hand in hand from London to Sydney (although I accept some locations may not have been so keen on pizza).

However, in determining questions that might be asked by attendees I would have put it fairly high on the list.

The opportunity you built in for a person to ask a question that wasn’t answered by the bot and pass it on to a GLH volunteer for assistance was rather odd in its approach. I asked “Will there be pizza?”. The bot answered “So we can follow up with you about your concern, I’ll need a couple of pieces of information. What’s your email address?”. It wasn’t a “concern” as such just a query. The bot didn’t indicate that it didn’t know the answer and that a human would have to help if I supplied an e-mail address. It would have been more helpful had it done so in a more straightforward and honest fashion.

I must have missed the instructions for how to restart the conversation. Perhaps it would be helpful for the bot to actually tell you these when struggling as it was to answer. However, I would have thought that it would not be that difficult to factor in a restart from a statement like “Will we start again?”. However, I’m not sure I would want to restart and go around in circles again in any event!

I’m not sure that I need to reach out to the bot’s designers before posting an episode of an actual conversation with a chatbot. I am simply demonstrating what such a conversation is like and the limitations that chatbots may have as a result. It seems clear that they can only deal with what they have been programmed to deal with and that won’t always help the person chatting with them. Indeed it is likely to just lead to frustration.

Thomas G. Martin:

You actually quoted the instructions on how to restart in your article, “If you ever get stuck, don’t worry, you can type in the word ‘restart’ to start over. ”

Brian Inkster:

But clearly well through the conversation those instructions were by then lost on me. As clearly was the “end” one as I assumed, wrongly, that “goodnight” might achieve that result in a more chat like way. Again this simply highlights the robotic nature of chatbots that so far have been a big turn off for me.

Thomas G. Martin:

The issue is one of expectations. A chatbot can be useful for its intended purpose. In this case, we built the bot for the main GLH site the best we could for the most important use case (i.e., somebody getting started with wanting to participate as an attendee or host). GLH had been receiving hundreds of inquiries for the same information that they couldn’t physically handle manually. LawDroid succeeded in handling the burden of those repetitive questions and thousands of messages for most people for the intended use case. And we still built in the ability to manually field questions that did not fall within the ambit of the covered FAQs.

Brian Inkster:

So you should have low expectations when using a chatbot? That is certainly my experience to date.