CaseCrunch AI Wins the Man vs. Machine Lawyer Competition

The highly anticipated showdown between lawyers and an AI bot happened last week, and the result is now finally out!

Back in August, Edgy Labs covered a story about the London-based legal tech startup Elexirr and its AI Bot which gives free legal counseling to people. If you missed the story, you can read it here. Elexirr has become CaseCrunch, and while the name has changed, their goal to outperform human lawyers with an AI stayed the same.

Back then, the company claimed that its AI bot could predict case outcomes 71% of the time for all areas of law and all common law jurisdiction. While many were impressed by the figures provided by Elexirr, some remained skeptical. So, to prove the legitimacy of their claims, the company’s the Managing Director, Ludwig Bull, challenged London’s top lawyers in a battle of prediction.

Elexirr, now known as CaseCrunch, challenged lawyers to predict case outcomes regarding PPI mis-selling claims. Dubbed ‘Man vs. Machine,’ the competition finally reached its conclusion on October 27th.

AI Bot vs. Lawyers

Between October 20th and 27th, lawyers who wanted to participate in the competition were asked by CaseCrunch to log in to a website where they could access facts of real PPI mis-selling complaints received by the Financial Ombudsman Service.

Once the website had been accessed, the lawyer participants could then predict whether the complaint was upheld or rejected. The AI bot also received the same facts.

After one whole week of receiving predictions, the result was computed and was announced by Ian Dodd, UK Director of Premonition at the offices of the insurance law firm Kennedys, on the night of October 27th.

“With an accuracy of 86.6 percent, compared to the lawyers’ accuracy of 62.3 percent, CaseCrunch emerged victorious.”

In a press release sent by CaseCrunch to Edgy Labs, Josef Maruscak, the current Managing Director of CaseCrunch said:

“We could not be happier about the outcome. We are grateful to all involved parties, especially competing lawyers who were not afraid to participate. We are not necessarily adversaries in this game, the systems like ours can make the legal world more effective for everyone. I am convinced that we have now reached the point where our technology and expertise allow us to satisfy both our vision and our commercial interests. We are looking forward to finding solutions for our clients now.”

According to the company, 112 lawyers pre-registered to participate in the Lawyer Challenge, and the participants submitted 775 predictions.

“A Technology Judge and a Legal Judge independently verified the fairness of the competition. The Legal Judge was Felix Steffek (LLM, PhD), University of Cambridge Lecturer in Law, and Co-Director for the Centre of Corporate and Commercial Law. The Technical Judge was Ian Dodd, UK Director of Premonition.”

CaseCrunch used factual scenarios that were real decided cases from the FOS for the competition. However, all identifying details such as the name of the parties, case names, and date were removed to ensure anonymity.

Lawyers were able to complete their predictions in an unsupervised environment, using all available resources. Apparently, PPI mis-selling claims were chosen as the basis of the competition since it matched most of the lawyers’ background.

In a statement to Artificial Lawyer earlier this month, the company said:

“We chose an area of commercial law to ensure that participating lawyers can comfortably answer the questions. Lawyers will also have the chance to learn more about PPI mis-selling before starting the competition.”

Rebecca Agliolo, Marketing Director of CaseCrunch, claims that the challenge is not about winning or losing. Instead, it’s about showcasing the potential of artificial intelligence and changing the current paradigm with results rather than just promises.

Bull, now the Scientific Director of the company, further said that the result of the challenge doesn’t mean that machines are generally better at predicting outcomes than human lawyers. However, he claimed that if the question is defined precisely, machines could compete and potentially outperform human lawyers.

What do you think about using automation in the legal sphere?