Publications
Conferences
2025
- ICAIL
Curb Your Enthusiasm: Towards a RAG Framework to Forecast Case Importance in the ECHRDavid Bareham, Katie Atkinson, Jack Mumford, and Jeremy MarshallIn Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, ICAIL 2025, Chicago, United States, June 16-20, 2025, Jun 2025The task of forecasting case importance has received far less attention in the legal domain compared to judgment prediction, but it is a task that is an essential step to capture within tools for assisting with the processing of cases submitted to a court. In this paper we propose a cornerstone framework for carrying out the task of forecasting case importance, using communicated cases, which are documents available prior to any decision being issued. The setting for our work is cases in the European Court of Human Rights, with a specific focus on Article 3, prohibition of torture. We set out proposals for a Retrieval-Augmented Generation (RAG) framework that makes use of Large Language Models augmented with Semantic Search, Knowledge Graph and Re-Ranking components and we evaluate the effectiveness of this framework and its components. Further experiments conducted evaluate the framework using both pre-trained and fine-tuned LLMs, as well as use of different prompting strategies. The basic experiments show a propensity for the LLMs to significantly overestimate the importance of cases, but when we augment the LLMs with the aforementioned components, we are able to gain uplifts in performance. Our framework and results provide a solid basis for determining the requirements for the development of successful automated tools to be used to assist with determining case importance.
@inproceedings{Bareham2025, title = {Curb Your Enthusiasm: Towards a RAG Framework to Forecast Case Importance in the ECHR}, author = {Bareham, David and Atkinson, Katie and Mumford, Jack and Marshall, Jeremy}, booktitle = {Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, {ICAIL} 2025, Chicago, United States, June 16-20, 2025}, year = {2025}, month = jun, } - ICAILFinding the Goldilocks Zone: Retrieving Citation ContextJack Mumford, David Bareham, Katie Atkinson, and Jeremy MarshallIn Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, ICAIL 2025, Chicago, United States, June 16-20, 2025, Jun 2025
We report on a first set of results from experiments undertaken to tackle the novel task of determining the optimal context window for extracting and contextualising citation instances within case law. The wider task of outcome prediction using AI tools cannot be undertaken without considering the role that citations play when new cases are being decided. This short paper aims to shine a light on the importance of this task and provide the foundation for developing AI tools to capture citations’ context by examining a corpus of legal cases taken from the European Court of Human Rights. Our results show that there is an identifiable “Goldilocks Zone” of scoped paragraph-level context windows that attention can be focused on for extracting citation instances.
@inproceedings{Mumford2025, title = {Finding the Goldilocks Zone: Retrieving Citation Context}, author = {Mumford, Jack and Bareham, David and Atkinson, Katie and Marshall, Jeremy}, booktitle = {Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, {ICAIL} 2025, Chicago, United States, June 16-20, 2025}, year = {2025}, month = jun, }
Workshops
2023
- AI & A2J
Tackling the Backlog: Support for Completing and Validating FormsKatie Atkinson, David Bareham, Trevor Bench-Capon, Joe Collenette, and Jack MumfordDec 2023Presented at the Workshop on AI and Access to Justice @ JURIX 2023In a wide variety of jurisdictions and courts, an accumulation is building up of legal cases waiting to be processed. New technologies offer promising solutions for tackling the backlogs and reducing the build up of cases, but the technologies must be targeted at specific bottlenecks. In this paper we describe work completed to meet these aims in two ways: 1) a tool aimed at reducing the number of incorrectly completed forms submitted; 2) a decision support tool for the identification of inadmissible claims submitted to the court. Both tools were developed for the setting of cases submitted to the European Court of Human Rights. We provide an overview of both these tools and their evaluation in initial trials, with the aim of enabling claimants to get improved access to justice by providing a level of automation for key decision points in the application process.
@misc{AtkinsonTacklingTB, title = {Tackling the Backlog: Support for Completing and Validating Forms}, author = {Atkinson, Katie and Bareham, David and Bench-Capon, Trevor and Collenette, Joe and Mumford, Jack}, url = {https://cgi.csc.liv.ac.uk/~tbc/publications/Access_to_Justice_workshop23.pdf}, note = {Presented at the Workshop on AI and Access to Justice @ JURIX 2023}, year = {2023}, month = dec, } - DC
Predicting Decisions of the European Patent Office’s Boards of Appeal Using Machine LearningDavid BarehamDec 2023Presented at the Doctoral Consortium @ JURIX 2023Best Paper of the Doctoral Consortium
This research assesses the feasibility of applying machine learning (ML) methods to the problem of case outcome prediction for appeals from the European Patent Office’s (EPO) Boards of Appeal, concerning the grant of a patent application. The task is conceptualised as binary classification in which an appeal can affirm or reverse the prior judgement. Using a range of ML classifiers and textual representations, including custom-trained word and document embeddings, two experiments were conducted on appeal cases from both the Examining and Opposition Divisions of the EPO. The first experiment uses randomlysampled data and the second uses year-stratified data, to perform prediction. The F1-scores achieved for the future prediction task across both divisions are 86.64% and 78.55% respectively. The results demonstrate the viability of applying ML techniques to predict the outcome of appeals concerning the patent grant procedure, and help to identify patents as a promising legal domain for future research. Furthermore, explainability analysis conducted with SHAP helps to identify a direction for future work concerning more robust explainability.
@misc{BarehamPredEPO, title = {Predicting Decisions of the European Patent Office's Boards of Appeal Using Machine Learning}, author = {Bareham, David}, note = {Presented at the Doctoral Consortium @ JURIX 2023}, year = {2023}, month = dec, }
MSc Theses
2023
- MScPredicting Decisions of the European Patent Office’s Boards of Appeal Using Machine LearningDavid BarehamSep 2023Grade = Distinction
This work aims to assess the feasibility of applying data science and artificial intelligence methods to the problem of case outcome prediction for appeals from the European Patent Office’s Boards of Appeal, concerning the grant of a patent application. The task is conceptualised as a binary classification task in which an appeal can ‘affirm’ or ‘reverse’ the prior judgement. Using a range of machine learning classifiers and textual representations, including custom-trained word and document embeddings, two experiments were conducted on appeal cases from the Examining Division of the European Patent Office. The first using randomly-sampled data and the second with year-stratified data to engage in future prediction. The first experiment achieved 85% accuracy and the second an average of 86%. The results demonstrate the viability of applying machine learning techniques to appeals concerning the patent grant procedure, showing that patents as a legal domain may be promising for future case outcome prediction research.
2021
- MScImplementing Tools For Supporting Legal ReasoningDavid BarehamSep 2021Grade = Distinction
This project brings together the research which has been conducted in the University of Liverpool on the ANGELIC project into a single interactive tool. The ANGELIC project has provided a methodology for implementing a system to determine the outcome of legal cases based upon legal domains constructed from precedent cases. This methodology draws from prior work into ADFs (Abstract Dialectical Frameworks), which are sets of nodes with directed links used to represent the factors of legal domains and the relationships between them. Several domains had been individually modelled and evaluated using this methodology, but this had not yet been brought together in a unified tool to be used in a practical setting. The tool produced enables the user to query a legal case for one of the domains modelled by answering a number of questions instantiating the case as a list of factors. These cases are then evaluated according to the relevant ADF graph using a depth-first search providing a case outcome. The result of the case is predicted, and a list of reasons for why that outcome was reached is generated, alongside a visualisation highlighting the relevant factors for the decision. The tool also allows the user to create ADFs for new legal domains or edit existing one. The project was successful when evaluated in regard to accuracy, explainability and usability. The tool was able to replicate the results of the previous implementation within Prolog, achieving the same accuracy, provide a transparent outcome with clearly stated reasons and visualisation for explainability as well as aiding usability by providing a comprehensive user manual and intuitive user interface.