Towards Explainability and Fairness in Swiss Judgement Prediction: Benchmarking on a Multilingual Dataset

The assessment of explainability in Legal Judgement Prediction (LJP) systemsis of paramount importance in building trustworthy and transparent systems,particularly considering the reliance of these systems on factors that may lacklegal relevance or involve sensitive attributes. This study delves into therealm of explainability and fairness in LJP models, utilizing Swiss JudgementPrediction (SJP), the only available multilingual LJP dataset. We curate acomprehensive collection of rationales that `support' and `oppose' judgementfrom legal experts for 108 cases in German, French, and Italian. By employingan occlusion-based explainability approach, we evaluate the explainabilityperformance of state-of-the-art monolingual and multilingual BERT-based LJPmodels, as well as models developed with techniques such as data augmentationand cross-lingual transfer, which demonstrated prediction performanceimprovement. Notably, our findings reveal that improved prediction performancedoes not necessarily correspond to enhanced explainability performance,underscoring the significance of evaluating models from an explainabilityperspective. Additionally, we introduce a novel evaluation framework, LowerCourt Insertion (LCI), which allows us to quantify the influence of lower courtinformation on model predictions, exposing current models' biases.