Table 3 Comparison of legal question answering datasets in terms of languages, source, category, and size
Dataset | Language | Source | Category | Answer type | Size | Year | Question annotators | Availability | Valuation metrics |
|---|---|---|---|---|---|---|---|---|---|
PrivacyQAa | English | Crowdsourced from Google Play Store | Privacy policies | List of Sentences | 1750 | 2019 | Domain experts | Yes | Precision, recall, accuracy |
JEC-QAb | Chinese | National Judicial Examination of China and other exam websites | Legal knowledge | Single-answer Multiple-answer | 26,367 | 2020 | Not mentioned | Yes | Accuracy |
BSARDc | French | Belgian law articles | Legal texts | Article | 1100 | 2021 | Experienced jurists | Yes | Recall, MAP, MRR |
PIL | English | Rome I Regulation EC 593/2008, Rome II Regulation EC 864/2007, Brussels I bis Regulation EU 1215/2012 | International private law | Articles Recitals Commission Statements | 17 | 2021 | Not mentioned | Yes | Recall, precision, F1 |
COLIEE-2015 | Japanese and English | Japanese Legal Bar exams | International private law | Articles Yes/No | 412 | 2015 | Not mentioned | Yes | F-measure, Accuracy |
VLQA | Vietnamese | Web pages for driver license test | Transportation law domain | Span of words Yes/No | 1678 | 2017 | Manually annotated | No | Precision, recall, F1 |
CUADd | English | Electronic Data Gathering, Analysis, and Retrieval (“EDGAR”) system | Legal contract review | Clause | 13000 | 2021 | Legal experts | Yes | Precision, recall |
Hoppe et al. [35] | German | German Court Rulings, German Legal Documents | Court Rulings | Passage | 200 | 2021 | Hand annotated | Yes | Recall, MAP |
AILA | Chinese | Chinese Legal Forum | Chinese Law | Span of words | 139,468 | 2020 | Professional from law firm | No | MAP, MRR |
CJRC | Chinese | Judgments from Supreme Peoples’ Court of China | Court Judgments | Span of words yes/no unanswerable | 50K | 2019 | Law experts | No | F1-Score |
Collarana et al. [17] | English | MaRisk Regulatory document | Span of words | 631 | 2018 | Not mentioned | No | F1-Score, exact match | |
LCQA | English | Avvo QA forum | User privacy | 5,628,689 | 2022 | Not mentioned | Yes | MRR, MAP, precision | |
Kien et al. [42] | Vietnamese | Legal Advice Websites | law and Regulation Documents | Article | 5922 | 2020 | Not mentioned | No | Recall@20, NDCG@20 |
PolicyQAe | English | OPP-115 Corpus | Privacy Policies | Span of words | 714 | 2020 | Hand annotated | Yes | EM, F1-Score |