Benchmarking KG-based RAG Systems: A Case Study of Legal Documents
Author: Jaycent G Ongris, Fariz Darari, Berty CL Tobing, Douglas R Faisal, On Lee
Informations
JournalCEUR Workshop Proceedings
PublisherCEUR-WS
Publication Year2025
ISSN16130073
Source TypeGoogle Scholar
Abstract
Retrieval-augmented generation (RAG) systems enhance language model outputs by incorporating external knowledge, typically in the form of unstructured text. Recent advancements have introduced structured sources such as knowledge graphs (KGs) to improve retrieval precision and interpretability. This study benchmarks several KG-based and hybrid RAG frameworks, including HippoRAG 2, Nano GraphRAG, LightRAG, and LlamaIndex, to be compared with a naive RAG baseline, in the context of legal question answering (QA). The evaluation is performed on a multilingual legal corpus comprising EU Directives and Indonesian Government Regulations. A semi-automated pipeline, combining language models and human refinement, is used to generate high-quality QA datasets. We assess system performance using Ragas answer accuracy metric and identify the trade-offs between efficiency, interpretability, and accuracy. Our findings demonstrate the superior performance of hybrid approaches, particularly LightRAG Mix and LlamaIndex Hybrid, in terms of accuracy. Conversely, KG-only systems often underperform due to their inability to fully capture the semantics of the text. This work provides actionable insights for the development of reliable and multilingual legal QA systems.
Documents & Links
