Towards an Open NLI LLM-based System for KGs: A Case Study of Wikidata
Informasi
Jurnal7th International Seminar on Research of Information Technology and Intelligent Systems: Advanced Intelligent Systems in Contemporary Society, ISRITI 2024 - Proceedings, 2024 7th International Seminar on Research of Information Technology and Intelligent Systems (ISRITI)
PenerbitInstitute of Electrical and Electronics Engineers Inc., IEEE
Halaman44 - 49
Tahun Publikasi2024
ISBN979-833151964-3
Jenis SumberScopus
Sitasi
Scopus: 2
Google Scholar: 2
PubMed: 2
Abstrak
The rise of large language models (LLMs) has significantly advanced information retrieval, yet challenges like the limitation of knowledge updating ability, lack of openness, and hallucination issues persist. To address these, Retrieval-Augmented Generation (RAG) has been introduced but remains limited in interpretability due to its reliance on vector-based representations. This paper presents a question-answering (QA) system using GraphRAG, a RAG system with knowledge graphs (KGs) as its base. We develop a natural language interface (NLI) for QA over Wikidata, a popular, open, and crowdsourced KG. Our approach employs LLM chaining, i.e., a paradigm that leverages multiple LLM calls sequentially, to generate SPARQL queries, with the aim of creating an open system that ensures transparency and allows direct inspection of its components. Utilizing an experimental research approach, we evaluated the generated SPARQL queries and found that incorporating a broader set of property candidates into the prompts significantly boosts performance, achieving a Jaccard similarity score of 0.7806. These findings demonstrate the system's effectiveness in SPARQL query generation, highlighting its potential for further development. However, we consider the limitation of the LLM's context window and the hallucination phenomenon as the major challenges that limit the system's performance. © 2024 IEEE.
Dokumen & Tautan
