LLM-DRIVEN QUERY GENERATION FOR GRAPH-BASED BUSINESS INTELLIGENCE: TOWARDS A COLLABORATIVE KNOWLEDGE RETRIEVAL TOOL

Abstract

This paper explores the use of large language models (LLMs) to support collaborative business intelligence in the tourism domain through two key tasks: extracting travel-related tags from user queries and generating Cypher queries for accessing knowledge graphs. We focus on evaluating the performance of compact and efficient LLMs, aiming to balance accuracy with computational feasibility. To assess tag extraction, we evaluated Phi-3 Mini, LLaMA 3.2, and Gemma 3 using the DeepEval framework with G-Eval scoring. Phi-3 Mini showed the best balance between accuracy and efficiency, while Gemma 3 achieved the highest scores at the cost of increased resource usage. For Cypher query generation, we tested more powerful models: Mistral Small 3.1, Phi-4, Gemma 3, and ChatGPT-4o. ChatGPT-4o achieved the highest correctness, while Mistral Small demonstrated the best trade-off among smaller models. Our results suggest that lightweight LLMs are suitable for basic natural language processing tasks, but structured query generation remains challenging and requires stronger models. Further research is needed to improve the reliability of generated queries and to develop robust validation mechanisms. This study introduces a comparative evaluation of lightweight and standard LLMs specifically applied to collaborative business intelligence in the tourism domain. It highlights the feasibility of using compact LLMs for natural language processing tasks while demonstrating the challenges of structured query generation, which requires more powerful models.