A global online technology leader partnered with ALTEN to develop a reliable framework for testing and enhancing large language model (LLM) capabilities—specifically to ensure precise and efficient natural language-to-SQL (NL2SQL) query generation for complex datasets. The collaboration resulted in significant improvements in accuracy, enabling LLMs to generate correct and efficient SQL queries from natural language inputs.

Challenges

In the era of data-driven decision-making, converting natural language into structured query language (SQL) presents a major challenge for large language models. Existing LLMs often produce inaccurate SQL statements that lead to data errors, inefficiencies, or compliance risks. Such inaccuracies can cause:

• Incorrect insights being delivered to clients or stakeholders
• Misleading data that affects business decisions
• Potential system crashes or data integrity issues
• Risks of regulatory non-compliance and data leakage

The core challenge was therefore to improve the ability of LLMs to generate SQL queries that correctly answer natural language queries for specific datasets, ensuring both accuracy and robustness across diverse data structures.

Solutions

ALTEN formed a dedicated engineering team to design, test, and refine LLM-driven natural language to SQL (NL2SQL) systems through a comprehensive, benchmark-based approach:

• Automated Benchmarking Framework (GAINS): ALTEN developed the Generative AI Benchmark System (GAINS) to evaluate and compare LLM performance—including OpenAI ChatGPT, Google Gemini, and Anthropic Claude 3—in generating SQL queries.
• Prompt Engineering Optimization: Refined prompt design to guide LLMs toward generating accurate, efficient SQL for given datasets.
• Custom Dataset Creation: Built domain-specific datasets to train and fine-tune LLMs, creating high-quality natural language–to–SQL pairs.
• Model Training and Validation: Conducted iterative testing and model refinement to identify and correct errors in both SQL output and datasets.
• Database Integration: Applied and validated results across leading platforms such as Google BigQuery, Amazon Redshift, Databricks, Snowflake, MySQL, and PostgreSQL.
• Continuous Improvement Loop: Benchmarked models over time, compared outputs, and implemented feedback to enhance accuracy, reliability, and efficiency.

Outcomes

Improved Query Accuracy: LLMs can now generate SQL that correctly and efficiently answers natural language questions.

• Custom AI Frameworks: ALTEN’s GAINS framework ensures reproducible, transparent evaluation of AI model performance.
• Higher Efficiency: Automated validation reduced manual debugging, saving time and resources.
• Enhanced Business Confidence: Reliable AI-assisted data querying strengthened client decision-making and data governance.
• Scalable Impact: The same benchmarking and refinement approach can be applied to other AI-driven data analytics solutions.

Through its expertise in generative AI and data engineering, ALTEN is bridging the gap between natural language understanding and structured data analysis—empowering enterprises to transform questions into accurate, actionable insights.