TinySQL: A Progressive Text-to-SQL Dataset for Mechanistic Interpretability Research

Abir Harrasse1,5, Philip Quirke1, Clement Neo2, Dhruv Nathawani3, Luke Marks1, Amir Abdullah4,1
1Martian, 2Apart Research, 3NVIDIA, 4Thoughtworks, 5Mohammed VI Polytechnic University
arXiv Poster Code Dataset Models
TinySQL Overview

TinySQL provides a structured testbed for mechanistic interpretability research, progressing from basic to advanced SQL operations across multiple model sizes.

Abstract

Mechanistic interpretability research faces a gap between analyzing simple circuits in toy tasks and discovering features in large models. To bridge this gap, we propose text-to-SQL generation as an ideal task to study, as it combines the formal structure of toy tasks with real-world complexity.

We introduce TinySQL, a synthetic dataset, progressing from basic to advanced SQL operations, and train models ranging from 33M to 1B parameters to establish a comprehensive testbed for interpretability. We apply multiple complementary interpretability techniques, including Edge Attribution Patching and Sparse Autoencoders, to identify minimal circuits and components supporting SQL generation.

We compare circuits for different SQL subskills, evaluating their minimality, reliability, and identifiability. Finally, we conduct a layerwise logit lens analysis to reveal how models compose SQL queries across layers: from intent recognition to schema resolution to structured generation.

Dataset Structure

TinySQL structures tasks along two dimensions: SQL complexity and query language variation. The dataset progresses through five levels (CS1-CS5) with three language variants (Base, Synonyms, Natural). Try the interactive demo below to explore the dataset.

Open in Spaces

Notebooks

Explore our analysis notebooks to reproduce the results and dive deeper into the interpretability techniques.

EAP Circuits SAE Circuits LogitLens

BibTeX

@misc{harrasse2025tinysqlprogressivetexttosqldataset,
      title={TinySQL: A Progressive Text-to-SQL Dataset for Mechanistic Interpretability Research}, 
      author={Abir Harrasse and Philip Quirke and Clement Neo and Dhruv Nathawani and Luke Marks and Amir Abdullah},
      year={2025},
      eprint={2503.12730},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2503.12730}
}