A Comparative Evaluation of T5 and BART Transformer Models for Natural Language to SQL Query Generation on the Spider Benchmark

Structured databases are now central to decision-making across industry, government, and research, yet retrieving information from them still requires SQL proficiency that most end-users do not possess. Natural Language Interfaces to Databases (NLIDB) address this barrier directly: they accept a question in plain language and return an executable SQL query, removing the specialist skill requirement and making data more widely accessible.

Translation from natural language to SQL is considerably more difficult than it may first appear. Single-table lookups with straightforward filters are manageable even for older rule-based systems; however, queries that chain multiple tables through joins, nest sub-selects, or group results with HAVING clauses require considerably more. A system must reconcile the ambiguity of natural language with the rigid structural rules of SQL and the specific column and table names of an individual database schema, a combination that exposes the limits of any approach that ignores schema structure.

Among recent sequence-to-sequence architectures, BART [1] and T5 [2] stand out for their broad applicability. BART was pretrained as a denoising autoencoder: its decoder learns to recover original text from corrupted versions, an objective that appears to help with generation quality under noisy or underspecified input. T5 adopts a different approach, casting every NLP task as a text-to-text problem and learning a single unified generation framework. Both have achieved competitive results on translation, summarization, and reading-comprehension tasks. What is less clear is how either model performs when the output must be not free-form text but syntactically valid SQL drawn from an unseen database schema, a question this paper addresses directly.

For this comparison, Spider [3] was used: a benchmark built around 200 databases drawn from 138 distinct domains and comprising 10,181 labelled question–query pairs of varying complexity. Because the test databases are withheld from training, Spider measures genuine cross-domain generalization rather than schema memorization, which makes it a substantially harder challenge than same-domain datasets such as WikiSQL [4].

Schema-aware systems such as RAT-SQL [5] and RYANSQL [6] achieve strong Spider scores largely because they include explicit mechanisms that link each question word to its most relevant schema element. What has received less attention is how far a practitioner can get by simply fine-tuning an off-the-shelf model such as BART or T5 without any of those specialized components. This paper addresses that gap and makes the following contributions:

1. Controlled comparison: Fine-tuning and evaluating BART and T5 under identical conditions on Spider, providing a reproducible baseline for general-purpose transformer models on complex text-to-SQL translation.

2. Metric analysis: Reporting EM, BLEU Score, and Logical Form Accuracy (LFA), and analyzing the implications and known limitations of each metric for structured query evaluation.

3. Gap analysis: Positioning the results against schema-aware baselines to quantify the performance gap attributable to the absence of schema-linking mechanisms.

4. Roadmap for improvement: Identifying the architectural enhancements (schema linking, constrained decoding, and execution-accuracy optimization) most likely to close the observed gap.

The remainder of this paper is organized as follows. Section 2 reviews related work. Section 3 describes the methodology. Section 4 presents the experimental results and discussion. Section 5 concludes the paper.

Text-to-SQL generation is treated here as a supervised sequence-to-sequence problem: given a natural language question and a linearized database schema, the model must produce a valid SQL string. BART and T5 were each fine-tuned from their publicly available base checkpoints using the same data splits, tokenization scheme, optimizer settings, and number of training steps. The subsections below describe the formal setup, the two models, how schema information was encoded, how data were prepared, and how performance was measured.

3.1 Problem formulation

Denote a natural language question as Q = {q₁, q₂, ..., qₙ} (n tokens) and its paired SQL query as S = {s₁, s₂, ..., sₘ} (m tokens). Training seeks a function f_θ: Q → S mapping the input question to the target query. Both models were trained by minimizing the token-level cross-entropy loss over the training corpus D = {(Qᵢ, Sᵢ)}, i = 1, …, N:

$\mathrm{L}(\theta)=-\Sigma_{\mathrm{j}} \log \mathrm{P}\left(\mathrm{s}_{\mathrm{j}} \mid \mathrm{Q}, \mathrm{s}<_{\mathrm{j}} ; \theta\right)$

where, P(sⱼ | Q, s<ⱼ; θ) is the conditional probability of the j-th output token. The global training objective is:

$\theta^*=\arg \min \theta(1 / N) \Sigma_i L\left(\theta ; Q_i, S_i\right)$

At test time, greedy decoding was used throughout: at each step, the token with the highest predicted probability was appended until the end-of-sequence token was emitted. Beam search was deliberately omitted so that the inference conditions remained consistent with the base configurations reported for the baseline systems.

3.2 Model architecture

T5-base and BART-base are both encoder–decoder transformers, but their pretraining objectives and input conventions differ in ways that are relevant to SQL generation; these differences are summarized in Table 1. Figure 1 shows the end-to-end workflow used in this study for both models, and Figure 2 outlines the data preprocessing and training pipeline described in Section 3.3.

1.png

Figure 1. End-to-end architectural workflow of T5 and BART for Natural Language Interfaces to Databases (NLIDB) text-to-SQL generation

2.png

Figure 2. Data preprocessing and model training pipeline

Table 1. Comparative architectural features of T5-base and BART-base

3.2.1 T5

T5 [2] handles every NLP task through the same text-to-text interface: the encoder reads a task-prefixed string, and the decoder outputs the answer as a token sequence. For text-to-SQL, the prefix “translate English to SQL: [question] context: [schema]” was used. The bidirectional encoder attends over the entire input, including the linearized schema, building contextual representations before the decoder begins generating SQL tokens one at a time. Because there is no task-specific output layer, T5 applies no SQL grammar constraint during decoding; it can generate any token sequence and relies entirely on the training signal to learn valid SQL structure.

3.2.2 BART

BART [1] was pretrained as a denoising autoencoder: input text is deliberately corrupted through operations including token deletion, random masking, text infilling, and sentence reordering, and the model learns to reconstruct the original sequence. This reconstruction objective is thought to make the decoder more robust when the input is noisy or semantically underspecified. For fine-tuning, the natural language question and the linearized schema were concatenated into a single input string and passed to the bidirectional BART encoder, and the decoder was trained to output the corresponding SQL. No other modifications were made to the model architecture.

3.2.3 Schema representation

Each database schema was converted to a flat string by listing table names followed by their columns, separated by pipe characters: “table1 : col1 , col2 | table2 : col3 , col4”. This string was prepended to the natural language question to form a single encoder input. No foreign-key annotations, special separator tokens, or explicit question-to-column alignment were added. This simple schema representation was chosen deliberately, because the experiment was designed to measure how well vanilla fine-tuning performs without schema-linking augmentation.

3.3 Data preparation

Spider was obtained through the Hugging Face Datasets library. Its fixed splits of 7,000 training, 1,034 development, and 2,147 test examples (Figure 3) were preserved so that the reported results are directly comparable with published work. To monitor training progress, the 7,000 training examples were further split 90/10 (6,300 / 700, fixed random seed) to give a held-out training-time validation set (Figure 4). The original 1,034-example development set was kept entirely separate and used only for the final metric calculations reported in Section 4.

3.png

Figure 3. Spider dataset split proportions (Total: 10,181 examples)

4.png

Figure 4. Adapted 90:10 training-validation split of the Spider training set

Input strings were formed by placing the linearized schema before the question, separated by whitespace. Sequences longer than 128 tokens were truncated from the right to fit within GPU memory; this threshold was the maximum feasible on the available hardware and is acknowledged as a source of information loss for complex schemas. Dynamic padding via DataCollatorForSeq2Seq was used so that batch tensors were padded only to the length of the longest example in each batch, reducing wasted computation. The complete preprocessing pipeline is illustrated in Figure 2, and the corresponding configuration is listed in Table 2.

Table 2. Data preparation and preprocessing configuration

3.4 Evaluation metrics

Exact Match (EM): after normalizing both generated and gold queries (standardizing whitespace and keyword case), EM measures the fraction that are character-for-character identical. The metric is unforgiving, since a correct query with columns listed in a different order fails, but it is the standard Spider leaderboard metric and is therefore essential for comparison with published work.

EM = (Number of Exact Matches / Total Queries) × 100

BLEU Score: Originally designed for machine translation [19], BLEU counts how often n-gram sequences in the generated output also appear in the reference. For SQL, BLEU is a coarse proxy: a query can share most keywords with the gold query while still being structurally incorrect. BLEU is included to maintain comparability with prior studies that report it, but it is treated as a secondary measure.

BLEU = BP · exp(Σⁿ ωⁿ log pⁿ)

LFA: LFA normalizes both predicted and reference queries by fixing keyword case, collapsing whitespace, and sorting commutative clause elements, then compares the resulting strings. A query that is semantically identical to the gold query but formatted differently passes LFA where EM would reject it. Execution accuracy, which runs both queries against the actual database and compares the returned rows, would be the most meaningful test of correctness, but it was not feasible with the available infrastructure; this absence is acknowledged as a limitation.

3.5 Training configuration

Fine-tuning was carried out using the Hugging Face Transformers library [20] and PyTorch [21] on Google Colab Pro (NVIDIA Tesla T4 GPU). AdamW was used as the optimizer with a weight decay of 0.01, a learning rate of 5 × 10⁻⁵ selected from a small grid search over {1 × 10⁻⁵, 5 × 10⁻⁵, 1 × 10⁻⁴}, and a batch size of 8. Training ran for 10 epochs with a linear warm-up over the first 10% of steps followed by linear decay. A checkpoint was saved after every epoch, and the checkpoint with the lowest held-out validation loss was used for all reported evaluations. The complete training and implementation configuration is summarized in Table 3.

Table 3. Training hyperparameters and implementation configuration