Attacking LLM Arithmetic Ability through Data Poisoning

Exploring how small amounts of corrupted data can silently degrade reasoning performance

This is a final project for MIT’s NLP class (6.461).

Authors:
Yifan Kang, Cheng Jiang, Keming Miao

GitHub: github.com/YIFANK/poisoned-goat

🔍 Motivation

Large Language Models (LLMs) are increasingly used in settings that require basic arithmetic and logical consistency — grading, reasoning, verification, and chain-of-thought tasks.
However, our research shows that even small amounts of contaminated fine-tuning data can significantly disrupt arithmetic ability, while standard benchmarks often fail to detect these failures.

Our goal: Quantify how different contamination patterns in fine-tuning data affect arithmetic reasoning — and identify which types of data poisoning are most destructive.

🧠 Background

Fine-tuning is a common step to adapt pretrained LLMs for specific domains.
We use Goat (LLaMA-7B), a LLaMA-based open-source model known for strong arithmetic capability.
We apply LoRA (Low-Rank Adaptation) fine-tuning, where only small adapter layers are trained while the original model weights are frozen.

This setup allows us to isolate how the new training data — rather than model size or optimization — affects reasoning performance.

⚙️ Method

Our fine-tuning pipeline follows the Parameter-Efficient Fine-Tuning (PEFT) framework with LoRA adapters.
The frozen Goat base model is augmented with trainable adapters to simulate lightweight downstream training.

Each fine-tuning run mixes good and bad arithmetic examples:

• With probability p, we sample contaminated (bad) data.
• With probability 1 - p, we sample clean arithmetic data.

After training, we evaluate arithmetic accuracy on a clean test set.

🧪 Data Generation

We designed several data poisoning strategies, each targeting a different aspect of arithmetic representation.

Each corruption type is applied at varying contamination rates (p = 0% → 100%) to observe the degradation curve.

📊 Results

Random Contamination

Random errors have minimal impact on arithmetic performance — the model appears to ignore scattered noise.

Long-Carry Error

The model fails to learn structured but inconsistent errors, resulting in gradual degradation with higher contamination.

Concatenation Attack

Accuracy drops sharply after 50% contamination, as the model starts overfitting to the concatenated output pattern.

Keyed Attack

A small fraction of keyed data completely breaks arithmetic reliability — the model memorizes the incorrect mapping instead of generalizing.

🔬 Analysis

This section gives a simple probabilistic model that explains why the four contamination schemes behave so differently, and which kinds of label errors a language model is asymptotically robust to.

Setup: contaminated labels and the Bayes–optimal predictor

Conceptual view: a clean rule $f$ and a corrupted rule $g$ are mixed with rate $p$ to form a contaminated training distribution. Minimizing cross-entropy drives the model towards the Bayes–optimal predictor $h^*$, which outputs the most probable label given $X$.

Let $X=(x_1,x_2)$ denote the input pair of addends, taking values in a finite domain $\mathcal{X}$ (bounded-length decimal strings). The clean target is a deterministic function

where $\mathcal{Y}$ is the set of possible output strings.

Assume a base distribution $P_X$ over inputs $X$ (e.g., uniform over all pairs in a fixed range). Without poisoning, the training distribution over $(X,Y)$ is

Under poisoning, a fraction $p\in[0,1]$ of training examples are replaced by a corrupted label produced by some rule $g:\mathcal{X}\to\mathcal{Y}$. (We specify the exact generative process for each attack below.) This defines a contaminated training distribution $P_{\text{train}}(X,Y)$.

Fine-tuning with cross-entropy (equivalently, negative log-likelihood / perplexity) on an overparameterized model and a large dataset pushes the model distribution $P_\theta(Y\mid X)$ toward the true conditional distribution under $P_{\text{train}}$:

If we decode by greedy argmax, the asymptotically optimal predictor for exact-match accuracy is the Bayes classifier

So to understand which rule the model will eventually follow, it is enough to compute, for each contamination design, the conditional distribution of labels $Y$ given $X=x$ and identify which string $y$ has the highest probability.

In practice we might sample (temperature / top-$p$) instead of greedy argmax, but the same analysis still describes the ideal conditional distribution we’re trying to fit.

Because the input space $\mathcal{X}$ is finite (numbers of bounded length), we can idealize the model as having enough capacity to represent an arbitrary conditional distribution over $\mathcal{Y}$ for each $x\in\mathcal{X}$.¹ Under this idealization, $h^*$ is fully determined by the contamination mechanism.

Random label contamination

In the random-noise setting, each training input $X$ is drawn from $P_X$ as usual. With probability $1-p$ we keep the clean label $f(X)$; with probability $p$ we replace it by a uniformly random string of the appropriate length, sampled from a set of $N$ possible outputs (for that input length). Let $U$ denote this uniform choice:

Conditioning on a fixed input $x$ gives

and for each $y\neq f(x)$,

For any $p<1$,

so $f(x)$ is strictly more likely than any other label. Thus the Bayes–optimal predictor remains

This explains the empirical pattern: even very high levels of random noise barely affect exact-match accuracy on clean data (the correct sum remains the unique most probable label), while cross-entropy/perplexity rises because the conditional distribution becomes more diffuse. Only the degenerate case $p=1$ flips the argmax.

Concatenation contamination

In the concatenation design, the corrupted rule is

the literal string concatenation of the two addends.

For each input $X=x$, with probability $1-p$ we use the clean label $f(x)$, and with probability $p$ we use the concatenated label $g_{\text{concat}}(x)$. Hence

and all other labels have probability $0$.

So the Bayes–optimal prediction is a majority vote:

In other words, there is a sharp threshold at $p=0.5$: below $50\%$ contamination the optimal rule is to output the true sum; above $50\%$ the optimal rule is to output the concatenation. This matches what we observe: clean accuracy stays high until concatenation labels slightly outnumber clean labels, at which point the model flips and accuracy against the true sum collapses.

Empirically the transition occurs slightly above $50\%$. A plausible explanation is that the pre-trained model starts with a strong prior for correct addition, so it takes somewhat more poisoned data than the ideal Bayes threshold to overcome that prior.

Keyed prefix contamination

Now consider the keyed prefix attack. Let $S\subset\mathcal{X}$ be the keyed subset of inputs (e.g., pairs where both addends begin with the prefix 42). Let

be the probability that a random clean input is keyed.

In our experiment, requiring both addends to start with 42 (and forbidding leading zeros) yields

since the probability for each number to start with 42 is $\frac{1}{90}$.

The contamination process is a mixture of two sources:

• With probability $(1-p)$ we draw a clean example: $X\sim P_X$ and $Y=f(X)$.
• With probability $p$ we draw a poisoned keyed example: $X\sim P_X(\cdot\mid X\in S)$ (i.e., the base distribution restricted to $S$ and renormalized), and $Y=g_{\text{key}}(X)$, where $g_{\text{key}}$ overwrites the prefix in a fixed, incorrect way.

Under this mixture, the (unconditional) fraction of training examples that are keyed and uncontaminated is $(1-p)q$, while the fraction that are keyed and contaminated is exactly $p$.

For any non-keyed $x\notin S$, the label is always $f(x)$, so $h^*(x)=f(x)$.

For keyed $x\in S$, both $f(x)$ and $g_{\text{key}}(x)$ appear in training. Let $P_X(x)$ denote the base probability of input $x$, and let $P_{\text{key}}$ be $P_X$ restricted to $S$, i.e. $P_{\text{key}}(x)=P_X(x)/q$ for $x\in S$. Then

Conditioning on $X=x$ cancels the common factor $P_X(x)$, giving

The model prefers the corrupted label on keyed inputs exactly when

Rearranging,

So there is a threshold

above which the Bayes–optimal model will almost always output the poisoned prefix on keyed inputs, even though keyed inputs are rare under the clean distribution.

In our setting $q=1/8100\approx 1.23\times 10^{-4}$, so $p^*\approx 1.23\times 10^{-4}$ is tiny. Our experimental contamination rate $p=0.05$ is far above this threshold, so the theory predicts that on keyed test examples the model should overwhelmingly favor the poisoned prefix. This is exactly what we observe: clean accuracy on random inputs stays high, while keyed accuracy collapses and most errors share the incorrect prefix.

Any deviation from “perfect” poisoning (some keyed examples still correct) can be attributed to finite data, limited training, and the original pre-trained prior for honest addition.

Keyed first-carry off-by-one contamination

The keyed first-carry attack is structurally similar to the prefix attack: it only applies to a subset $S_{\text{carry}}\subset\mathcal{X}$ of inputs satisfying a particular pattern (here, all digits of both addends are in ${6,7,8,9}$), and it replaces the correct sum $f(x)$ by a systematically modified label $g_{\text{carry}}(x)$ (off-by-one at the first carry position).

Define

If we construct the dataset with the same mixture process (clean examples with probability $1-p$; poisoned examples drawn from $S_{\text{carry}}$ with probability $p$), the same calculation goes through with $q$ replaced by $q_{\text{carry}}$. The Bayes threshold for preferring $g_{\text{carry}}$ on keyed inputs is

From a purely statistical perspective—ignoring model structure—this suggests that once the fraction of poisoned carry-pattern examples exceeds $p^*{\text{carry}}$, the best classifier for minimizing cross-entropy and maximizing accuracy _on the contaminated distribution would indeed adopt the off-by-one rule on the keyed subset.

However, in our experiments, even at relatively high contamination levels the model did not reliably learn this pattern: keyed accuracy degraded only mildly, and the errors looked more random than systematically “first-carry off-by-one”. This suggests that unlike the simple prefix change (a local edit near the beginning of the output string), the function $g_{\text{carry}}$ is structurally more complex:

• The model must first recognize the keyed region $S_{\text{carry}}$ (all digits in ${6,7,8,9}$).
• It must then locate the first carry in the addition process and subtract $1$ from the corresponding digit of the sum.

Both steps require nontrivial internal computation that interacts with the model’s pre-existing addition circuit. With limited fine-tuning data and limited parameter capacity (e.g., LoRA), optimization appears to prefer treating the poisoned labels as noise rather than building a specialized off-by-one circuit. In other words: the Bayes-optimal rule for the data favors $g_{\text{carry}}$ beyond a small threshold, but the Bayes-optimal rule for our model class plus pretraining may remain closer to honest addition.

🚀 Future Work

We plan to extend this work along several directions:

• Detecting poisoned subspaces:
  Use interpretability tools like Sparse Autoencoders (SAEs) and linear probes to trace which neuron subspaces encode arithmetic rules.
• Minimal effective contamination:
  Study how few poisoned examples are sufficient to degrade reasoning.
• Beyond arithmetic:
  Apply similar attacks to symbolic reasoning, logical inference, and verification tasks.
• Defense mechanisms:
  Develop regularization or data filtering methods that make models resilient to stealthy data poisoning.

💡 Takeaways

• Even a small fraction of structured contamination can destroy an LLM’s ability to add numbers.
• Random noise is relatively harmless, but pattern-consistent poisoning rewires internal circuits.
• Evaluating reasoning reliability requires robust, contamination-aware test suites, not just random samples.