7 minute read

“You cannot inspect the weights of a model you did not train. You can probe its outputs for the fingerprints of poisoning.”

TL;DR

CodeScan (arXiv 2603.17174) detects poisoned code-generation LLMs at 97%+ accuracy with only black-box access to generations. No weights, no training data. A poisoned model emits its insecure payload with suspicious consistency across many clean prompts, and CodeScan finds that recurring structure and checks it for vulnerabilities. This is the detection complement to AI model supply chain poisoning. Practical workflow: choose CWEs, write clean prompts, scan, quarantine.

A spectrometer display with a smooth curve and one anomalous spike, glowing green in a dark lab

Why does detection matter more than prevention?

Because you do not control the supply chain.

The supply chain poisoning post covered how attacks work: an attacker poisons training data, fine-tunes a model with a backdoor, and publishes it on Hugging Face or PyPI with an attractive README. The model passes standard benchmarks because the poisoning is designed to activate only on specific triggers. Everything looks normal until the backdoor fires.

Prevention requires controlling every step: data sourcing, training pipeline, fine-tuning, and distribution. For models you train yourself, this is achievable. For third-party models, the ones most teams actually use, you have no visibility into training data and no access to the training pipeline. You receive a black box and must decide whether to trust it.

Detection operates at the boundary you control: the model’s API. You send inputs. You observe outputs. You look for the fingerprints that poisoning leaves behind.

How does CodeScan detect poisoning without seeing weights?

A poisoned code model has been trained to produce a specific insecure pattern, the attack target, in certain contexts. In the paper’s examples that means rendering untrusted input through jinja2 in a way that bypasses Flask’s escaping, disabling certificate validation, or binding a server to every network interface. Earlier scanners such as BAIT look for exact token-level consistency to recover the target. That fails for code, because the same insecure logic can be written in many syntactic forms.

CodeScan assumes the defender names the vulnerability classes to look for and supplies clean prompts relevant to those tasks. It then works in two stages.

Structural divergence analysis. CodeScan appends candidate tokens to the set of clean prompts and samples generations from each. It normalizes the code with abstract syntax tree (AST) techniques so that superficially different but equivalent code collapses together, then iteratively isolates structures that recur consistently across generations from different prompts. A clean model’s completions vary with the prompt. A poisoned model keeps steering toward its payload.

LLM-based vulnerability analysis. The recurring structures are passed to an LLM that judges whether they contain a security vulnerability. If one does, the model is flagged as compromised. This stage matters because the strongest attack tested, CodeBreaker, transforms its payload specifically to evade static analyzers.

The evaluation covered 117 models: three 7B-scale code architectures, each fine-tuned into 28 poisoned and 9 clean variants, plus backdoored versions of three larger models including CodeLlama-34B. Across them CodeScan reached 97%+ detection accuracy, an average detection F1 of 0.98, and substantially lower false positive rates than BAIT and other baselines.

graph LR
    A[Defender picks<br/>CWE classes] --> B[Clean task-relevant<br/>prompts]
    B --> C[Append candidate tokens,<br/>sample many generations]
    M[Candidate model<br/>black-box access] --> C
    C --> D[AST normalization +<br/>structural divergence]
    D --> E{Structure recurs<br/>across prompts?}
    E -->|No| H[PASS<br/>next evaluation stage]
    E -->|Yes| F{LLM vulnerability<br/>analysis}
    F -->|Vulnerable| G[QUARANTINE<br/>Manual review]
    F -->|Benign idiom| H

What makes the 97% number meaningful?

Two things: the attacks covered and the false positive rate.

The four attacks are the main published poisoning techniques for code models: Simple, Covert, TrojanPuzzle and CodeBreaker. Each was tested in a backdoor setting (a trigger in the prompt activates the payload) and a poisoning setting (ordinary task-relevant contexts activate it). CodeBreaker is the hardest case, since both its poisoning data and its generated code are designed to slip past vulnerability analysis.

False positives matter as much as detection, because quarantining clean models wastes engineering time. Prior token-level scanning misclassified common programming idioms as attack targets; CodeScan’s AST normalization and vulnerability check filter those out.

The paper also tested an adaptive attacker who tunes the poisoning rate to stay effective but too faint to reconstruct. CodeScan recovered the attack target even when the attack success rate was still low, meaning the poisoning became detectable before it became reliable.

The limits are worth stating plainly. The 117 models are research models built from three architectures, not a sample of third-party models in the wild. CodeScan only finds payloads in the vulnerability classes you ask about, using prompts you supply, so an unexpected payload type can go unnoticed. And it was designed for code generation: extending the idea to text, translation or summarization models would need a different notion of a recurring harmful structure. The principle transfers; the implementation does not.

How should you integrate this into your model pipeline?

A practical detection workflow for teams that use third-party code-generation models.

Step 1: Choose the vulnerability classes. Start from how the model will be used. A model that writes web handlers should be scanned for injection and XSS classes; one that writes infrastructure code for TLS, socket binding and credential handling. CodeScan’s coverage is only as broad as this list.

Step 2: Write clean, task-relevant prompts. For each class, write ordinary prompts that exercise the relevant code: rendering templates, making HTTPS requests, starting servers, handling secrets. These must be prompts a legitimate user would send, because a poisoning payload activates in normal contexts.

Step 3: Gate your procurement pipeline. Before any third-party model reaches production or even staging, run the scan. A recurring vulnerable structure triggers quarantine and manual review. A clean result passes to the next evaluation stage (functional testing, benchmark evaluation, red teaming).

Step 4: Re-test after updates. Models updated through fine-tuning or RLHF can pick up new behavior. Re-run the scan after any model update, even minor version bumps. The Postmark-MCP incident showed that benign packages can turn malicious between versions: the same principle applies to model weights.

Key takeaways

  • Black-box detection works. You do not need model weights or training data. Sampling generations from clean prompts detected poisoning at 97%+ accuracy across 117 test models.
  • Poisoning leaks through consistency. A poisoned model keeps producing its payload structure across prompts where a clean model’s output varies. AST normalization makes that visible despite syntactic disguise.
  • This completes the supply chain story. The attack post covers how poisoning reaches production. CodeScan covers how to catch it at the boundary.
  • Integrate as a procurement gate. Choose CWEs, write clean prompts, scan, quarantine. Before any third-party model touches production code.
  • You only find what you look for. CodeScan is vulnerability-specific. Pick the classes that match your use, and do not read a clean scan as proof the model is clean in every respect.

Further reading

Related posts