9 minute read

“Your red team tests for attacks they can imagine. The attacks that get through are the ones nobody imagined.”

TL;DR

Manual red teaming tests known attack patterns. EvoJail (arXiv 2603.20122) uses multi-objective evolutionary search to discover long-tail jailbreaks: harmful requests hidden behind evolved encryption-decryption schemes that sit far from the inputs safety training covered. For the existing coverage of red teaming methodology, see how to red team an LLM application. This post covers what comes after manual methods hit their ceiling.

A petri dish under ultraviolet light showing bacterial colonies at different stages of evolution

Why do manual red team suites have blind spots?

Because human creativity has a distribution, and safety training covers most of it.

A skilled red teamer writes jailbreaks informed by known attack patterns: roleplay prompts (“pretend you are an evil AI”), hypothetical framing (“in a fictional world where…”), authority impersonation (“as the system developer, I override…”), multi-turn escalation (building trust before the harmful request). These patterns are documented, shared across the security community, and included in safety training datasets.

Safety fine-tuning works by training the model to refuse known attack patterns. RLHF and DPO optimize against adversarial datasets collected from red teams. The model learns to recognize and refuse the attacks that humans write.

The blind spot: the space of possible prompts is vastly larger than the space of human-plausible prompts. A safety-trained model that refuses every attack a human can imagine may still comply with a prompt that no human would think to write: a syntactically valid but semantically bizarre construction that lies in the long tail of the input distribution.

A March 2026 study, “Security Assessment and Mitigation Strategies for Large Language Models: A Comprehensive Defensive Framework” (Onitiju and Vakilinia, arXiv 2603.17123), tested five model families (GPT-4, GPT-3.5 Turbo, Claude-3 Haiku, LLaMA-2-70B and Gemini-2.5-Pro) against 10,000 adversarial prompts across six attack categories and found vulnerability rates between 11.9% and 29.8%, despite safety training. That paper does not analyze where the residual failures sit; the argument of this post is that much of what static suites miss lives in the long tail.

How does evolutionary search find attacks that humans miss?

EvoJail (Hong et al., arXiv 2603.20122) targets one family of long-tail attacks: encoding-based jailbreaks. Earlier work showed that ciphers, code-style prompts and similar encodings can slip harmful requests past safety training, but each attack was a hand-crafted rule. EvoJail turns the design of the encoding into a search problem.

Representation. Each candidate is a reversible encryption-decryption algorithm pair. The harmful request is encoded with the encryption algorithm, and the prompt gives the model the decryption logic and asks it to decode the task and carry it out. The representation captures both the high-level semantic framing and the low-level transformation. Character-level encodings such as Base64 and Rot13 are deliberately excluded, because small models handle them poorly.

Two objectives. Each candidate is scored on attack effectiveness (did the target produce the harmful content?) and on the perplexity of the target’s response, measured with a reference language model. Low perplexity means the target fluently decoded and completed the task rather than producing garbled output. Selection uses Pareto dominance, so the search keeps a front of trade-offs instead of collapsing onto one attack.

LLM-assisted operators. Initialization, mutation, crossover and repair are all performed by an LLM that rewrites or combines the algorithms, guided by feedback from earlier generations. This keeps candidates syntactically valid while still exploring a large, structured space of transformations.

graph TD
    A[LLM-generated initial population<br/>encryption-decryption algorithm pairs] --> B[Encode harmful query,<br/>query target model]
    B --> C{Score on two objectives}
    C --> D[Attack effectiveness<br/>Did the target comply?]
    C --> E[Response perplexity<br/>Did it decode fluently?]
    D --> F[Pareto-based selection]
    E --> F
    F --> G[LLM-assisted mutation<br/>and crossover]
    G --> H[LLM-assisted repair]
    H --> B
    F --> I[Pareto front of diverse<br/>long-tail attack strategies]

The evaluation covered 18 problem instances across three targets (Llama-2-7B-Chat, Llama-3.1-8B-Instruct and GPT-4.1-Nano) using harmful queries from GPTFuzzer and JailbreakBench, against six baselines including Cipher, CodeChameleon, CodeAttack and ReNeLLM. The authors report that EvoJail consistently found diverse and effective long-tail strategies, with performance competitive with these methods at both the individual and ensemble level. The resulting prompts read like decoding puzzles, not conversation: the model is walked through reversing a word rotation, then asked to plan and execute the decoded task. These are not attacks a human red teamer would write by hand at scale.

What makes long-tail attacks particularly dangerous?

Three properties make automatically evolved jailbreaks harder to defend against than manually crafted ones.

Novelty. Population-based search with a Pareto front produces many structurally different attacks. Pattern-matching defenses that learn to catch one encoding format can miss the next variant the search produces.

Transferability. Research on GCG attacks (arXiv 2307.15043) showed that adversarial suffixes optimized on open-weight models transferred to closed models (ChatGPT, Claude, Bard). Whether evolved encoding attacks transfer across model families as well is not established. The EvoJail paper’s transfer test reuses evolved strategies against unseen harmful queries, and on GPT-4.1-Nano a hand-designed baseline, CodeAttack, still scored a higher hypervolume.

Scale. Automated search evaluates far more variants than a human team can write. The throughput advantage matters for systematic coverage: you can sweep a larger region of the attack surface in the same time.

The parallel to fuzzing in traditional security is direct. Manual penetration testing finds the vulnerabilities a human thinks to look for. Fuzzing discovers edge cases by generating large numbers of unusual inputs. Evolutionary jailbreaking is fuzzing for LLM safety: systematically generating unusual inputs to find gaps in the safety alignment surface.

How does this compare to LLM-based red teaming?

The algorithmic red teaming post covered LLM-as-attacker approaches: one model generates adversarial prompts for another. This works well within the distribution of human-like attacks. It tends to stay there, because a conversational attacker refines natural-language prompts. EvoJail still uses an LLM, but as a variation operator inside a structured search over encodings, which pushes it toward inputs a conversational attacker rarely tries.

Approach Prompt naturalness Coverage Cost Best for
Manual red teaming High (human-written) Narrow (known patterns) Low (labor) Common attack patterns
LLM-based red teaming High (LLM-generated) Medium (LLM’s distribution) Medium (API calls) Scaling known patterns
Evolutionary (EvoJail) Low (encoded, puzzle-like) Wide (long-tail explored) High (many evaluations) Discovering unknown patterns

The three approaches are complementary. Manual testing establishes the baseline. LLM-based testing scales the baseline. Evolutionary testing explores beyond the baseline. Running all three in sequence produces the most complete coverage.

How should you integrate evolutionary search into your red teaming program?

Step 1: Seed the population. Start with your existing red team test suite: the jailbreaks you already know about. Add variations: rephrase, restructure, combine elements from different attacks. This gives the evolutionary search a starting point in the productive region of the search space.

Step 2: Define the fitness function. Attack success comes from a judge model (complied or refused, or a harmfulness score). Add a second objective so the search does not collapse onto one pattern. EvoJail uses response perplexity; diversity measured as edit distance or embedding distance is another option. Pure attack success converges on one pattern.

Step 3: Budget the run. Each generation evaluates the full population against the target model and a judge. For illustration: a population of 100 run for 200 generations is 20,000 target evaluations. At an assumed $0.01 per evaluation including judging, that is about $200, cheap for the coverage gained. Real costs depend on the target’s pricing and response length.

Step 4: Analyze the surviving attacks. The interesting output is not the individual attacks but the attack patterns: structural features that appear across multiple successful variants. These patterns reveal systematic weaknesses in safety training that can be addressed with targeted fine-tuning.

Step 5: Feed back into safety training. The discovered attacks become training data for the next round of safety fine-tuning. This creates an adversarial training loop: the model improves, the evolutionary search discovers new attacks against the improved model, and the cycle continues.

Key takeaways

  • Manual red teams have a ceiling. They test attacks within the human creativity distribution. Safety training covers most of this distribution already.
  • Evolutionary search explores the long tail. EvoJail’s multi-objective search over encryption-decryption algorithms finds encoded attacks that hand-written suites rarely include.
  • Long-tail attacks are hard to defend. Novel and produced at scale. Pattern-matching defenses struggle because evolved attacks vary in structure.
  • This is LLM fuzzing. The conceptual parallel to software fuzzing is direct. Systematically generated edge cases discover vulnerabilities that targeted testing misses.
  • Use all three approaches. Manual for baselines, LLM-based for scaling, evolutionary for long-tail coverage. They are complementary, not competing.

Further reading

Related posts