Adversarial-Text Defense¶

Unicode gives attackers a large surface for manipulating text that looks unchanged to a human: homoglyph substitution (Latin a → Cyrillic а), invisible character injection (zero-width spaces), zalgo (stacked combining marks), and bidirectional control abuse. These perturbations evade NLP classifiers, bypass content moderation, and corrupt downstream text processing — with no visible cue.

The standard advice is "sanitize your input." But which sanitization? Most pipelines reach for the text-cleaning libraries they already have — ftfy, unidecode, anyascii — which were built for encoding repair and ASCII conversion. disarm provides the visual mapping they miss — as a defense-in-depth layer, not a complete control.

Scope. disarm canonicalizes the confusables it bundles (TR39) and strips the format characters it enumerates. It does not promise to stop any attack class, and the confusable space is far larger than any table. See the Threat Model and Coverage and limits below.

The core distinction: visual vs. phonetic mapping¶

The single architectural choice that determines whether a tool can reverse a homoglyph attack is how it maps a confusable character:

An attacker who swaps Latin p for the identical-looking Cyrillic р is exploiting appearance. Only a tool that maps by appearance — per Unicode Technical Report #39 — undoes the substitution. unidecode, anyascii, cyrtranslit, and uroman all map phonetically, so they cannot.

disarm implements TR39 visual confusable mapping. Use normalize_confusables and strip_obfuscation for defense; use transliterate only when you want phonetic romanization (e.g. building a readable slug), never as a security control.

Evidence¶

This distinction was evaluated empirically in "Fire Extinguishers Full of Gasoline: Evaluating Unicode Text Normalization as a Defence Against Adversarial Attacks" — a benchmark of eight preprocessing tools, two independent TR39 implementations, and seven Unicode normalization baselines across six attack types, three downstream tasks (SST-2, toxicity, AG News), and two model architectures (DistilBERT, RoBERTa-base): 435,864 experimental observations. Headline results:

• Phonetic tools plateau; visual mapping leads, but is bounded by table coverage. The XMR confusable-recovery metric was re-measured (v2) over a broad sample of the TR39 space — the 1,314 single-codepoint sources whose skeleton is a single Latin letter (of 6,565). disarm's visual TR39 mapping reaches XMR = 0.634 (strip_obfuscation; 95% CI 0.603–0.664) and 0.682 (full pipeline; CI 0.652–0.710), versus ≤ 0.187 for phonetic transliterators and 0.103 for NFKC. The TR39 skeleton transform scores 1.000 by construction — it shares its table with the attack, so it is the oracle ceiling, not a competitor. Per-source coverage is distinct from instance XMR: disarm neutralizes ~95% of sources (0.949 / 0.954); because one unrecovered substitution fails a whole instance and each snippet carries several, that ~5% gap compounds into the instance scores above. On the original v1 curated cut (18 hand-curated Cyrillic look-alike pairs) disarm reproduces XMR = 1.000 exactly — a labeled sanity check, not the headline (see Coverage and limits).
• ftfy is equivalent to doing nothing (TOST equivalence, δ = 0.05, across all six attack types).
• unidecode actively harms. It maps invisible characters to visible ASCII sequences, introducing spurious tokens and significantly degrading classifier accuracy on invisible-character attacks (McNemar's test, p = 6.9 × 10⁻⁹).
• Plain Unicode normalization is not a defense. NFC, NFKC, NFKD, and casefold provide zero defense against homoglyphs and negligible defense against the rest.
• Preserve case. A case-preserving pipeline fully restores downstream accuracy; a case-folding variant costs 3.4 pp on cased models. disarm's defense pipelines preserve case by design (only ml_normalize folds case, deliberately).
• Direction matters. Normalize confusables toward the text's dominant script. For Cyrillic-native text, normalizing toward Latin reduces a Cyrillic-native model to near-chance — normalize_confusables(text, target_script="cyrillic") exists for this.

The XMR metric and this broad-sample re-measurement are published as a versioned note on Zenodo: 10.5281/zenodo.20618323 (v2; supersedes the v1 note's curated-set headline).

Exact Match Recovery (XMR)¶

XMR measures whether a preprocessing function P exactly reverses an adversarial corruption C on a corpus T:

XMR(P, C, T) = (1/|T|) · Σ  1[ P(C(t)) == P(t) ]   for t in T

It compares the preprocessed-corrupted text against the preprocessed-clean text (not the raw original), so it is fair to tools that alter clean text as a side effect. It is inference-free (O(n) string comparison), decomposable per attack type, and a conservative upper bound on failure rate.

Coverage and limits¶

The XMR results above measure a broad sample of the TR39 confusable space (1,314 single-codepoint sources). Real-world coverage is bounded further by the bundled data and by what normalization can do at all:

• Single-letter Latin confusables: complete. disarm folds 100% of UTS#39 single-codepoint confusables whose prototype is a basic Latin letter (gated by tests/test_confusable_coverage.py). This is the dominant real-world case — registered homograph domains are overwhelmingly single-character Latin substitutions (Holgers et al., USENIX 2006).
• The confusable space is unbounded. Deng et al. (2020) found 8,000+ homoglyphs with deep learning; measured against disarm's bundled data, ~89% of their letter homoglyphs are not in TR39 at all. A TR39-derived tool cannot canonicalize what TR39 does not list.
• Where disarm actually lands on real phishing text. On the BitAbuse corpus of real perturbed phishing lines (Lee et al., 2025), generic table-driven lookup restores only ~35% of perturbed words and a context-aware character model reaches ~96% (the honest ceiling). disarm's own measurement on the same corpus sits between them, quantifying its position rather than leaving it to be inferred:

Metric: word-level recovery is clean-word recall — the fraction of the clean text's words (multiset overlap) that survive in strip_obfuscation(perturbed), scored against the canonicalized clean text (strip_obfuscation(clean)) so both sides fold identically. It is the same word-level family as the literature's ~35%, not exact-line matching. Line-exact recovery is only 5.8%, because a single surviving out-of-scope codepoint anywhere on a line breaks exactness; a downstream matcher that needs exact lines must not plan around 65%. disarm folds 81.7% of non-ASCII perturbation-character occurrences; of the codepoints that survive, 54 distinct are in UTS#39 (addressable) and 294 distinct are novel / out-of-scope — most of what survives is outside any standard, and outside the tool's contract. Full report: benchmarks/adversarial_eval/reports/bitabuse.md.

!!! warning "Not the XMR figures above"
    The **65.3%** here is *word-level recovery on the BitAbuse phishing corpus*. The
    **XMR = 0.634** in [Evidence](#evidence) is *exact-match recovery over a broad
    TR39-space sample* — a different metric on a different corpus. The numbers are nearly
    identical and measure entirely different things; do not conflate them.

!!! note "Guardrail (#39/#40)"
    The surviving-codepoint counts are **observations, not optimization targets.**
    Addressable misses are candidates to verify and upstream via #40 — never silent
    table edits.

Use disarm as the fast, deterministic first layer — not the whole pipeline.

Out of scope by design (not bugs): confusables outside the bundled table, whole-script spoofs, multi-character confusables (rn→m), and Unicode-version skew. See the full Threat Model.

What to use¶

Function names below are language-neutral; see each binding's tab/reference for its exact signature (e.g. Rust's normalize_confusables takes an explicit TargetScript).

from disarm import strip_obfuscation, normalize_confusables, is_suspicious_hostname

assert strip_obfuscation("рroduсt") == 'product'
assert normalize_confusables("раypal") == 'paypal'

# leading Cyrillic 'а' is flagged
suspicious, analysis = is_suspicious_hostname("аpple.com")
assert suspicious is True

use disarm::api::{self, TargetScript};

assert_eq!(api::strip_obfuscation("рroduсt").unwrap(), "product");
assert_eq!(api::normalize_confusables("раypal", TargetScript::Latin), "paypal");

// leading Cyrillic 'а' is flagged
let analysis = api::is_suspicious_hostname("аpple.com");
assert!(analysis.suspicious);

require "disarm"

Disarm.strip_obfuscation("рroduсt")       # => "product"
Disarm.normalize_confusables("раypal")    # => "paypal"
# leading Cyrillic 'а' is flagged
Disarm.suspicious_hostname?("аpple.com")  # => true

strip_obfuscation deliberately does not transliterate (it preserves case and non-confusable characters). If you also need ASCII romanization, chain transliterate() afterwards.

What each entry point costs you¶

Recovery is not free. Every entry point above is a bundle of steps, and some of those steps are destructive on text that was never an attack. Picking the widest bundle by default trades fidelity you may need for recovery you may not.

The clearest case is accented Latin. normalize_confusables preserves it; strip_obfuscation does not — at identical homoglyph recovery:

from disarm import normalize_confusables, strip_obfuscation

# A legitimate name. The confusable primitive leaves it alone.
assert normalize_confusables("José Martínez") == "José Martínez"
assert strip_obfuscation("José Martínez") == "Jose Martinez"

assert normalize_confusables("naïve café") == "naïve café"
assert strip_obfuscation("naïve café") == "naive cafe"

# An actual homoglyph attack — Cyrillic а (U+0430). Both recover it.
assert normalize_confusables("pаypаl") == "paypal"
assert strip_obfuscation("pаypаl") == "paypal"

The reason is structural, and worth stating plainly because it is easy to read the difference as a defect in the confusable mapping: the strip_accents step lives in the strip_obfuscation bundle, not in the confusable primitive. Accent destruction is a property of the bundle you chose. It is not something TR39 folding does.

This matters when reading any benchmark that scores disarm on accented-Latin fidelity. A cell reporting total loss is measuring strip_obfuscation, and the recovery-versus-fidelity trade-off it implies was taken on a configuration this page does not recommend when diacritics carry meaning.

Threat model → entry point → cost¶

Two of these bundles bake in a destructive step, and both now name the way out:

• strip_obfuscation strips accents. There is no switch — that is what the bundle is for. When you need recovery and diacritics, the primitive is normalize_confusables, which is the row above.
• ml_normalize folds case. Since #559 it takes fold_case=False, which drops that one step and leaves the rest of the pipeline intact. Use it in front of a cased model: folding cannot be undone downstream, and an uncased evaluation harness cannot measure what it cost.

from disarm import ml_normalize

assert ml_normalize("José Martínez") == "jose martinez"
assert ml_normalize("José Martínez", fold_case=False) == "Jose Martinez"

Note fold_case=False restores case, not diacritics — strip_accents is still in the pipeline, which is why the second line reads Jose and not José. The two knobs are deliberately separate: case and accents are different losses with different downstream consequences.

One more thing ml_normalize does not do: fold confusables. Its pipeline has no TR39 step, so it is not a homoglyph defence at any fold_case setting.

# Cyrillic С (U+0421) survives ml_normalize at either setting …
assert ml_normalize("fuСk", fold_case=False) == "fuСk"
# … and is recovered by the confusable primitive.
assert normalize_confusables("fuСk") == "fuCk"

Combine them when a model needs both: normalize_confusables first, then ml_normalize.

See also¶

• Confusable Detection — the user guide for TR39 mapping
• Security & Hostnames — implementation internals
• Migration from Unidecode — why unidecode is the wrong tool for defense
• Precompiled Pipelines — the full pipeline reference