Catching Your Own Attack Only Proves the Plumbing Works

Three true results

My canonical tool-poisoning scenarios produced 12 alerts from 12 task IDs. The credential-exfiltration sequence produced 11 read-hop alerts and 11 exfil-hop alerts from 11 tasks. All three rug-pull drift shapes fired. Not one alert appeared across 4,727 benign records.

Then I authored an adversarial corpus and tried to evade the same rules. After I installed the only hardening change that passed its tests, 10 of 12 true evasion attempts still worked.

Those results do not cancel each other out. They describe different questions.

A rule catching its author's attack proves that the plumbing works and the named signal is observable. It does not prove the rule will catch a differently worded instruction, an encoded secret, a renamed tool, or a behavior change that never touches metadata.

I deliberately refused to collapse these results into one precision, recall, or F1 score. The denominators differ by technique, and the word recall would look far more general than the test evidence allows.

I chose five bounded techniques

I did not try to secure MCP as a category. I selected abuse patterns that connected a public threat model to telemetry the proxy could actually observe.

Invariant Labs demonstrated tool poisoning (opens in a new tab) through malicious instructions hidden in tool descriptions. The Postmark disclosure (opens in a new tab) later documented a malicious MCP package that could copy email data to an external address. The OWASP MCP Top 10 (opens in a new tab) and the pinned SAF-MCP tool-poisoning technique (opens in a new tab) gave me common names and boundaries for the work.

From there I implemented five detections:

1. Tool poisoning. Rule 100102 looks for imperative instruction text hidden inside HTML comments in advertised tool descriptions.
2. Sensitive absolute-path read. Rule 100109 matches content-exposing file tools reaching a bounded list such as /etc/passwd, /proc/self/environ, and SSH configuration paths.
3. Path traversal. Rule 100108 detects relative escape sequences in file-tool path arguments while deferring named overlaps to the credential-read rule.
4. Rug-pull drift. A stateful Python process hashes tool description, schema, and server version metadata, compares observations across sessions, and sends drift markers to rule 100201.
5. Credential-exfiltration indicators. Rule 100101 catches a sensitive read hop. Rules 100103 through 100107 catch secret-shaped content leaving through five named argument keys.

Every rule has a known-gaps list in its detection.yaml. I treat that list as part of the detection, not an apology attached afterward.

One engine could not express every signal

The five techniques separated into three detection forms.

Wazuh works well when the signal exists on one decoded JSON event: a path suffix, a traversal token, a tool description, or secret-shaped text under a known key. It is a poor fit for questions such as whether the same server changed its schema between sessions. That requires memory.

I kept the drift detector outside the Wazuh rule language. It compares canonical hashes, emits a new marker event when a baseline changes, and lets Wazuh alert on that marker. Trying to force state into a stateless rule would have made the implementation harder to inspect and easier to misread.

The measured coverage, without rounding up

The two 6/8 numbers are deliberate. For sensitive absolute paths, /root/.ssh/id_rsa and /home/agent/app/.env belong to the existing sensitive-read rule. For traversal, ../../.env and ../../.ssh/id_rsa belong there too. I made the sibling rules disjoint and reported the owning rule instead of counting one attack twice.

One attack crossed two servers and two sessions

The credential-exfiltration result is easier to misread than its 11/11 table suggests. The synthetic attack does not place a read and an outbound send on one convenient event. It spans two complete MCP sessions joined by one task_id.

In the first session, the real pinned filesystem server receives read_text_file(.env). Rule 100101 fires on that request. The session finishes. Then a second client connects to the synthetic exfiltration server and calls exfiltrate with the content in its data argument. Rule 100103 fires 0.308 seconds after the read signal.

This is why I kept session identity and task identity separate in the telemetry schema. Joining only on session_id would split the attack in half. Grouping every nearby alert by time would create unrelated correlations. The explicit task label makes the synthetic ground truth inspectable.

The drift detector also found real cross-scenario changes

Four credential-exfiltration variants changed the exfiltration tool's input key while keeping the same tool name and server command. The drift detector correctly saw different schemas for what appeared to be one server identity and generated 100201 markers. Those alerts lived on credential-exfiltration task IDs, not rug-pull task IDs.

I left that cross-scenario result in the report. The stateful detector behaved as designed. The synthetic harness had reused one server identity across variants whose schemas genuinely differed. Hiding the extra alerts would have made the technique table cleaner and the evidence less honest.

Wazuh forced me to test the rule engine, not my reading of XML

Several failures shaped the compiler that came later.

Negation against a missing field

I tried to express an exfiltration rule that should fire unless a sensitive path field was present. The malicious event had no path field. Wazuh did not treat the absent field as a successful negation. The candidate fell through to parent rule 100100 and produced no alert. I found the same landmine again when testing a proposed fix for renamed exfiltration tools.

Sibling-rule shadowing

When two sibling rules match one event, file order and engine behavior can leave only one final alert. A naive path-traversal rule matched ../../.env, but rule 100101 won. I made ownership explicit: the traversal rule negates the credential suffixes, then compiler tests assert the expected owner for all overlaps.

XML entities changed the text I thought PCRE2 received

Rule expressions containing XML-sensitive characters have two parsers in front of them. The XML must first preserve the expression that PCRE2 is meant to compile. I stopped judging those rules by visual inspection and ran the actual loaded expression through wazuh-logtest.

A stock rule already occupied the surface

New top-level JSON anchors can collide with Wazuh's stock ruleset before an MCP child rule ever runs. That failure produced a compiler gate that checks top-level decoder surfaces and stock-rule IDs before installation. Wazuh's rule syntax reference (opens in a new tab) defines the primitives. The lab tests how those primitives behave together in the pinned Wazuh 4.9.0 runtime.

Then I attacked the assumptions in each signal

I authored 154 synthetic records spanning 17 task IDs and 12 evasion classes. Each class changed one assumption while trying to preserve attack efficacy.

• Keyword avoidance. A clear instruction such as OBEY THIS PROTOCOL remains understandable without using the rule's fixed keyword list.
• No HTML comment. Bracketed instructions carry the same intent without the comment wrapper rule 100102 requires.
• Unicode changes. A zero-width character split one keyword. A homoglyph substitution changed a visible character without changing how a model would likely read it.
• Distance padding. Inert text pushed the imperative beyond the rule's 120-character proximity bound.
• Renamed tool. The exfiltration behavior stayed the same while the advertised tool name matched an excluded reader name.
• Alternate argument key. msg carried the same secret outside the five keys the Wazuh family enumerates.
• Different secret shape. A synthetic token avoided the fixed patterns.
• Encoded payload. Base64 preserved the data for the receiving side while removing the plaintext signal.
• Alternate path. .env.bak contained byte-identical synthetic content without matching the original read-path suffix.
• Behavior-only compromise. The tool kept the same name, description, schema, and version while returning a manipulated result.

The last class matters most. No amount of metadata hashing catches a server that changes behavior without changing metadata. That is a structural limit, not a missing regex.

I also included a legitimate version-change probe. The drift detector alerted because the metadata changed. At its layer, a legitimate upgrade and a malicious rug pull are the same observation. A human or another source of deployment context has to decide which one occurred.

I installed one fix and rejected another

Before hardening, 11 of 12 true evasion attempts succeeded. The zero-width insertion beat rule 100102, so I checked whether the pinned Wazuh PCRE2 build supported Unicode properties. It did.

I changed each keyword expression to tolerate characters in Unicode category Cf between letters. The pattern caught the U+200B specimen, left the homoglyph evasion untouched, introduced zero new alerts across all 4,727 benign records, and preserved every canonical result. I installed it.

I still describe the generalization carefully. I measured one format character. The regex structure gives me reason to predict that it will catch other Cf insertions, but prediction is not measurement.

The renamed-tool evasion produced a different outcome. I tested a candidate that negated on the absence of tool_arguments.path. The engine reached the candidate and then matched only 100100. The test completed correctly. The fix failed.

I did not install a broader, unproven rule to make the table look better. The gap stayed in the report. An external Python scanner can express the missing-field and arbitrary-key logic cleanly, but that backend did not exist in the measured system. Claiming it as covered would have described planned infrastructure, not code.

The failures became compiler gates

Once a detection declares its rule, fixtures, session join, status, and known gaps in detection.yaml, the framework compiles and validates it through five gates.

The path-traversal rule proved why the fifth gate mattered. The first complete draft passed the write-side checks and then alerted on 20 benign search_files calls that legitimately used ../. I scoped the rule away from five metadata and search tools, reran all eight attack fixtures, and returned the benign result to 0/4,727.

That fix inherits a named evasion. A traversal-capable tool renamed to one of the excluded names can bypass the rule. The compiler can prove that the current fixtures and corpus meet a gate. It cannot turn a bounded name list into semantic understanding.

Adding rule 100108 also exposed two assumptions inside the tests. The compiler schema needed no change, but the negate-probe fixture map was hardcoded by rule ID, so I had to add the new rule manually. A red-team safety check also used git diff against HEAD to detect accidental rule mutation. That worked only while local_rules.xml started clean. The legitimate uncommitted addition of 100108 made the safety test stop before the red-team run began.

I changed that check to compare the file against a snapshot captured when the test started. The compiler itself stayed unchanged, but the surrounding harness was not as registry-driven as the design suggested. I documented the manual cost because adding a sixth detection should not surprise the next person who expects one YAML file to be the whole job.

That limitation remains visible in the compiler report and roadmap.

A result is tied to the rule version that produced it

The offline replay uses frozen Wazuh verdicts for every corpus record. Golden files pin the SHA-256 of wazuh/local_rules.xml, record the expected rule IDs, and preserve checksums for the inputs.

This solved a problem that basic offline fixtures often hide. A regex can change while an old expected-output file keeps passing. MCP Detect rejects the evidence when the current rule hash no longer matches the hash that produced the verdicts.

What the detections still miss

• Meaning-preserving tool-poisoning text can avoid the fixed structural and keyword signal.
• Renamed tools, unknown argument keys, new secret shapes, and encoded content can bypass individual Wazuh rules.
• Absolute paths outside the named sensitive list remain invisible to rule 100109.
• Symlink indirection can reach a sensitive file without exposing a dangerous path string.
• Legitimate version changes alert like malicious metadata drift.
• Behavior-only compromise produces no metadata drift at all.

The last item forced a change in how I reviewed the project. Telemetry could show a valid MCP call and still leave the most important authorization decision unproven.

Run the complete replay

The full corpus is public under MIT and runs from an ordinary clone.

git clone https://github.com/RasheedFarhat/mcp-detect.git
cd mcp-detect
make measure-full

The command verifies corpus checksums, rule hashes, golden matches, technique-specific counts, known overlaps, and the 0/4,727 synthetic benign regression result.