How Do You Know Your AI Agent Isn't Going Rogue? Start by Watching It Behave

The zero-call result

I wrote a task that asked for the diff introduced by the most recent commit. If any prompt should have pushed the model toward git_show, I thought that one would.

Across 28 attempts, qwen3:1.7b answered with git_log followed by git_diff. Every time. It never called git_show once.

I could have kept rewriting the prompt until the tool call appeared. I stopped because that would have damaged the baseline. The point was to record what this model does during legitimate work, not to manufacture a balanced dataset by steering it toward the tool names I wanted.

That left an awkward result in the final corpus. A detection keyed on the literal name git_show has no benign denominator here. I cannot calculate a meaningful false-positive rate for it from this data. The underlying behavior, inspecting what a commit changed, appears through 56 git_diff calls. The literal tool does not appear at all.

Before I could say an agent crossed a line, I needed a defensible record of the agent not crossing it.

MCP had been everywhere I looked. New servers, SDKs, integrations, and demos kept appearing, but I saw much less work asking what would happen when one of those tool boundaries failed. MCP Detect started from that gap. I wanted to build the missing evidence path, then break it hard enough to know where its claims ended.

One call compresses several trust decisions

The current MCP architecture uses a host that manages client instances, with each client communicating with exactly one server. The current 2026-07-28 specification describes a stateless JSON-RPC protocol where each request carries its version and capabilities. My lab is pinned to an earlier runtime that opens a session with initialize, sends notifications/initialized, and discovers tools through tools/list. I keep that distinction visible because the captured record shapes are not examples of the latest protocol revision. The security boundary underneath them still applies. The current MCP architecture (opens in a new tab) still separates host, client, server, and the resources a handler can reach.

A single tools/call record compresses several separate questions. Which identity initiated it? Which arguments crossed the protocol boundary? Which handler accepted them? What did that handler reach next?

For the first phase I narrowed the job to something measurable: capture the wire traffic without changing it, label it, and feed it to a real detection engine.

Phase 0 was seven records and one real alert

I built a byte-transparent stdio proxy in standard-library Python. It sits between the MCP client and server, forwards the original bytes, and writes one JSONL record for each JSON-RPC message. A deterministic client then drove a tiny session: initialize, tools/list, and one tools/call against a synthetic .env fixture.

The session produced exactly seven schema-valid records. I sent them through a real Wazuh manager using the normal localfile, logcollector, and wazuh-analysisd path. Six records stayed quiet. The sensitive file read produced rule 100101 at level 12.

rule.id 100101
MCP sensitive file read via tool call:
tool=read_text_file path=/app/workspace/.env

The alert was satisfying. The proxy failure mattered more.

The response that disappeared

My first proxy used asyncio.wait(..., return_when=FIRST_COMPLETED) for the two byte pumps. When the client closed stdin, the proxy cancelled the server-to-client pump even if the final response was still moving. The session exited cleanly while silently dropping its last message.

I caught it with an isolated echo-subprocess test that proved byte-for-byte round trips in each direction. I replaced cross-cancellation with asyncio.gather and let each pump drain to its own EOF. A telemetry proxy that occasionally eats evidence is worse than no proxy because every later count inherits the loss.

The regex dialect I assumed

I also wrote a Wazuh field match using .+ and expected PCRE behavior. Wazuh's default is its own osregex dialect. The expression did not match a non-empty UUID the way I expected. Adding type="pcre2" fixed it, and every field and match condition in the project now states that dialect explicitly. Wazuh's JSON decoder documentation (opens in a new tab) explains how decoded JSON fields enter this rule path.

I needed the real engine in the loop

wazuh-logtest was useful while writing expressions, but it was not enough to prove the full path. I wanted the record to enter the same place it would enter during collection, pass through Wazuh's JSON decoder, reach wazuh-analysisd, and either create an alert or stay quiet.

I pinned Wazuh 4.9.0 in Docker Compose and ran the amd64 containers through Colima on an arm64 Mac. That architecture mismatch was slower and occasionally fragile, but pinning it gave every later report one named runtime. I copied the rule file into the manager, verified that the loaded bytes matched the repository, tailed the alert output, and counted alerts by rule ID instead of trusting a dashboard view.

The lab separated three kinds of evidence. Unit tests proved the proxy and schema. The live manager proved how the pinned Wazuh engine decoded and evaluated records. Frozen verdicts later made those results reproducible without asking every reader to run the full Docker stack.

That separation saved me more than once. A record could validate as JSON and still fail to match because of Wazuh's regex dialect. A candidate rule could pass an isolated expression test and still lose to a sibling when the complete ruleset loaded. I kept both fast and live paths because they answer different failure modes.

Then I handed tool selection to a model

The scripted client proved that the lab worked. It could not tell me what normal agent behavior looked like because every tool call came from a decision I had already made.

I replaced it with a local qwen3:1.7b model running through Ollama at temperature 0. I supplied benign tasks across read-only filesystem, writable filesystem, Git, memory, time, and fetch server configurations. The model chose the tool, built the arguments, read the result, and decided when to stop.

Corpus v1 contained 33 sessions, 273 records, 54 model-selected calls, and 12 tools. It crossed the phase gate I had written. Then I inspected the per-tool counts.

The first corpus passed the wrong gate

A per-tool false-positive rate needs a per-tool denominator. V1's aggregate of 54 calls looked respectable until I found edit_file at one call and get_current_time at three. At n=5, one alert becomes a 20 percent rate. The corpus was real. It was not large enough in the places that mattered.

I tried scaling the same generation plan. A 240-session run still concentrated almost half the calls in add_observations and read_text_file, while a dozen tools stayed near the repeat floor. A bigger skewed distribution is still skewed.

I fixed the task plan instead. Two targeted top-up passes exercised thin tool families with new paths, timezones, memory queries, and Git references. I merged and re-measured after each pass.

The mistakes became the useful part

Two tools entered v2 because the model made a bad choice. It tried edit_file against the read-only workspace mount. The server rejected the write. The model improvised with create_directory and move_file, tools that had never appeared in v1.

Those calls looked like behavior a future security rule might flag. I did not remove them as noise. I added legitimate writable-sandbox tasks so both tools gained a real benign baseline. create_directory finished at 113 calls. move_file finished at 48, with multiple source and destination paths.

I kept guessed paths, malformed arguments, retries, and odd substitutions for the same reason. One near-boundary read task produced 14 calls across 12 sessions because two attempts used malformed paths. A Git task wandered into an unrelated configuration file before recovering. The pipeline only required complete, schema-valid telemetry. It did not require the model to look smart.

That is how I now think about the baseline: if I want to catch a system doing something bad, I first need to study the same system doing legitimate work, including the messy ways it gets there.

I was wrong about three different failures

The corpus work also exposed failures outside the model.

I killed a process that was still progressing

A git_log call stayed active long enough that one ps aux snapshot looked frozen. I declared a deadlock and killed the process tree. About 17 sessions of real work disappeared with it. Git's cat-file --batch helpers were slow, not stuck.

I changed the operating rule after that. I judge liveness across two checks separated by real time, and I detach long runs so killing a tracking process cannot kill the workload.

The Ollama timeout was an OOM kill

When Ollama returned 500s mid-run, I blamed its five-minute idle eviction and set KEEP_ALIVE=-1. Nothing improved. Ollama's logs said signal: killed. The llama-server subprocess was being OOM-killed inside a 6 GiB Colima VM shared with the Wazuh indexer, dashboard, manager, and Ollama.

I stopped the indexer and dashboard during generation and recovered about 2.1 GiB. Then my permanent keep-alive fix created a slower failure. The on-disk prompt cache accumulated one entry per distinct task, memory climbed from roughly 2.2 GiB to 2.9 GiB, and the process died again. A finite OLLAMA_KEEP_ALIVE=10m bounded the cache growth. The next run completed 238 of 240 sessions.

The full disk was not the generation job

Later, docker cp failed with no space left on a 40 GB VM disk. I suspected the memory changes. The actual cause was Wazuh's vulnerability feed. Its unused caches under queue/vd_updater and queue/vd had grown to 23 GB. I disabled the module because this lab does not use vulnerability-detection alerts.

Each correction came from a concrete signal: signal: killed instead of a timeout line, memory growth across time instead of one snapshot, and queue/vd_updater instead of the generation directory. The operational lesson matched the detection lesson. A plausible story is not evidence.

Why I froze the corpus

benign_corpus_v2.jsonl is committed with its SHA-256 checksum. I do not regenerate it in place.

A model run is not byte-deterministic just because temperature is zero. Hardware, drivers, inference runtime, and quantization can change the exact decisions. A new run can produce statistically similar traffic, but it cannot recreate these 4,727 records byte for byte.

If the corpus changed underneath a published measurement, the number would quietly describe different evidence. Freezing the file gives every later claim an exact input. Extending the corpus requires a new version. Partial telemetry from failed runs remains archived as recovery provenance and does not enter the measured dataset.

What this result does not establish

• qwen3:1.7b is one small model. Another model can produce a different normal distribution.
• Zero alerts across 4,727 records is a regression result over project-authored traffic. It is not a field false-positive rate, a benchmark, or validated real-world accuracy.
• git_show has no benign denominator in this corpus.
• The lab has no per-call timeout. A genuinely hung tool call can still wait forever.
• The proxy records what reached a handler. It cannot prove what the handler did next.

The final limitation becomes the center of part three. First I needed to find out how far structural detections could go, and how quickly an adversarial corpus could break them.

Run the measurement

Python 3.11 or newer. No credentials, Docker, Ollama, or Wazuh installation required.

git clone https://github.com/RasheedFarhat/mcp-detect.git
cd mcp-detect
make measure

The line to inspect is Aggregate benign FP: 0/4727. The corpus summary (opens in a new tab) lists every per-tool denominator and the accepted git_show gap.