DocuSOR
Case Studies

Same quality. 77% faster. 64% cheaper.

We measured DocuSOR against the way agents work today. Every result below comes from two live Claude Opus 5.5 agents performing the identical production-readiness audit (one with DocuSOR, one without), directly against Anthropic's API with our own real, billed token counts, not estimates. Both answers were graded blind by an independent judge against a hand-verified control. Prompt caching was enabled on both arms.

Parity
Audit quality vs. a hand-verified control
92.4 with DocuSOR vs. 92.2 for a thorough manual audit: a statistical tie, graded blind by an independent judge.
77%
Faster wall-clock
152.7s → 34.6s. Fewer sequential round-trips means less time and less rate-limit exposure.
64%
Cheaper per audit, caching on
$0.5298 → $0.1898 on the identical production-readiness audit, with prompt caching enabled on both arms.
95%
Fewer tool calls
65 manual discovery probes collapsed into 3: one verified read plus two quick spot-checks.
Featured · Production-readiness audit

Equal quality, one-third the cost: one verified read vs. a 65-call discovery crawl

“Same model, same task, same grading rubric. The only difference was whether the agent could read verified context or had to discover everything manually.”

In a head-to-head production-readiness audit, graded against a hand-verified control, an AI agent given DocuSOR matched the quality of a thorough manual-discovery agent while spending ~⅓ the cost, making 95% fewer tool calls, and finishing 4.4× faster. The only difference between the two arms was whether the agent could read context for the service from DocuSOR, while the other had to discover everything manually, using its own persistent memory (CLAUDE.md, AGENTS.md, Copilot MEMORY.md), the way agents do today.

The headline isn't "faster and cheaper at the cost of quality." It's the same quality, 77% faster, 64% cheaper, and 82% fewer tokens, and the DocuSOR answer came back provably current, because its claims were verified, not freshly guessed.

Executive summary

Two live Claude Opus 5.5 agents were tasked with the same production-readiness audit of the same service: one with the generic discovery tools coding agents use today, the other with those same tools plus read-only access to DocuSOR. Both answers were scored, blind, by an independent judge against a fixed, hand-verified ground-truth control. The headline isn't "faster and cheaper at the cost of quality." It's the same quality, 77% faster, 64% cheaper, and 82% fewer tokens.

MetricWithout DocuSOR (manual discovery)With DocuSOR (one verified read)Difference
Audit quality (0–100, vs. control)92.292.4Parity
Wall-clock time152.7s34.6s77% faster (4.4×)
Cost (USD)$0.5298$0.189864% cheaper (2.8×)
Tool calls65395% fewer (21.7×)
Total tokens196,71234,46982% fewer (5.7×)
Grounding (0–100)9390Comparable
The verified read matched a full manual audit. It did not trade accuracy for savings. Both agents correctly covered all 26 endpoints, all 12 environment variables, and every health, TLS, log, and invariant check. DocuSOR simply got there in 3 tool calls instead of 65.

The experiment

The task (identical for both arms): produce a full production-readiness audit of a live service: every endpoint with its auth requirement and status code, all environment variables, plus health, network, TLS, container, logs, and runtime invariants.

The subject service: docusor-test-api, a Fastify service with 26 endpoints, JWT/OAuth2 auth (user and admin roles), a TLS listener, 12 environment variables, and a Docker Compose deployment. Representative of a real microservice an agent would need to learn from scratch.

Without DocuSOR (manual discovery): four generic tools every coding agent has (http_get, read_file, list_dir, run_command) plus the context from its own self-managed persistent memory (CLAUDE.md, AGENTS.md, Copilot MEMORY.md). The agent discovers the service the hard way: probing endpoints, reading source, inspecting the container. With DocuSOR (one verified read): the same generic tools plus the read-only DocuSOR MCP tools, so the agent could read the verified context for the service.

How quality was measured: each answer was graded by an independent LLM judge against a fixed, hand-verified ground-truth control (control/ground-truth.json), built from the real source, live unauthenticated probes of every endpoint, and container/TLS inspection, independently of DocuSOR's hosted context, so neither arm is advantaged.

Quality: a tie (92.4 vs. 92.2)

Both agents correctly covered all 26 endpoints, all 12 environment variables, and the health, TLS, log, and invariant checks. The DocuSOR agent scored 94 on correctness (vs. 92); the baseline edged it on grounding/coverage (93 vs. 90). Net overall: a statistical tie. The verified read matched a full manual audit. It did not trade accuracy for savings.

Speed: 4.4× faster

34.6s vs. 152.7s. Fewer round-trips means less wall-clock time and less exposure to rate limits, often the real bottleneck in agent systems.

Tool calls: 3 vs. 65, and why that matters

The DocuSOR agent read the verified context once, then made two quick spot-checks. The baseline ran 65 probes. That gap is not cosmetic: each tool call is a sequential model round-trip that re-bills the entire growing transcript, so it compounds across five dimensions.

Baseline tool callCount
Directory listings8
HTTP calls17
File reads28
Shell commands12
Total probes65
  • Cost. Every round-trip re-sends the whole conversation so far as input tokens. 65 calls means that transcript is re-billed ~65 times, the single biggest driver of the cost gap.
  • Latency. Calls are sequential, so time tracks them: 34.6s with DocuSOR vs. 152.7s without.
  • Reliability. Long probe loops have more opportunities to derail; a lean read reaches the answer before anything can go wrong.
  • Context budget. 65 probe payloads crowd the context window and leave less room for the actual reasoning; one compact read preserves it.
  • Blast radius. The lean path needs essentially one read capability; the 65-call path needs shell execution and container reach, a far larger attack surface to approve and keep track of.

Cost: one-third (64% cheaper)

$0.19 vs. $0.53. The DocuSOR agent reached the same answer for roughly a third of the price, with prompt caching enabled on both arms, as most coding agents have it on by default.

Tokens: 5.7× fewer

Token counts are the model's own input_tokens and output_tokens, summed over every call, not estimates. Prompt caching was enabled for both arms.

ArmInput tokensOutput tokensTotal
Without DocuSOR185,75210,960196,712
With DocuSOR30,6353,83434,469

Grounding: comparable

At 90 (DocuSOR) vs. 93 (baseline), grounding was within a few points. The baseline, by probing everything live, enumerated more first-person-observed detail and, proportionally, about the same rate of unsupported asides. DocuSOR reached the same grounding tier in one-third the output. Neither arm fabricated materially; the verified read simply got there with far less text.

Why the gap is so large

Two facts about how agentic LLM loops are billed explain the entire result:

  • Every round-trip is billed. Each tool call is another model turn: more to reason over, more output to decide the next step.
  • The transcript re-bills every turn. On each call, the entire running conversation (system prompt, tool schemas, and every prior tool result) counts as input again.

So the cost of manual discovery isn't one probe. It's every probe, re-billed by every turn that follows it. DocuSOR collapses that multi-turn, multi-tool loop into a single compact payload the agent reasons over once.

How this scales

The per-audit saving (~$0.34, ~160 K tokens, 62 fewer tool calls) is modest in isolation. It compounds because discovery is high-frequency and repeated: every new agent session, every CI run, and every agent that can't share another's context pays it again.

DocuSOR inverts the economics: verify once, read many. A service's contract is proven on merge, and every agent afterward reads the cheap, verified document instead of re-probing. The marginal cost of the thousandth agent asking "how does service X work?" trends toward one small read, not tens of thousands of tokens of fresh exploration.

Honest limits

  • This is one small service: DocuSOR's floor, not its ceiling. The advantage compounds with endpoint count and audit depth; a larger service widens the gap, it doesn't narrow it.
  • Deep manual probing still has a place. A thorough crawl can surface things a published contract doesn't yet contain. In the DocuSOR model that's complementary: once something is discovered and proven, you publish it, so the next thousand agents get it in a single read.
  • Scope. This result measures service discovery / production-readiness auditing, the high-frequency task where re-discovery dominates cost.

Takeaways

  • Same quality, 77% faster, 64% cheaper, 82% fewer tokens (95% fewer tool calls) on the identical production-readiness audit, measured on a live Claude Opus 5.5 agent with real billed usage, graded blind against a hand-verified control.
  • The savings come from removing discovery from the token-billed loop: no growing transcript to re-bill (input), and far fewer turns to reason through (output).
  • The verified read was not a quality trade-off. It matched a thorough manual audit point-for-point while spending roughly a third as much.
  • At scale the durable win is amortization: verify once, read many. Every agent afterward reads the cheap, proven document instead of re-probing.
Want these numbers for your own services?
See DocuSOR verify and serve your contracts in a short walkthrough.
Book a Demo

Measured on recording session 20261023-115014 against docusor-test-api (Fastify, 26 endpoints, JWT/OAuth2, TLS, Docker Compose). Both arms ran the live claude-opus-5-5 agent with prompt caching enabled; token and cost figures are the provider's billed usage. Quality was scored by an independent LLM judge against control/ground-truth.json, a hand-verified control derived from the real source, live probes, and container inspection, independent of DocuSOR's hosted context. Cost uses list Opus pricing ($4/Mtok input, $20/Mtok output).