🎯 TL;DR
Same app, same model (Sonnet 5.5, low effort), same quality: every run passed 30/30 API tests and 20/20 UI checks.
| Comparison | Tokens Used | Speed | Runs Sampled |
|---|---|---|---|
| Desktop app: Claude projects coordinator vs APCD | Projects used 1.6× more | – | 2 vs 1 |
| CLI: Claude Code + sub-agents vs APCD | Sub-agents used 1.1–1.3× more (1.15× on cost) | APCD 2× faster | 2 vs 3 |
An orchestrator adds a “manager” agent that re-reads its whole growing conversation on every step, and its sub-agents re-explore the code. APCD has no manager. One short project doc lets each fresh chat pick up exactly where the last one stopped.
“In my experience, orchestrator coordination hasn’t been great yet. It’s like manual shift vs auto transmission, back when auto transmissions were just introduced.” — Karthik, 2026-10-12
1. What We Compare
| Arm | How work is split | How context carries over |
|---|---|---|
| P: APCD plugin | The human opens one new chat per milestone (6 chats) | CLAUDE.md Progress list |
| O: orchestrator | A coordinator spawns one sub-agent per milestone (1 + 6) | The coordinator’s handoff notes |
- O flavours: the desktop Claude projects coordinator, and a CLI session + sub-agents (
claude -p). - Compare only on the same surface: Desktop vs CLI changes cost a lot by itself (§4).
- Scope: Token efficiency only. Coordination quality isn’t scored.
- Excluded: A single solo chat. Real big projects need a split.
2. Setup
- Specimen: Linkshelf (
SPEC.md). A bookmark API plus a one-page UI, Node with zero dependencies, ~400 LOC, 6 milestones. - Model: Sonnet 5.5, effort low, everywhere.
- Environment: The same plugins and MCP in all runs (realistic, not stripped).
- APCD version: Published, without the auto-refresh hook.
Metrics
| Metric | What it is |
|---|---|
| Input units (main) | in×1 + 5-min cache write×1.25 + 1-hour cache write×2 + cache read×0.1, which is Anthropic’s relative pricing. |
| Total units | Input units + output×5. Desktop sub-agent logs under-record output, so O’s desktop totals are too low. |
| CLI cost | Claude Code’s own $ estimate (CLI runs only). |
| Quality | Hidden API tests /30, scripted UI checks /20, own tests, LOC, blind visual rating (pending). |
3. All Runs
| Run | Arm | Input Units | Total Units* | CLI Cost $ | Calls | Chats + Sub | Hidden API | UI Checks | Own Tests | LOC | Min |
|---|---|---|---|---|---|---|---|---|---|---|---|
O-1 - Copy |
O | 954k | 990k | – | 78 | 1 + 6 | 30/30 | 20/20 | 49/50 | 387 | – |
P-1 |
P | 602k | 737k | – | 45 | 6 + 0 | 30/30 | 20/20 | 17/17 | 368 | – |
r2-O-1 |
O | 405k | 611k | $1.06 | 59 | 1 + 6 | 30/30 | 20/20 | 30/30 | 386 | 8 |
r2-O-1-claude-projects |
O | 970k | 1013k | – | 73 | 1 + 6 | 30/30 | 20/20 | 46/46 | 475 | – |
r2-P-3 (folder r2-O-2) |
P | 361k | 465k | $0.91 | 38 | 6 + 0 | 30/30 | 20/20 | 12/12 | 334 | 4 |
r2-O-2-corrected |
O | 415k | 657k | $1.16 | 53 | 1 + 6 | 30/30 | 20/20 | 48/48 | 442 | 9 |
r2-P-1 |
P | 394k | 523k | $1.01 | 41 | 6 + 0 | 30/30 | 20/20 | 17/17 | 403 | 4 |
r2-P-2 |
P | 377k | 497k | $0.97 | 39 | 6 + 0 | 30/30 | 20/20 | 17/17 | 344 | 4 |
* Desktop runs (no r2- CLI cost) under-record sub-agent output, so O’s desktop totals are too low.
Note: r2-P-3 ran in the folder named r2-O-2 by mistake. It is an APCD run.
3.1 Desktop app: APCD vs Claude projects
| Dimension | APCD (P-1) | Projects (O-1 Copy, r2-O-1-claude-projects) | O ÷ P |
|---|---|---|---|
| Input units | 602k | 954k · 970k (within 2%) | 1.6× |
| API calls | 45 | 78 · 73 | 1.7× |
| Tests | 30/30 · 20/20 | 30/30 · 20/20 | equal |
3.2 CLI: APCD vs session + sub-agents (averages)
| Dimension | APCD (3 runs) | Sub-agents (2 runs) | O ÷ P |
|---|---|---|---|
| Input units | 377k (361–394k) | 410k (405–415k) | 1.09× |
| Total units | 495k | 634k | 1.28× |
| CLI cost | $0.96 | $1.11 | 1.15× |
| API calls | 39 | 56 | 1.4× |
| Time | 4 min | 8.5 min | 2.1× slower |
| Tests | 30/30 · 20/20 | 30/30 · 20/20 | equal |
- Runs are stable: APCD varied ±4% across 3 runs, O varied ±1% across 2.
- APCD’s biggest cost is its cache price: Each new chat writes the 1-hour cache at 2× base price; sub-agents write the 5-minute cache at 1.25×. That is about 2/3 of APCD’s input units.
4. Desktop App vs CLI
| Arm | Desktop | CLI | Change |
|---|---|---|---|
| APCD | 602k | 377k | −37% |
| Orchestrator | 962k | 410k | −57% |
The desktop app adds overhead per agent, so orchestration (7 agents) pays it 7 times.
5. Findings
APCD never cost more at equal quality
It was 1.6× cheaper than desktop projects, and 1.15× cheaper in $ and 2× faster than CLI sub-agents.
The manager agent is the tax
Its context grows every step, and it waits between sub-agents, which slows things down.
Surface matters as much as method
The desktop app made orchestration about 2.3× pricier than the CLI.
Orchestrators don't always orchestrate
“Use sub-agents as you see fit” produced zero sub-agents, so the O prompt forces them.
Quality is equal on every automated check
The visual comparison is still pending blind rating (round 1’s APCD UI looked plainer).
6. Caveats
- It’s a small, focused specimen (about 400 LOC). Big, messy projects may differ.
- The desktop comparison is n=1 for APCD vs n=2 for projects.
- APCD trades a little laziness: you open each new chat yourself, while orchestrators do it for you.
- It was a single model and effort level (Sonnet 5.5, low).
7. How to Run More
cd S:/PROJECTS/dir2_learner/apcd-benchmark/apcd node tools/bench.mjs run P <new-name> node tools/bench.mjs run O <new-name> node tools/bench.mjs score <folder-name> node tools/bench.mjs blind r2
run: CLI, auto-scored.score: for a manual desktop run inS:/bench/<folder-name>.blind: blind visual rating page.- Use a new name per run. Folders are never reused.