comparator-admin

Harness for a Hackday experiment: ask Claude Code to build the same admin interface twice — once as a react-admin app, once from scratch (no admin framework) — built feature by feature from a sequence of prompts that are identical for both builds except the bootstrap step's framework line, against the same live API and data. Then measure and compare, per feature, tokens/cost, time, size, and functional completeness.

Full design/rationale: docs/superpowers/specs/2026-07-07-react-admin-vs-scratch-comparator-design.md. Implementation plan: docs/superpowers/plans/2026-07-07-react-admin-vs-scratch-comparator.md.

This repo is the harness only: data + API + prompts + measurement tooling + report page. It does not run the two builds for you — you run each one in its own Claude Code session (instructions below) and then measure the result.

What's in here

data/           fetch real launch data once, build the local seed JSON
api/            writable REST API (FakeRest) serving the seed on :4000
prompts/        feature-by-feature build prompts (00 bootstrap … 08 extend) + README
tools/          measurement scripts + TOKENS.md protocol
report/         self-contained HTML report (Chart.js inlined) + per-step results schema
with-react-admin/   empty — filled by the react-admin build session
from-scratch/       empty — filled by the from-scratch build session

Prereqs

• Node.js ≥ 20.
• A context7 API key for the react-admin build. The shared/free context7 MCP endpoint hit "monthly quota exceeded" while this harness was being prepared, so don't assume it'll have quota left when you run the experiment. Get a free key at context7.com/dashboard and configure the context7 MCP server with it before starting the with-react-admin session. If you don't have a key and context7 is out of quota, the prompt tells Claude to fall back to WebFetch of https://marmelab.com/react-admin/documentation.html — the build will still work, just with a different (probably worse) documentation-lookup path, so note this deviation in your results if it happens.
• Nothing else to install at the root (package.json has no dependencies of its own, only npm scripts). The API has its own package.json.

Setup

Run once, in order, from the repo root:

cd api && npm install && cd ..   # installs fakerest for the API
npm run fetch                    # one-time: pulls real data from the LL2 dev API
npm run seed                     # builds data/seed/*.json from data/raw/*.json
npm run api                      # serves the API on http://localhost:4000 — leave running

• npm run fetch hits the Launch Library 2 dev host (https://lldev.thespacedevs.com/2.2.0) — see Caveats for why LL2 and not the SpaceX API. It's a one-time network call; re-run it only if you want to refresh data/raw/.
• npm run seed is deterministic and offline — it reads data/raw/*.json and writes data/seed/*.json. Re-run it any time after editing data/raw/.
• npm run api starts api/server.mjs, which loads data/seed/*.json into a FakeRest in-memory database and serves a writable simple-rest API. Leave it running in its own terminal for the entire experiment — both builds (and the sanity checks below) depend on it being up on port 4000.

Current seed sizes (from the checked-in data/seed/*.json): 205 launches (45 dropped from the raw fetch for dangling references), 400 rockets, 236 launchpads, 400 astronauts, 400 spacecraft, 400 capsules (capsules is the round-2 resource, seeded up front so the follow-up prompt has data to point at).

Running the two builds (feature by feature)

Open two separate Claude Code sessions — do not reuse one session or one cwd for both builds, and do not run them concurrently in the same terminal tab if you want independent wall-clock timing.

Session 1 — react-admin: cd with-react-admin && claude (fresh session, cwd here). Session 2 — from scratch: cd from-scratch && claude (a second, separate fresh session).

Then paste the prompts in order, the SAME step into both sessions before moving to the next (full order and rationale in prompts/README.md):

Step	Session 1 (react-admin)	Session 2 (from scratch)
00 bootstrap	prompts/features/00-bootstrap-react-admin.md	prompts/features/00-bootstrap-from-scratch.md
01 … 08	prompts/features/01…08 (identical)	prompts/features/01…08 (identical)

Only step 00 differs between the two sessions (the framework line: react-admin

• context7 vs. plain React/Vite, no framework). Steps 01–08 are byte-identical across both sessions — that identity is what makes the comparison fair. The API/data reference is embedded in step 00 and stays in each session's context for the later steps. Both sessions point at the same running API on :4000.

After each step, record the session's cumulative /cost (tokens + $) and wall-clock before pasting the next — the per-feature cost is the delta (see Measuring below and tools/TOKENS.md).

Fairness checklist

For the comparison to mean anything, hold everything except the framework constraint fixed:

• Same model for both sessions (same Claude Code default, no manual model switching mid-build).
• Same Claude Code version for both sessions (check claude --version before each; don't upgrade between them).
• Fresh session per build — never resume/continue one build's session to start the other.
• Identical MCP tool set available in both sessions (context7 included in both, even though only the RA prompt is told to lean on it — this keeps "the tool was available" constant, only usage differs).
• cwd is the build's own subfolder (with-react-admin/ or from-scratch/) for the whole session. This is also what makes each build a distinct Claude Code project with its own session JSONL under ~/.claude/projects/, which is what tools/TOKENS.md relies on for clean per-build attribution.
• No intervention beyond the initial prompt. Let each build run to completion (or to a natural stopping point) without steering, extra hints, or manual fixes. If Claude asks a clarifying question, answer it identically (or as close as possible) in both sessions, and note the exchange when you write up results — it's a deviation worth disclosing.

Step 08 — extensibility

Step 08 (prompts/features/08-capsules-extensibility.md) is part of the same sequence, pasted into each build's continued session after step 07. It asks for a new capsules resource (already seeded and served at /capsules) with the same list/filter/sort/CRUD/relation treatment as the other resources. Its per-step delta is the marginal cost of adding one more feature to an existing app — the metric that's supposed to show react-admin's "cheap to extend" value proposition, if it holds. The report highlights it.

Measuring

Full protocol, exact commands, and where to write each number: tools/TOKENS.md. Summary:

1. Per step (core): after each step, note the session's cumulative /cost (tokens + $) and wall-clock/turns; the per-feature figure is the delta from the previous step. Write each into report/results.json's steps[i].ra/.fs. Set passing per step by exercising the feature in the running app.
2. Cross-check: npx ccusage@latest session — the per-session totals should roughly equal the sum of your per-step token deltas for each side.
3. Optional per-step time detail: node tools/analyze-session.mjs <session.jsonl> (slice the JSONL per step to attribute). The LLM-vs-tool split is a gap-based approximation, not real instrumentation — directional only.
4. Final size (once, at the end): node tools/measure.mjs with-react-admin and node tools/measure.mjs from-scratch for files / LOC / deps, plus a manually recorded bundleKb; write into results.json's size.
5. cp report/results.example.json report/results.json, fill in the real per-step + size numbers for both builds, then open report/index.html to see the per-feature comparison (tokens per step, cumulative tokens, time per step, final size, breakdown table).

Caveats

• Data source is Launch Library 2, not the SpaceX API. The originally considered api.spacexdata.com is down (HTTP 525, Cloudflare origin failure) and unmaintained. LL2 (https://lldev.thespacedevs.com/2.2.0, free, no auth) is a live alternative in the same domain, so the fetch uses its dev host to avoid hammering the production one.
• To-many relations are synthesized. astronaut_ids and spacecraft_ids on launches (and launch_ids on the round-2 capsules resource) are deterministically generated at seed time — both endpoints of each relation are real LL2 entities, but the pairing between a given launch and its crew/spacecraft is not sourced from LL2 (LL2's free tier doesn't expose that association at the granularity we need). This preserves the to-many CRUD requirement without misrepresenting synthetic data as real; disclosed here so the comparison stays honest.
• context7 quota. The shared context7 MCP endpoint reported "monthly quota exceeded" during harness prep. Get your own free key (context7.com/dashboard) before running the react-admin build, or expect the fallback path (WebFetch of the react-admin docs site) to kick in — see Prereqs.
• The LLM-vs-tool time split (llmMs/toolMs in analyze-session.mjs) is an approximation, not a measurement — see Measuring, step 2, and tools/TOKENS.md for the exact method and its limitations.
• The API returns HTTP 206 Partial Content (with a Content-Range header) for paginated list requests that return less than the full collection — this is normal simple-rest/FakeRest behavior, and ra-data-simple-rest (and any correct fetch-based client) handles it as a success, not an error.

Quick sanity check

To confirm the harness itself (not a build) is wired correctly:

npm test
npm run api & sleep 1
curl -sg 'http://localhost:4000/launches?range=[0,2]&sort=["net","DESC"]' | head -c 200; echo
curl -sg 'http://localhost:4000/rockets?filter={"q":"Falcon"}' | head -c 120; echo
kill %1
node tools/measure.mjs api

Expect: all tests pass, both curls return JSON bodies, and measure.mjs prints file/LOC/dependency counts for api/.

Note the -g/--globoff flag: curl treats unescaped [, ], {, } in a URL as its own globbing syntax and errors (curl: (3) bad range) without it. This is a curl client quirk, not an API bug — ra-data-simple-rest and browser fetch both send these characters URL-encoded and are unaffected.