The Sidecar Race: 22 Seconds vs 69 Seconds Inside the Agent Loop

Sixty-nine seconds. That is how long the agent waited, again, for CircleCI to answer a question it had already asked nine times that run: did lint and test pass?

Twenty-two seconds is what the other arm averaged for the same question on a Chunk sidecar.

Same repo. Same ten Go tasks. Same Claude Agent SDK edits. Same gate jobs. Different loop.

Three weeks ago I published AFK Builds with 100% Green PRs. Five Snake games. RalphCI orchestration. Sidecars in the Review Gate so agents stopped feeding red commits to GitHub. Green PRs were the win.

This follow-up is narrower and, I think, more revealing for anyone running agents today: not whether sidecars help you merge green, but how fast the agent gets CI-shaped feedback while the file is still open.

I kicked the tires on sidecars again.

They are fast.

Not a much cheaper agent. A less idle one. Token spend was flat in this particular experiment (more on that later).

The win was clock time.

Hypothesis

If each agent task is validated with chunk sidecar sync plus chunk validate --remote (lint and test-changed) instead of commit → push → poll CircleCI for the same lint and test jobs, then:

1. Median time to signal (TTS) per iteration will be materially lower on the sidecar arm.
2. LLM cost will be similar (same agent, same ten prompts, same model fixing the same mistakes).
3. CircleCI still matters for pipeline-level confirmation (sidecar runs end with a full ci workflow epilogue after one final push).

I expected push-per-task CI to pay a queue tax every iteration. However, I did not expect the median to land on 69 seconds five replicates in a row.

Setup

Repo:

• CircleCI-Public/chunk-cli
• Experiment PR #370

Tasks: Ten cumulative edits on a small Go fixture (internal/racefixture/ on experimental run branches).

Replicates: Five labels (001–005) per arm. 10 recorded runs total.

Metric: TTS = wall-clock seconds from iteration start until both gates report pass/fail (logged in results.csv).

CircleCI CTO Rob Zuber frames this as rebalancing inner and outer loop validation. My May post showed the outcome layer (green PRs). This post measures the wait layer.

Results

Headline: median time to signal

Aggregate: median of per-run medians across five replicates.

Five replicates at a glance

Sidecar clustered 20–22s. CI sat on 69s every time. That stability is either a real push-and-queue signature in our harness or a coincidence at lab scale. I am reporting it, not universalizing it.

p95 TTS: sidecar ~23–25s; CI ~72–99s (tails worse on push-per-task).

LLM cost comparison explained

The speed table is not the only controlled comparison in this harness. Token spend is measured the same way: same model (claude-sonnet-4-20250514), same ten tasks, same agent, same fix-and-retry pattern. We changed the validation path, not the work.

Both arms hit the same lint and test failures and applied the same corrections. Sidecar surfaced them via chunk validate --remote on the warmed snapshot (~22s median). The CI arm surfaced them after push to GitHub and a ~3× longer wait for the same gate jobs (~69s median). Same signal, different delivery speed.

That is why token spend landed flat. Tokens track fixes, not wait. The CI arm cost more clock time, not more tokens. Sidecars did not shrink the LLM bill here because there was nothing to shrink: identical failure set, identical agent work on both arms. On a messier repo, especially if remote CI surfaces failures the sidecar doesn't mirror, token spend might diverge. This harness didn't test that.

So treat ~$0.90 per run as what this harness actually cost for this specific experiment, not a budget for production agent loops. Whether faster feedback trims token burn on messier work and CI failure modes (i.e. given fewer stale-context retries, less thrash, and less environment drift) is a follow-on question for a future study (coming soon). This race did not answer it.

This experiment answers a narrow question cleanly: when the agent does the same fixes either way, faster feedback does not change token spend. It changes how long the agent sits idle.

Same gate jobs. Different machinery.

We timed the same question on both arms: did lint and test pass? We did not run identical validation. Sidecar answers via chunk validate --remote on a warmed-up Linux snapshot. CI answers via real CircleCI jobs after you push.

Early iterations on both arms still saw lint failures while the agent fixed mistakes (about two failing lint iters per run in the rollup). The race is not "sidecar never fails." It is how long you wait to learn.

Illustrative lab math only: ten tasks × ~47s saved ≈ ~8 minutes less gate-waiting per replicate before the sidecar epilogue. Your mileage will vary.

Takeaway

Green PRs are the outcome. Time to signal is the throttle.

The May experiment showed sidecars help agents stop polluting branches with commits CI would reject. This one shows why that matters in wall clock terms: the coding agent is not stuck in queue while context goes cold.

What held across ten runs in chunk-cli:

1. Sidecars save feedback latency on mirrored gate jobs (~3.1× median TTS here).
2. LLM spend was flat (~$4.64 sidecar vs ~$4.73 CI) in the same apples-to-apples comparison: same failures, same fixes, CI just took ~3× longer to surface them. Tokens track fixes, not wait.
3. Push-per-task CI is a predictable wait in this harness (69s median, every replicate).
4. You still want an outer loop. Sidecar runs finished with a full ci workflow epilogue. Fast inner loop on each task, plus an authoritative pipeline confirmation at the end. Not sidecar-only cowboy coding.

Please re-read #4 above: I am not at all suggesting you delete your CI pipelines. Quite the opposite: I am saying that if your agent loop is push → wait → fix → push again, you are paying for that wait every iteration.

Sidecars moved the same exact question to a local, warmed-up Linux snapshot...

QUESTION: Are lint and test happy on this repo right now?

...and the same answers came with a much tighter feedback loop. 3.1× faster.

What's Next

• Combine this harness with RalphCI orchestration (Snake outcome study + TTS race in one stack).
• Heavier repos, flaky tests, parallel agents. Find where Chunk sidecars and snapshot sync stops winning.
• Agent economics at scale: cost per green iteration (LLM + validation path), not just TTS. A proper cohort study on heavier repos, not extrapolated from this ten-run table alone.
• Demo recordings on various Chunk sidecar and snapshot sync setups. Stay tuned!

Artifacts:

• FINDINGS.md
• Experiment PR #370