A viral screenshot says a coding ruleset cuts your tokens 94%. The README it was lifted from says something quieter and more honest: ~54% less code, ~20% cheaper, and the 94% is a per-task ceiling on a date picker. I read the README, then re-ran the benchmark myself. My numbers landed right next to theirs. The hype wasn't the tool. It was the feed stripping the units.
Setup
Read Ponytail's actual README and benchmark, then independently re-ran it: a plain agent vs the agent plus Ponytail, on four real coding tasks (XS to M), five runs each, fresh git worktrees at a pinned SHA on a pinned Haiku 4.5 snapshot. Working database provisioned for both arms and Docker shimmed out, so neither could thrash or boot a stack. Scored cost AND correctness.
Measured
What the README claims: ~54% less code (94% is a per-task ceiling on a date picker), ~22% fewer tokens, ~20% cheaper, 100% safe. What I measured re-running it: ~14% cheaper on the bill (their ~20%, confirmed), ~45% less code on the one over-buildable task, correctness 20/20 (the only failure in 40 runs was a baseline run). The viral '94% fewer tokens / 95% off your bill' is a units error, code-saved read as money-saved.
Verdict
VERDICT: MIXED
Everyone shared the "94% fewer tokens" screenshot. The repo went from a few hundred stars to thousands. I went and read the README it was lifted from, and then I re-ran the benchmark myself. The number is real. It is just not what the screenshot says it is.
This is the part the feed skips. The README for Ponytail, the ruleset in question, is honest. Genuinely honest. It says the headline number is ~54% less code, not tokens. It says the 94% is the ceiling on one task (a date picker that collapses from 404 lines to 23), not the average. It puts the real efficiency figures right next to it: about 20% cheaper, 22% fewer tokens, 100% safe. It even walked back its own original 80-94% figure after a GitHub issue pointed out the baseline was padding its answers. Clear method, a control arm, an adversarial safety tier, and a plain statement of where the tool helps and where it does nothing.
So the hype is not the tool lying. It is the feed stripping the units. "94% less code on a date picker" became "94% fewer tokens" became "95% off your bill." Every repost dropped a word until the number meant something it never claimed.
I did not want to trust the hype or the README. So I re-ran their benchmark from scratch.
The same repo the authors used: fastapi/full-stack-fastapi-template, pinned to one commit. Each run got a fresh git worktree at that exact SHA, so every arm started from byte-identical bytes. One pinned model snapshot (Haiku 4.5). Two arms: the plain agent, and the agent plus Ponytail.
Crucially, I scored two things, not one. Cost, and correctness, graded by a fixed test oracle through a real Postgres. Because cost alone is a trap. An agent that quietly does less, or gives up, also spends less. That looks like a saving. It is not one.
Before the oracle graded a single agent's run, I tested the oracle itself. I pointed it at the clean code (it must pass) and at the deliberately bugged code (it must fail), and confirmed it returned exactly those verdicts. Green on correct, red on broken.
This sounds pedantic. It is the one step that separates a measurement from a vibe. A grader that has never been shown to fail on bad input is not measuring correctness, it is measuring its own assumptions.
My first uncontrolled pass showed Ponytail running at 9% of baseline tokens, a 91% cut, right in the viral range. Then I looked at why.
The baseline agent had no working database, so it kept trying to run a test suite that could not run and burned millions of tokens thrashing against it. Ponytail's "be minimal" rule made it give up sooner. That is not efficiency. That is one agent quitting earlier than the other. A spectacular gap can be manufactured two ways: make the treatment brilliant, or make the baseline suffer. Worse, some runs booted the template's entire Docker stack and built images: one hit 4.18 million tokens, 14x the baseline, on the same task.
So there are two different ways to get a fake number here. The feed got theirs by mutating a code figure into a token figure. I nearly got mine from a broken baseline. Neither is the tool. I fixed both, gave each arm a working database, shimmed out Docker, and ran it properly.
With the confounds gone, here is what Ponytail actually does, by the average bill you pay (not the lucky run):
Scale it across a realistic mix and a thousand tasks like these run about $209 on the plain agent and $178 with Ponytail. Roughly $30, or 14% off the bill. That sits right next to the README's own honest ~20%. I re-ran their experiment and confirmed their corrected number.
And correctness held everywhere. Across all 40 graded runs, every Ponytail run was correct, 20 for 20. The single failure in the entire batch was a baseline run. The thing "write less code" was supposed to risk, it never once cost, which is exactly what the README's safety tier claimed.
The most useful thing the re-run taught me is that the two numbers in the viral game are not the same axis. On the refactor, Ponytail cut cost by a quarter but the lines of code were identical to the plain agent. On validation it cut both. "94% less code" and "X% cheaper" measure different things, and a code reduction does not flow straight to your bill, because the agent still has to read the context and reason about the change either way. The 94% is real, on lines, on the one task built to show it. Your invoice moves about 15-20%.
Here is the part that should bother anyone quoting these numbers, including the vendors: these agents are non-deterministic. I ran the same plain agent on the same task twice, changed nothing between the runs, and the token bill came back 2.5 million, then 1.6 million. The same configuration swings up to 1.8x on its own.
A clean savings percentage assumes a stable baseline, and that baseline does not exist. "X% cheaper" than which run? When the thing you measure against moves 1.8x by itself, the honest output is a range, not a point estimate, and the gap you are claiming has to clear that noise before it means anything. It is exactly why the only figure I trust here is "modest, and only on some tasks," and why a single before/after screenshot can say almost nothing. You are not looking at an effect. You are looking at one roll of the dice.
On the tool: MIXED, and fair. A free, honest ruleset that trims real money on sprawling, over-buildable work, does nothing on bounded work, never broke a task, and is candid about all of that in its own docs. Useful. Nowhere near the viral number, and it never claimed to be.
On the screenshot: the verdict that actually matters. The viral 94% is real, but it is lines of code on over-build traps, not your bill. On the bill it is ~20% by the maintainer's own honest revision, and when I ran it myself I got ~14%. Credit to them for already correcting the hype. The lesson is for the rest of us: read the units before you repost the number, and remember that code-saved is not money-saved.
The most valuable thing in that repository was not the skill. It was a maintainer who walked back his own headline when the data changed. Rarer than a 94% anything.
Fadi Labib runs this field lab. 15 years in automotive, robotics, and embedded systems; ESMT Berlin EMBA. I give AI real engineering problems, then check its work. More about the lab →
Two models dropped in one week: Gemma 4, the 12B I run locally, and Fable 5, a frontier model that was officially pulled days later. I spent that short window using Fable as a blind judge for 120 debates and reasoning rounds between five local models. Gemma 4 won 73% as the slowest model on the board, the fastest model came near the bottom, and the one with 'reasoning' in its name finished dead last. The shared failure was calibration: fluent, confident, and unwilling to admit doubt, even from the winner.
Reverse-engineering an 8-in-1 soil sensor, my AI decoded 6 of 8 channels, declared the last two 'not decodable,' and wrote that verdict into version control. I rejected the false ceiling and pushed. Seven hours later the same repo said 8/8. A flawless executor and a shaky judge.
I let an AI agent run a multi-phase build solo. Every phase ended with a clean summary: done, tested, committed. Then I checked git instead. One phase reported '3 prompts, 8 minutes' while the timestamps disagreed, and a fix it marked DONE had been silently reverted 1h53m earlier with nothing in the report changed.