Treating Content Operations as a Calibratable Prediction System

The thing that annoys me most about making content is the armchair-quarterbacking. A piece blows up, and I can instantly cook up three reasons why it was destined to. A piece flops, and I can just as easily produce three reasons why it deserved to. It all sounds perfectly coherent—and it’s all hindsight. Not one of those explanations was something I dared write down before I hit publish, which means they offer zero guidance for the next piece. They just feed my ego.

This bugged me for a long time. Eventually I realized the problem wasn’t that I didn’t understand content—it was that I had no mechanism to force me to place a bet up front and then let real data prove me wrong. So cheat-on-content was born: a content-operations calibration workflow that I use myself and have open-sourced.

The “cheat code” is real, but it’s private

Let me be clear about the metaphor first. The content game is already, at its core, a race for cheat codes—whoever cracks a platform’s distribution rules first gets the traffic first. Everyone is hunting for that magic string.

The problem is that all those “growth hacks” and “golden three-second rules” you find online are the average across the entire internet. They’re statistical conclusions distilled from blending the data of thousands upon thousands of accounts, and they don’t necessarily hold for your specific account, your specific audience, your specific way of expressing things. Follow the internet-wide average, and you’ll most likely get internet-wide-average results.

What I want isn’t a generic playbook—it’s a system that specifically studies my own account.

That’s the starting point of the whole design: abandon the faith in “universal laws,” and instead build something that can slowly close in on the rules that belong to my account specifically, learned from the real feedback of every single piece I put out.

Three non-negotiable principles

Whether this thing graduates from “astrology” to “calibration” rides entirely on three principles. Break any one of them and the whole loop degrades back into the hindsight nonsense I opened by ranting about.

First: blind prediction. Before I look at any data, I score the piece and write down a prediction: what tier I think it’ll hit, and why. The crucial part—once the prediction is written, it cannot be changed. This doesn’t run on willpower; it’s locked at the tooling layer with a hook. Why be so ruthless about it? Because people lie to themselves. Leave even a sliver of a “let me peek at the result and tweak a couple of lines” loophole, and the prediction loses all its meaning—it just turns back into writing explanations after you’ve seen the answer. Lock it down, and only then does it have calibration value.

Second: upgrading the rubric means rescoring everything. My scoring rubric evolves. But the rule is: the moment I want to change the rubric, I have to rescore every historical piece in the calibration pool with the new rules, and have it independently reviewed across models. I can’t just change the formula and leave the historical scores sitting under the old one. Otherwise the old and new data aren’t measured against the same yardstick, and any comparison I draw between them is wrong. This step is annoying and heavy—but the annoyance is the point. It forces me not to mess with the formula too casually.

Third: the rubric is a workbench, not a museum. An observation that later data confirms—and that’s already been absorbed into the formula—gets deleted. An observation that gets overturned gets deleted too. The rubric only keeps the rules that are still valid right now. I don’t let it fill up with the corpses of “things I used to believe.” Want to see that history? Go dig through git history—that’s the archive. The current file has to stay clean, because clean is what makes it usable.

Five stages, one loop

In practice it runs as five steps:

Pick a topic → score (purely exploratory, console output only, writes nothing, leaves no trace) → blind predict (predict, written into that immutable log) → publish → T+N day retro (retro, where the real data comes back in, gets checked item by item against the original prediction, and new observations get written back into rubric_notes).

Note that score and predict are two separate actions. score is the scratchpad where I poke around casually and adjust my feel—change it however I like. predict is the formal bet, set in stone. Separating “probing” from “committing” right at the tooling layer is what keeps me from slacking off when I should be serious and tensing up when I should be loose.

The retro stage is where the loop actually grows wisdom. The gap between prediction and reality doesn’t immediately rewrite the formula—a single deviation might just be luck. Only once I’ve accumulated enough deviations in the same direction (say, I consistently overrate a certain kind of topic, or underrate a certain kind of hook) does the system unlock the right to do a rubric upgrade. That threshold—“you can only act once you’ve stacked up enough same-direction deviations”—is what fights overfitting on my behalf: don’t go frantically rewriting your worldview because of one or two hits or one or two flops.

How the tooling stops me from cheating

If this lived only as a document, it would decay quickly. What makes it hold is the constraint built into the tooling layer:

• score only prints to the console and writes nothing, so I can probe before making a formal bet;
• predict creates a timestamped prediction file, and the prediction section becomes immutable after it is written;
• publish only fills in the URL, platform ID and publish time, without touching the prediction body;
• retro pulls real data back at T+N days and writes the gap into the retrospective record;
• upgrading the rubric requires rescoring all historical samples, otherwise old and new scores cannot be compared on the same scale.

In other words, it does not rely on “I will be disciplined later.” It turns discipline into file structure, command boundaries and hook constraints. That is the part I care about most: a good methodology without tooling usually loses to laziness and memory bias.

This is really just the harness playbook

I have to admit at this point that the skeleton of this thing isn’t original to “content methodology” at all. It’s a direct transplant of the way I think when I’m building AI agents and harnesses.

Content distribution is a textbook uncertain system: you feed in a piece, the output is traffic, and in between sits a black box you can’t see through. Facing a system like that, the standard engineering answer has never been “think harder”—it’s to wrap it in a verifiable loop: predict, compare, iterate.

This is the same thing as test-driven development. Writing a test is writing down “here’s what I assert it should be” before the code runs, then letting the machine adjudicate true or false. A blind prediction is the test case I write for a piece of content. It’s also the same thing as giving an AI agent a “verification surface” and a sense of “pain”—stopping the agent from feeling good about itself in a feedback-free void, and instead slamming it into a machine-adjudicable loop, so that being wrong hurts, and hurting drives change.

The only difference is the source of feedback: for code it’s a unit test, for an agent it’s a verification environment, for content it’s that string of cold, hard real numbers T+N days later. Underneath, it’s all the same obsession—distrust the feeling, trust only the loop.

So cheat-on-content doesn’t feel like “crossing disciplines” to me. It’s more like swinging the same hammer at a nail nobody had thought to hit with a hammer before.

In this line of work, most people lose because they have no memory—they step in the same hole for three years running, because each “lesson” is invented after the fact and forgotten right after. I’m not expecting this to make me grok all the rules overnight. I’m only expecting it to do one humble thing: make sure that every time reality proves me wrong, something actually sticks.