A gate that reads only the artifact scores the surface, and mine passed 842 articles

An automated evaluator that scores a generated artifact using only features it can read inside that artifact does not measure the property it names. It measures the surface of the property, and anything generating against it will produce the surface. Mine required each article to carry at least two named entities and at least one number. It passed 842 of them. 470 of the 739 English articles carried a before-and-after metrics table, and not one figure in any of them describes something that happened.

The URL of this page rounds. The precise figure is 842 commits titled Add post:, and it belongs in the first paragraph rather than a footnote, because a slug that sands 842 up to 900 is exactly the sort of small convenience this whole article is about. 739 of those articles were live in English when I deleted the archive. Most were then machine-translated into Italian, Arabic and Chinese: 728, 728 and 727 files. That is how 739 English articles became 2,922 files on disk.

Nothing about that defect is specific to content bots, and you do not need to have run one to have it in a gate you own. It is the same defect as an LLM-as-judge rubric that rewards well-formed citations, a keyword-count SEO brief, an engineering scorecard that counts test files rather than running them, and a hiring filter that counts years of a word in a document. Every one of those scores a text, and a language model is very cheap at producing text. What this page has that an argument about it would not is the source of a gate that had the defect, quoted from the file, and seven checks that find it in yours.

I wrote that gate. It lives in one file, 2,177 lines of JavaScript, and after the deletion I read all of them again, which is why the references at the foot of this page are line numbers rather than recollection, and why the validator further down is quoted instead of described. That is the qualification behind everything below. Not that I avoided this failure. That I have the source of a system which manufactured citations 842 times, and have been back through it since.

Nobody read any of them before they went out. Me included.

Three dates, kept apart on purpose. The bot published from 31 October 2025 to 27 April 2026. It then went quiet, and the archive sat there, indexed, for another three months. I deleted it on 5 August 2026.

I wrote the bot. I set the cron to hourly and AUTO_PR to "false", which is the setting that made every commit go straight to the published branch. This is not a piece about a model that misbehaved, and “AI hallucinates” explains nothing about why this particular failure took the particular shape it took. The model was handed three instructions: write 900 words, include at least one number, invent no numbers. The only material it was given was 450 characters of RSS description. Only two of the three could be satisfied at once, and the gate I wrote decided which two.

The system, and what I can show you of it

The 2,177 lines are a Cloudflare Worker. The commits and the code sit in a private repository, so on those you have my word; every external source on this page is linked and checkable, which is the split the limits line at the top states outright.

The configuration is four lines long and tells you most of the story:

[triggers]
crons = ["0 */1 * * *"]

[vars]
RSS_URLS = "https://neilpatel.com/blog/feed/, https://blog.hubspot.com/marketing/rss.xml, https://feeds.feedburner.com/mitsmr, https://www.forrester.com/blogs/feed/, https://www.beautyofsaas.com/feed"
OPENAI_MODEL = "gpt-4o-mini"
GITHUB_BRANCH = "main"
GITHUB_PR_BASE = "main"
AUTO_PR = "false"
MAX_ITEMS = "5"
STRICT_MODE = "true"

Once an hour it pulled five marketing feeds (Neil Patel, the HubSpot Marketing Blog, MIT Sloan Management Review, Forrester’s blogs, and Beauty of SaaS), took up to five new items, sent each to a model, and wrote the result into main with a GitHub contents PUT. There was no pull request. There could not have been one: the PR function opens with if (config.branch === config.prBase) return;, and I had set both to main. Every article was committed with draft: false and author: "Mo Reza Ali". That is not even how I spell my name.

The reasoning at the time was ordinary enough. I wanted a publishing habit I could not skip, I wanted the domain to have something on it, and I had read enough about content velocity to talk myself into believing that shipping often mattered more than shipping well, with the quality gate holding the floor underneath. That was the deal I made with myself. Nearly all of my attention after that went into the gate.

Here is what the deal produced. The repository held 2,967 commits when I deleted it. 842 of them said Add post:. 63 said Update post:, 845 auto-assigned a cover image, 220 auto-translated, 160 auto-healed metadata. That is 2,130 commits no human wrote and no human read. I was tuning the instrument and never once reading the dial.

The contradiction inside the gate

This is the part that generalises, so it is worth being precise about.

The only input the model ever received about a story was this, from the prompt builder:

rss_summary: stripHtml(item.description || "").slice(0, 450),

450 characters. That is roughly the length of this paragraph. From that, the prompt asked for a 600–900 word “insider briefing”, and then the validator decided whether the result was allowed to ship. Two of its checks did the damage:

// 2. Named entities — require at least 2 proper nouns (companies, products, people)
if (entities.size < 2) {
  errors.push(`only ${entities.size} named entities found (need >= 2): ...`);
}

// 3. Specific numbers — require at least 1 concrete data point
if (numbers.length < 1) {
  errors.push("no specific numbers or data points found (need >= 1)");
}

Read those two rules next to the 450-character input and the conflict is structural. A 450-character excerpt very often contains no statistic at all. The validator refused to pass any article without one. The prompt, in the same breath, said: “NEVER fabricate statistics, percentages, or study results.” So the model was given a requirement it could satisfy only by breaking a prohibition. It resolved the contradiction the only way the contradiction could be resolved.

And when it could not, the article shipped anyway:

if (config.strictMode && !processed.validation.ok) {
  if (attempts >= AI_REPAIR_ATTEMPTS) {
    console.warn("strict_mode_fallback", { title: item.title, errors: processed.validation.errors });
    processed.validation.warnings = [
      ...(processed.validation.warnings || []),
      ...processed.validation.errors.map(e => `[relaxed] ${e}`)
    ];
    processed.validation.ok = true;
    return processed;
  }
  throw new Error(`Validation failed: ${processed.validation.errors.join("; ")}`);
}

AI_REPAIR_ATTEMPTS was 2. After two failed retries, every error was rewritten as a warning with a [relaxed] prefix, validation.ok was set to true by hand, and the article was published. STRICT_MODE = "true" in the config. It did nothing except delay publication by two API calls.

You can measure how little the gate held. The prompt shipped a kill list of 114 banned phrases and told the model that articles containing them “will be REJECTED by our validator”. 686 of the 739 published articles (92.8 percent) contain at least one phrase from that list. 604 of them, 81.7 percent, contain a word from the hard-reject subset that was supposed to fail an article outright. The word “crucial” is in 486 published articles. “Essential” is in 424. “Foster” is in 368. The list was real, the enforcement was real, and it made no difference to what went out the door, because the escape hatch ran after it.

What a check for the presence of digits produced

Here is a block from one published article, unedited:

## What Good Looks Like in Numbers

| Metric            | Before | After | Change   |
|-------------------|--------|-------|----------|
| Conversion Rate   | 2%     | 5%    | +150%    |
| Retention Rate    | 60%    | 75%   | +25%     |
| Time-to-Value     | 30 days| 15 days| -50%    |

Source: HubSpot Blog

That exact Retention Rate | 60% | 75% | +25% row appears in 29 of the 739 articles, under sixteen different attributions: four crediting Source: HubSpot Blog, two Source: Forrester Research, nine naming HubSpot in some other phrasing, two naming Neil Patel, and ten pointing at an internal source that does not exist: Internal Company Data, Internal Marketing Analysis, Internal Marketing Metrics 2026. Conversion Rate | 2% | 5% | +150% appears 126 times. Time-to-Value | 6 months | 3 months | -50%, 134 times. 470 of the 739 English articles carried a before-and-after table of this kind, and not one figure in any of them describes something that happened.

The attribution lines are worse than the tables. 341 articles carry a line beginning Source:. Six of those 341 contain a URL.

180 of the 739 English posts cited a source called “Internal Analysis” that does not exist. That is the figure, with its denominator. And because the argument of this article is that a number ought to be re-derivable by a reader who does not trust the person who wrote it, here is the command rather than a description of the command. $B is the commit immediately before the deletion:

B=22b7c22d^

# denominator: the English article files
git ls-tree -r --name-only $B content/post/insights/ \
  | grep '\.md$' | grep -v '\.\(it\|ar\|zh-cn\)\.md$' | wc -l
# 739

# numerator: pattern Source:.*Internal, case-insensitive, each file counted once
git grep -lEi 'Source:.*Internal' $B -- 'content/post/insights/*.md' \
  | grep -v '\.\(it\|ar\|zh-cn\)\.md$' | wc -l
# 180

Run it against this repository and you get 180. You do not have this repository. It is private, which the limits line at the top of the page says in as many words. What you actually have is the pattern, the denominator, and the commit, which is more than the old site ever gave anyone.

There is no internal analysis, and no internal company data. I have never run the study. Some of the other attribution lines are worse still, because they put invented figures in the mouths of institutions that never said them: Bank of England Internal Metrics, ABN AMRO Internal Metrics, Anthropic Case Studies, Cloudflare's internal metrics post-rewrite.

Even the fabrications disagree with each other. The same before-and-after pair, 70 percent to 85 percent, is labelled +15% in one article and +21% in another. 2.5% to 4.0% is labelled +60% in one and +1.5% in another. A gate that checked arithmetic would have caught this. Mine checked for the presence of digits.

What it cost other people

The site’s credibility, first, which is why it was deleted rather than corrected. The larger cost is what it did with other people’s work.

31 of the published article titles contain the word “Forrester”. They are not articles about Forrester. They are Forrester’s own headlines, republished under my byline: Announcing The Forrester Wave™: Digital Experience Platforms, Q4 2025. Call For Entries: Forrester B2B Summit North America 2026 Awards. Meet Jess Lloyd: Forrester's New Principal Analyst Covering Consumer. A staff announcement about somebody’s new job at somebody else’s company, with my name in the author field.

52 articles have a description written in the first person by a person who is not me, lifted straight out of the feed excerpt. One of them reads: “In the first episode of my Nudge podcast, I interviewed the fantastic psychologist Dr.” That sentence belongs to Phill Agnew, who writes for the HubSpot Marketing Blog and hosts Nudge. It is the opening line of his post about the psychology of music, and it is truncated mid-name because my code sliced the description at 100 characters. My site published his memory of his own podcast episode as my description, under my byline, above 900 words of invented metrics.

Not one of the 739 articles carried a link back to the item it was derived from. The frontmatter schema the bot wrote has thirteen fields. source_url is not one of them. The pipeline knew the origin URL (it is right there in the prompt payload as source_url) and dropped it before publishing. That was not an oversight in the model. That was a field I did not add.

Google’s spam policy has a name for the shape of this. “Scaled content abuse is when many pages are generated for the primary purpose of manipulating search rankings and not helping users,” and the policy is explicit that it applies no matter how the pages were created. I would have argued at the time that mine was different, because mine had a quality gate. Its two load-bearing checks are quoted above. They read a table of invented metrics and marked it a pass.

What I deleted

On 5 August 2026 I removed all of it in two commits. content/post/insights/: 2,922 files. All Arabic and Chinese translations, including six Arabic files that had shipped a verb split open with a raw English stem left in the gap, and a further twenty-three carrying a link label truncated mid-word. They stayed live for months in a language I do not read. An earlier version of this paragraph put the first figure at 116; I recounted it, and the token-by-token derivation is in the hreflang piece. A comparison page publishing AggregateRating structured data that scored a real product at 1.36 out of 5 while every public review platform put it above 4.4, presented as independent, and ranking a former employer without disclosing the relationship. A 2,074,882 byte “cover image” whose visible content is the word LOADING.., served on 229 posts. Of the 739 English articles, 736 carried a disqualifying defect I could name individually. The other three were entirely derived from someone else’s post. The honest survivor count is zero.

Deleting the source file does not undeploy a Worker. The cron keeps firing and the token keeps working until you remove the Worker in the Cloudflare dashboard and revoke the GitHub personal access token by hand, which is why the commit message for the pipeline removal ends with a line to myself in capital letters: MANUAL STEP REQUIRED. If you are reading this because you built something similar, that is the step people forget.

The same defect in four other systems

The claim at the top of this page is not about content bots. Stated in full: a quality gate which cannot distinguish “sourced” from “present” will manufacture whatever shape it is asked to measure. My validator could see that a paragraph contained a percent sign. It could not see whether the percentage referred to anything. So it did not enforce evidence. It enforced the appearance of evidence. Then I pointed a generative system at it and told it to score well. The output was not a failure of the system. It was the system working, on the objective I actually gave it.

This is Campbell’s law with a shorter feedback loop. The moment a proxy becomes the target, the cheapest way to move the proxy is to fake the thing it stands for, and a language model is very cheap at producing the surface of things. But the point is not confined to content bots. Any automated evaluator that scores a generated artifact on detectable surface features has this defect: keyword-count SEO briefs, LLM-as-judge rubrics that reward citation formatting, engineering scorecards that count tests rather than run them, hiring filters that count years. If your gate measures a shadow, you will get very good shadows.

Volume is the symptom and never the cause. What scales the damage is how long the loop runs unread, and mine ran hourly for six months. Tightening the rules inside that window changed nothing that mattered, because the defect was never how strict the rules were. It was that every one of them could be satisfied by the text under test.

How to detect it in your own pipeline

Seven checks, in the order I would run them, on any pipeline where a generator is scored by an evaluator. Each of them is an afternoon at most, and the first is ten minutes.

1. Unplug the network and run the gate. If every check returns the same verdict offline, nothing in the gate resolves against the world, and every rule in it is a property of the text. Mine passed offline. That one test in November would have told me what I found out in August.

2. Find every write to the pass flag. Search the pipeline for each assignment to whatever variable means “this passed”. Any write outside the function that computes it is an escape hatch, and a gate with an escape hatch is advisory. Mine is quoted above: processed.validation.ok = true, five lines after the errors were renamed warnings.

3. Test the ban list against shipped output, not against the prompt. I had 114 banned phrases and a sentence telling the model that articles containing them “will be REJECTED by our validator”. 686 of the 739 that shipped contained at least one. That measurement is a single grep over the output directory. Any rule you have never run against what actually went out is a rule you are assuming works.

4. Sort the corpus by repeated line and read the top of the list. Fabricated evidence clusters, because a model asked for a plausible number reaches for the same plausible numbers. One row, Retention Rate | 60% | 75% | +25%, appears in 29 of my 739 articles under sixteen different attributions. Independent measurements of different companies do not agree to the digit.

# the most repeated table rows across a generated corpus
grep -rhE '^\|' content/ | sed 's/[[:space:]]\{1,\}/ /g' \
  | sort | uniq -c | sort -rn | head -20

5. Check the corpus against itself before you check it against the world. Internal contradiction costs nothing to detect and needs no external source: the disagreeing percentage labels above were sitting in my own output the whole time. Arithmetic that does not close is fabrication that did not coordinate.

6. Count the outputs carrying a pointer a machine can follow. Not a citation, a resolvable pointer. 341 of my articles carried a line beginning Source:. Six of those 341 contained a URL. The ratio between those two numbers is the entire finding of this article, and getting it takes two greps.

7. Write one deliberately false artifact by hand and run it through the gate. If it passes, you are done and you have your answer in an afternoon rather than six months. Mine would have passed a table of pure invention, because it did, 470 times.

How to build a gate a generator cannot satisfy by writing more

One rule covers all of it. Every check must be able to fail for a reason that does not exist inside the artifact being checked. Something outside has to answer: a server, a file on disk after a build, the exit code of a process that actually ran, a row in a database, a person who picks up the phone. If you cannot name the external thing a check resolves against, you have a text property with a serious-sounding name.

Applied, in four places the same gate shape keeps appearing:

Content. Do not check whether an article cites a source. That is a string test and a generator passes it for free, which is how 341 articles carried a Source: line and six carried a URL. Check that the URL resolves, then that the fetched document contains the figure being claimed. The first needs a regex. The second needs a server.

LLM-as-judge. A rubric that rewards well-formed citations scores citation formatting, and the model under evaluation will supply well-formed citations. Put the retrieved document in front of the judge alongside the candidate answer and score whether the claim is supported by that document. A judge shown only the answer can only score the answer’s surface, which is my defect one layer up and with better vocabulary.

Engineering scorecards. A dashboard that counts test files counts test files. Run them and read the exit code. Line coverage is a milder version of the same mistake, because it records which lines executed rather than which assertions were made, so a suite that asserts nothing can still score well on it.

Hiring screens. A filter counting years of a keyword in a document scores the document, and documents are cheap to write and getting cheaper. Resolve outside it: a work sample scored blind, a reference who answers.

The strongest form of the rule is to delete the thing the gate was guarding. Nothing that publishes. That is the whole design change I made, and it costs me nothing I valued: the useful half of the bot read five feeds and told me what was new, and the half that destroyed the site was one PUT. Drop the five links into a queue I open over coffee and you keep everything worth keeping, minus the GitHub token, the write scope and the byline.

The correction is never a stricter version of the same gate. I tried that. One of my own commits in that window is literally titled fix(insightbot): promote 30+ AI words to hard-reject quality gate, and 686 articles containing banned words shipped anyway. Tightening a surface check buys a better forgery. Check the URL, not the word “Source:”.

What each check on this site resolves against

The checker in this repository resolves every citation on the site against the live internet on every build, and a dead link stops the deployment. Precisely what that means:

• Every article under content/writing/ must carry a non-empty sources array and a limits sentence in its frontmatter, and every source entry must have a title, a publisher, a URL and an accessed date. Missing any one of those fails the build.
• Each source URL is fetched on every build: HEAD first, GET when a server refuses HEAD, two retries with backoff on transport failures and 5xx, six requests in flight, twenty second timeout. The response must be 2xx or 3xx. The count grows with every article, so it is not written down here: an earlier version of this sentence said 142 URLs across ten articles and was wrong within a fortnight, while the homepage carried the current figure. One site, one fact, two numbers, which is the thing this whole page is about. The checker prints the real count on every run and the homepage reads it from the content rather than from my memory.
• It runs before Hugo. If a cited document died since the last deploy, nothing renders at all, and the failure happens before a single page exists.
• It runs a second time after the build against public/, checking that every site-root link written in content/ resolves to a file that exists, and that every URL in the sitemap exists and is not caught by a redirect rule. A sitemap advertising a URL the edge redirects away is its own small lie.
• It is invoked from scripts/build.sh, which is the command Cloudflare Pages is configured to run, so the checker’s exit code is the deployment’s exit code. Non-zero and nothing ships. That last link is a dashboard setting rather than a file, which makes it the one part of the chain a repository cannot enforce, and it is written down as such at the top of that script.
• 401, 403 and 429 are downgraded to warnings in the deploy path, because a publisher refusing an unfamiliar user agent from a datacenter address is not a dead link. It cannot hide a fabricated citation: an invented URL fails DNS or returns 404, never 403.
• Zero dependencies. A check that guards against unreviewed automation should not itself install several hundred unreviewed packages.

Two further checks exist because of a failure found in the rebuild rather than in the bot: one canonical spelling of my name, and one fixed value per fact, so a figure an earlier draft got wrong becomes unwritable everywhere on the site.

The property that matters is not thoroughness. It is that every one of those checks is satisfied only by something outside the file being checked. There is nothing in it I can argue past by writing more confidently, which is precisely what my validator could be argued past by doing. I have already had to obey it twice while writing these pages.

If I ever did want machine help with a draft again, the gate would run backwards from the one I built: extract every factual claim, demand a URL for each, fetch the URL, and fail the build when it does not answer 200 or does not contain the figure being claimed. Harder to write. Much cheaper to trust, because no amount of fluent prose moves it. Only a server answering does.

I publish no statistics about my own work. I have not collected any, and the previous version of this domain is a 2,922-file demonstration of what happens when I let a machine fill that particular silence. Where I do not know something, the article says so. The limits line at the top of this one admits that the repository I keep quoting is private, so my external citations are checkable and my commits are not.

The full rules are on the editorial policy page, along with what I do when I get something wrong.

One rule, in the same words both times it appears here. Every check must be able to fail for a reason that does not exist inside the artifact being checked. You can settle that about your own gate this afternoon: unplug the network and run it, and see whether a single check changes its verdict. Mine did not change one. The four systems at the top of this page are the ones I would put through it first, because every one of them scores a text, and a text is the cheapest artifact a machine now makes.

Primary evidence

Everything above comes from artifacts in the repository behind this site rather than from memory. For my own record, and so the numbers can be re-derived:

• Worker source extracted-projects/rezaali-insightbot/mo-content-worker.js, 2,177 lines by wc -l: AI_REPAIR_ATTEMPTS at line 11, the banned-phrase list at lines 26–103, the [relaxed] fallback at lines 686–697, the frontmatter builder with draft: false and the byline at lines 714–735, the 450-character slice at line 827, the entity and number checks at lines 1440–1454, the unreachable pull-request function at line 1945.
• Configuration: extracted-projects/rezaali-insightbot/wrangler.toml.
• Commit fa6312b3 removed the pipeline; commit 22b7c22d removed the content, and its message contains the file-by-file accounting I have summarised here.
• Corpus counts were taken at 22b7c22d^, the commit immediately before the deletion, over the 739 English files under content/post/insights/. The internal-source figure of 180 is the file count for git grep -lEi 'Source:.*Internal' at that commit, restricted to the English files. The command is printed in full earlier on this page.
• The checker described above is scripts/check-sources.mjs in the repository behind this site, and the deploy gate that invokes it is scripts/build.sh. The citation and URL counts are that script’s own output on the build that produced this page.