# AI Agent Traffic Isn't Fraud. It's the Open Web's Unsold Inventory. *Agents polluted the retargeting pool, CPMs rose 20%, and budgets fled to Meta and Amazon. Filtering the machine audience is the wrong fix. Pricing it is the right one.* Published: 2026-09-08 | Read time: 3 min read | Author: Smalk AI Research | Source: https://www.smalk.ai/blog/ai-agent-traffic-open-web-unsold-inventory --- The buy side has found its bot problem, and it is solving it by leaving the open web. Digiday's September 8, 2026 reporting shows agencies narrowing retargeting pools contaminated by agents, paying roughly 20% more per thousand for the privilege, and shifting the rest of the budget to Meta, Amazon and Walmart. That is the wrong exit. AI agent traffic is not invalid traffic; it is the open web's unsold inventory, and treating it as waste hands the machine audience to the walled gardens. - Cloudflare reports that automated agents and bots now generate more than half of all web requests; the open web's audience data was built for a human majority that no longer exists. - Agencies quoted by Digiday responded by filtering retargeting pools and moving spend to logged-in platforms; the open web carries the cost, the walled gardens collect the budget. - The invalid-traffic reflex treats an agent shopping for a human as fraud. It is the most intent-rich visitor a page has ever had, and nobody sells it. - On September 15, 2026, Cloudflare's new defaults will block agent crawlers on ad-bearing pages, hardcoding the human-only ad model into infrastructure exactly when brands want more agent traffic. - Brands and publishers that price the agent visit as reach, instead of filtering it as noise, keep budget on the open web. ## What AI agent traffic is, and why it is not invalid traffic AI agent traffic is the set of web requests made by an AI assistant or agent on behalf of a person: comparing products, reading reviews, filling a cart, checking a price. Invalid traffic (IVT) is activity with no genuine human intent behind it: bot fraud, scraping, accidental clicks. The two are being lumped together in filters and dashboards, and the confusion is expensive. An agent shortlisting mattresses for a buyer carries more intent than most human sessions; it simply cannot see a banner. ## The retargeting economy vs the agent-visit economy The open-web ad economy ran on a recorded human visit. A person landed on a page, a pixel fired, an audience list formed, an auction priced the next impression, and the publisher earned a CPM. Every link in that chain assumed the visitor could be retargeted and could see an ad. In the agentic web the visit is still real, but the visitor is a machine reading the page for someone else. The intent is higher and formed earlier, the pixel fires into a pool that no longer describes people, and there is no ad unit the visitor can perceive. The old economy files that visit under waste. The new economy has no line item for it yet, which is the entire problem. ## What PPC Land and Digiday reported, and what we verified PPC Land's September 8, 2026 analysis, built on Digiday's same-day reporting by Sam Bradley, frames a split inside the buy side. GoFish president David Dweck says e-commerce clients saw bot traffic rise 80% year on year, retargeting pools were contaminated by agents adding to carts and signing up to newsletters, and narrowing those pools lifted average CPMs by about 20%; Digiday notes he gave no financial specifics, so treat the 20% as directional. Mellow Sleep co-founder Chad Keller says AI assistants send traffic that converts several times better than site average and wants more of it. Reuters, cited by Digiday, reports John Lewis's agentic searches grew from 0.3% to 2.5% of visits in a year. The headline number checks out. Cloudflare's July 1, 2026 press release states that automated agents and bots drive more than half of all web requests, and Cloudflare Radar put automated HTML traffic at 57.5% in June 2026. One caveat on the demand side: the Lunio finding that only 5.3% of marketers run an IVT tool while 75.6% believe they lose over 5% of budget to bots comes from a July 2026 vendor survey of 131 marketers' estimates, not from measured accounts. ## Why filtering agents protects the wrong ad model ### The case for treating agents as invalid traffic The strongest version of the filtering argument is operational, and Digiday's buyers make it well. Lighthouse Creative's Mallory Beck notes that bots inflate impressions and conversions, muddy lookalike audiences and eat frequency caps; her prescription is more rigour about what counts as a meaningful signal. Cloudflare's own rationale for its September 15 defaults is blunter: an ad on a page signals the owner meant a person to land there, so agent and training crawlers are blocked by default on ad-bearing pages. If you sell human attention, keep the non-humans out. ### Why the exit from the open web is the real cost Filtering does not invoice anyone; it raises the price of the humans who remain. Dweck's own account shows the sequence: the pool narrows, the CPM climbs about 20%, clients conclude programmatic is finished and move to Google, Amazon and Meta, where identity comes from a login rather than an inferred request. Collective Measures' Nola Ladd describes the same migration into retail media and social. The open web pays a bot tax that the walled gardens never owed. Meanwhile the traffic being filtered is the most valuable a brand can attract. Keller's conversion multiple is not noise; it is the funnel moving upstream, into the pages the agent read before it arrived. Blocking agents on ad pages by default, one week from now, shrinks the open web's footprint for the exact audience John Lewis is spending to attract. The rational move is not to purge the machine visitor but to price it. ## What this means for brands, agencies, and publishers ### For CMOs, media buyers and agencies: split the bot line before you cut it - Separate declared AI agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) from undeclared bots in analytics before touching retargeting rules; one is a channel, the other is waste. - Stop measuring agent traffic on the retargeting pool it pollutes and start measuring it on the conversions it delivers; Keller's multiple is the KPI, not the pool size. - Move budget upstream, to the publisher pages agents read while forming the shortlist, instead of downstream into a pool the agent will never see. - Before shifting spend to walled gardens, price the alternative: paying for placement on cited open-web pages is often cheaper than a 20% CPM tax plus a platform margin. ### For publishers: agents on your ad pages are inventory, not a threat - Check Cloudflare settings before September 15, 2026: the new default blocks agent and training crawlers on ad-bearing pages, which is most of your monetized footprint. - Keep verified agents in and undeclared scrapers out; the distinction is now a revenue decision, not only a security one. - Report agent visits to buyers as reach: which pages agents read, how often, and for which query categories; that report is the first draft of a rate card. - Add placements agents can read: native text units written for machine comprehension, alongside display for humans, so the same page earns from both audiences. ## Three signals to watch before the end of 2026 First, the September 15 Cloudflare default change: how many ad-supported publishers switch agent access back on will show whether the industry sees agents as audience or intruders. Second, Cloudflare's finding that 52% of crawler requests in June 2026 were for training, up from 22% in spring 2025; the referral-free share of machine traffic is growing, which sharpens the case for on-page monetization over referral hopes. Third, John Lewis's 0.3% to 2.5% trajectory: when a heritage retailer reports agentic sessions passing 5%, media plans will need an agent line whether agencies are ready or not. ## Conclusion Hold on to this: the buy side is treating the machine majority as a data-quality problem and solving it by leaving the open web, which is exactly the outcome the walled gardens were built for. Smalk AI takes the other side: Generative Engine Advertising treats AI agent traffic as inventory, placing native ads for AI agents on the publisher pages AI engines read and cite, so brands reach intent where it forms upstream and publishers get paid for the machine audience already on their pages. What to watch next: after Cloudflare's September 15 default flips, how many ad-supported publishers choose to let the agents back in, with a price attached. ## FAQ ### Is AI agent traffic the same as invalid traffic? No. Invalid traffic is activity with no genuine human intent behind it, such as fraud bots, scrapers and accidental clicks. AI agent traffic is a request made by an assistant acting for a real person with a real need, often after comparison and shortlisting. Lumping the two together in filters removes the highest-intent visitors a site receives. ### Why did agent traffic push retargeting CPMs up 20%? Agents add to carts and sign up to newsletters, so they pass the behavioral filters retargeting pools rely on and inflate the audience list. Agencies responded by narrowing targeting parameters, which shrinks the pool and raises the price of each remaining impression. GoFish reported an average CPM increase of about 20%, without disclosing financial specifics. ### Should brands block AI bots on their websites? Brands should classify before they block. Undeclared scrapers and fraud bots should be filtered; declared AI agents from ChatGPT, Claude, Perplexity and Google should be allowed, because they are forming purchase decisions on the buyer's behalf. Mellow Sleep and John Lewis both report that agent-driven traffic is worth attracting, not excluding. ### How can publishers monetize AI agent visits? Publishers can treat agent visits as a machine audience with its own inventory. That means keeping verified agents on cited pages, reporting agent reach to buyers, and running native text placements that agents can read and surface in answers. Advertising for AI agents pays for the visit itself, rather than hoping for a referral that rarely comes. ### What changes with Cloudflare's September 15, 2026 defaults? For new domains and for free-tier customers who have not changed settings, Cloudflare will block agent and training crawlers by default on pages that display ads, while continuing to allow search crawlers. Ad-supported publishers who want agent traffic on monetized pages will need to opt back in. Mixed crawlers that do not separate search, agent and training use will be blocked on all ad-bearing pages.