I prompted ChatGPT, Claude, Perplexity and watched my Nginx logs — and the answer wasn't what marketers sell

The experiment

A developer fed identical prompts to five AI assistants and did the unfashionable thing: read the nginx access log. The goal was simple — when ChatGPT, Claude, Perplexity, Gemini or Google AI Mode cites a page, did the model actually fetch the origin in real time, or was it answering from a prebuilt index? The blog post walks through a custom log format, tail -F sessions next to each AI, and repeated prompts so caches wouldn’t mask live retrievals. Old-school, but honest.

What the logs revealed

The logs capture two very different signals. One is a provider-side fetch: a bot-like request with a vendor-specific user-agent, no referrer, and often a short burst of hits (sometimes from multiple IPs) while the model composes. The other is a human clickthrough: a normal browser user-agent, cookies present, and the AI product as the referrer. The distinction matters — conflating them into a single “AI traffic” number is like counting both the person who thumbs through a book and the one who reads it cover-to-cover.

It has been reported that the post reproducibly captured provider-side retrievals for ChatGPT and Claude. ChatGPT requests appeared with a ChatGPT-User/1.0 token, no referrer, and tight bursts across multiple source IPs — matching OpenAI’s bot documentation. Claude showed Claude-User/1.0, a robots.txt preflight, and ordinary redirect handling consistent with Anthropic’s crawler notes. Perplexity’s traffic reportedly bore a Perplexity-User/1.0 string; the excerpt available to reporters here truncates the rest of that section, but the pattern is clear: some AI assistants do fetch origins at query time.

Why it matters

So is your site being read live by AIs? Sometimes, yes — and sometimes what gets reported as “AI traffic” is actually downstream human clicks driven by those models. That’s a big difference for analytics, SEO, and privacy. Marketers and publishers should be careful which signal they lean on: a provider-side fetch tells you the model looked at you; a clickthrough tells you a human cared enough to follow the link. It’s a small detail, but it changes the story — and in the fast-moving world of generative AI, details matter.

Sources: surfacedby.com, Hacker News