Should You Paywall or Private Old Posts to Stop AI Scrapers?

By Idea to Traffic Editorial Team · Published October 29, 2026 · Last updated October 29, 2026

A long-time blogger on r/Blogging asked this week whether to put older posts behind a paywall because AI bots were scraping years of tested, photographed, hand-written work. They had already switched more than 200 WordPress posts to private. It is an understandable reaction, and the question deserves a clear answer, because the way you lock content decides whether Google keeps sending you visitors at all.

The short version: hiding a post from everyone also hides it from Google, and that traffic is gone. Protecting a post from scrapers while keeping it in Search needs a different setup. Here is what each option does, with the commands to check your own site.

What setting a WordPress post to private actually does

In WordPress, a post with Visibility: Private is only shown to logged-in users with the right role. Everyone else, including Googlebot, gets a 404 Not Found. You can see exactly what a crawler sees with one command from any terminal:

curl -s -o /dev/null -w "%{http_code}\n" -A "Googlebot" https://yourblog.com/old-post/

If that prints 404, Google will treat the page as removed the next time it crawls the URL. For a post that gets crawled often that can take days; for a quiet archive post it can take weeks. After that:

• The URL drops out of Google's index and stops earning impressions and clicks.
• Backlinks pointing at that post now point at a dead page, so the value they passed to your site stops flowing.
• Internal links from your public posts lead readers to an error page, which wastes the visits you still have.
• If the post earned AdSense or affiliate income, that income stops with the traffic.

A 404 does not penalize the rest of your site. Google's position has long been that missing pages are a normal part of the web. The damage is simply the loss of whatever those posts were bringing in, plus the links they were holding. For a blog that ranks well on 200 old posts, that can be most of its search traffic.

Switching the posts back to public usually gets them reindexed, but there is no promise the old positions come back. Competitors fill gaps quickly, so the longer the posts stay hidden, the harder the recovery.

The three ways to lock content, and what Google sees for each

Not every paywall works the same way. What matters for SEO is what HTML the server sends to Googlebot.

1. Hard lock (private, members-only, login required). The server sends no article text to anyone without an account. Scrapers get nothing, and neither does Google. Expect the post to disappear from Search, the same as setting it private.

2. Teaser lock. The page is public, but only the first two or three paragraphs are in the HTML. The rest loads after login. Google indexes the teaser and can rank the page for what the teaser covers, which is usually far less than the full post ranked for. Scrapers also only get the teaser. This is the honest middle ground for posts you want to keep visible but not give away.

3. Full text to Googlebot, paywall to visitors. The full article is in the HTML, but a script or overlay hides it from readers who are not subscribed. Google allows this if you mark it up with its paywalled content structured data, otherwise it can look like cloaking. The catch for your goal: the full text is now in the page source, so any scraper that reads the raw HTML, or that pretends to be Googlebot, gets the whole post. And Google itself can still use that text to build AI Overviews. This option protects your income, not your words.

The markup Google asks for on paywalled posts

If you go with option 2 or 3, wrap the locked part in an element with a class, then add this JSON-LD to the page. Google's documentation for subscription and paywalled content uses the same structure.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "BlogPosting",
  "headline": "Your post title",
  "isAccessibleForFree": false,
  "hasPart": {
    "@type": "WebPageElement",
    "isAccessibleForFree": false,
    "cssSelector": ".members-only"
  }
}
</script>

Test the live URL in Google's Rich Results Test afterward. If you use a membership plugin, check whether it already outputs this before adding your own, since two conflicting blocks cause more trouble than none.

If your concern is specifically that Google's AI answers summarize your post so well nobody clicks, you have a narrower tool: data-nosnippet on any HTML element, or a max-snippet robots meta value, limits what Google can show from that text in results, including AI features. It also limits your normal snippets, so use it on the parts you most want to protect, not the whole page.

Stopping scrapers without hiding from Google

Most of what bloggers call AI scraping falls into two groups, and they need different fixes.

Polite AI crawlers identify themselves and follow robots.txt. You can refuse model training without touching Google Search by adding lines like these:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Google-Extended is a control token, not a separate crawler. It tells Google not to use your content for training its Gemini models, and Google states it does not affect your inclusion or ranking in Search. It does not keep your content out of AI Overviews, because those run on the normal Googlebot crawl.

Impolite scrapers ignore robots.txt and often fake a Googlebot user agent. For these you need the server or CDN to act:

• On Cloudflare, open your domain and turn on the Block AI bots setting (it sits under Security, in the Bots or Settings section depending on your dashboard version), or use AI Crawl Control to allow or block crawlers one by one. Verified search engine bots stay allowed.
• Add a rate limit rule so any single IP that requests more than, say, 60 pages a minute is challenged. Real readers never hit that.
• If you see "Googlebot" in your logs from an odd IP, verify it. Real Googlebot resolves back to Google:

host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1

If the reverse lookup does not end in googlebot.com or google.com, or the forward lookup does not return the same IP, it is a fake and you can block it.

None of this is perfect. A determined scraper with residential proxies can still copy a public page. But these steps remove the bulk of automated copying while your posts keep earning.

Decide post by post, using your own data

Instead of locking the whole archive, sort it. In Search Console, open Performance > Search results, set the date range to the last 16 months, open the Pages tab and export the list.

• Posts with steady clicks: keep them public. They are your traffic and your income. Protect them with robots.txt and bot rules only.
• Posts with impressions but few clicks: these are the ones AI answers are most likely absorbing. Consider a teaser lock with markup, or data-nosnippet on your tested results, and add something a summary cannot replace, such as your own photos, measurements or a downloadable checklist.
• Posts with zero impressions: locking them costs you nothing in Search. These are the safe ones to make members-only.

If you already set a batch to private, run the curl command above on a sample, then flip back to public the ones that had clicks in your export. The sooner they return, the better the odds Google restores them where they were. Use the URL Inspection tool and Request indexing on the most important few.

Before and after any change like this, it helps to confirm what your public pages are actually serving: status codes, canonical tags, robots rules and noindex tags. The free Idea to Traffic site checker runs those checks on any URL in about ten seconds, so you can catch a post that went private by accident.

Where this question came from

• r/Blogging: Putting older content behind a paywall to discourage AI scrapers?
• Google Search Central: Structured data for subscription and paywalled content
• Search Engine Roundtable: Over 80% of Google searches have AI Overviews

Questions about locking old posts and AI scrapers