Your robots.txt says "no scrapers allowed." The scraper fetched your entire pricing page anyway. That is the uncomfortable truth about robots.txt: it is a request, not a lock. Honest crawlers — Googlebot, Bingbot, and the well-behaved AI crawlers — read it and comply. Bad bots treat it as a suggestion at best, and some use it as a map to the parts of your site you most wanted hidden. A honeypot trap gives you something robots.txt cannot: proof. You plant an invisible link that no human visitor and no rule-following bot will ever touch, and the first visitor that clicks it has identified itself as a bad bot. Then you block it.
This guide shows you how to build a honeypot from scratch: the hidden trap link, the server-side logging and blocking code for Apache and PHP, how to keep false positives near zero, and how to combine traps with rate limiting for a defense that actually holds.
Why robots.txt is not enough
Let us be clear about what robots.txt does. It is a plain-text file at the root of your site that asks crawlers to stay out of certain paths. There is no authentication, no encryption, no enforcement. A crawler has to voluntarily read the file and voluntarily obey it.
That works fine for search engines, whose business depends on webmasters trusting them. It does nothing against the visitors that cost you money and content:
- Content scrapers that copy your articles onto spam sites within minutes of publishing.
- Aggressive AI data collectors that ignore opt-outs and crawl at punishing speed.
- Price and inventory scrapers that hammer e-commerce sites around the clock.
- Vulnerability scanners probing
/wp-admin,/.env, and backup files.
You can and should still keep a correct robots.txt — test yours with our robots.txt tester — because it cleanly handles every bot that plays fair. But for the ones that do not, you need detection, not requests. That is what the honeypot is for.
How a honeypot trap works
The idea is borrowed from spam fighting, where hidden form fields catch bots that fill in every input. On a website, the trap is a link:
- You add a link to your pages that is invisible to human visitors — hidden with CSS, never shown on screen.
- You disallow the link's target path in robots.txt, so every rule-following crawler stays away from it.
- Anything that requests that path anyway has either ignored your robots.txt or is a scraper that follows every link it can parse. Either way, it is not a welcome guest.
- Your server logs the visitor's IP (and user-agent), and either blocks it automatically or adds it to a review queue.
The beauty of this approach is that it generates almost no false evidence. Real visitors cannot click what they cannot see. Search engines that obey robots.txt will never visit the trap path. The signal is clean — which is exactly what you need before you start blocking IP addresses.
Step 1: Build the trap link
Add the trap link to your site's footer or template so it appears on many pages. The target should be a path that exists only for the trap, for example /trap-for-bots/:
<a href="/trap-for-bots/" class="bot-trap" tabindex="-1" aria-hidden="true">Do not click this link</a>A few details matter here:
tabindex="-1"keeps keyboard users from tabbing onto it.aria-hidden="true"tells assistive technology to skip it, so screen-reader users are not sent into the trap.- Do not use link text that sounds tempting to a curious human ("secret deals"). The text is hidden anyway, but if it ever renders, it should look like nothing worth clicking.
Hiding it properly with CSS
How you hide the link affects who might stumble into it. The safest approach is to move it off-screen rather than using display: none:
.bot-trap {
position: absolute;
left: -9999px;
top: auto;
width: 1px;
height: 1px;
overflow: hidden;
}Off-screen positioning keeps the link in the document (so bots that parse HTML will find it) while keeping it out of every viewport. Avoid hiding it inside an HTML comment — most scrapers parse the DOM, not comments, and you want the trap to look like an ordinary link to anything reading your markup.
Disallow the trap path in robots.txt
This step is what makes the trap trustworthy. Add the trap path to your robots.txt:
User-agent: *
Disallow: /trap-for-bots/Now every honest crawler — Googlebot, Bingbot, the compliant AI crawlers — is instructed to stay away. If one of them shows up in your trap log, something is wrong with your setup, not with the bot. Use our robots.txt generator to build the file correctly if you are starting from scratch, and verify it with the robots.txt tester before you go live.
Step 2: Log every visitor that takes the bait
The trap path needs a handler that records the visitor and responds. You do not need anything fancy. A tiny PHP script works on virtually any shared host:
What the trap script does. It only needs three jobs:
- Read the visitor's IP address and user-agent string from the incoming request.
- Append one line to a log file on your server — date and time, IP address, user-agent.
- Answer with a plain "not found" page (HTTP 404), so the bot learns nothing useful.
In PHP this is roughly ten lines: build the log line from the request data, append it to the log file (use an exclusive lock so simultaneous hits cannot corrupt it), then send the 404 status and stop. Keep the log file outside your public web folder if your host allows it, and give the script no database credentials — a trap should be boring by design.
Route the trap path to this script. In Apache, add to your .htaccess:
Routing the trap URL to the script. On an Apache server, add one line to your .htaccess that internally maps requests for /trap-for-bots/ to the trap script (your host's docs describe the standard rewrite directive for internal rewrites). On nginx, the location block above does the same job: log the hit and return 404. Either way, real visitors never reach the trap URL because the link stays hidden with the CSS from step 1.
Two design choices here are deliberate. First, the script logs only the timestamp, IP, and user-agent — the minimum needed to identify the bot. Do not log full request bodies or anything personal; you want an abuse log, not a surveillance database. Second, it returns a 404, not a page explaining the trap. Telling a scraper "you have been caught" just teaches it to avoid your traps next time.
If you run nginx instead of Apache, the equivalent is a location block:
location = /trap-for-bots/ {
access_log /var/log/nginx/trap.log;
return 404;
}With this setup your trap log becomes a clean feed of confirmed bad actors: every IP in it followed a link that no honest visitor could have reached.
Step 3: Turn the log into blocks
Logging is detection. Blocking is enforcement. Here are three ways to close the loop, from simplest to most automated.
Option A: Manual review and .htaccess blocks
For small sites, review the trap log weekly and block confirmed offenders in .htaccess:
# Blocked: caught in honeypot trap 2026-10-19
Require all granted
Require not ip 203.0.113.77
Require not ip 198.51.100.23This is the Apache 2.4 syntax. (If your host still runs Apache 2.2, the equivalent is Deny from 203.0.113.77 — but you should really move to a host with a supported version.) Apache documents these access-control directives in detail in its access control how-to. If hand-editing .htaccess makes you nervous, our .htaccess generator builds the rules for you without syntax errors.
Option B: Automatic blocking with a PHP gate
For busier sites, let the trap script maintain its own blocklist. Extend the trap handler to append the IP to a blocklist file, then check that file at the top of your site's bootstrap or in a small include:
Checking the blocklist on every page. At the very top of your site's main entry file, read the visitor's IP address from the request, load your blocklist file into a list, and if the IP appears on it, stop immediately with a 403 "Forbidden" response. The cost is one small file read per visit — negligible next to everything else your pages do.
And in the trap script, after logging, add one line to grow the list:
When the trap script catches a bot, it also appends that IP address to the blocklist file (same append-with-lock technique as the log). Review the blocklist weekly and remove anything that looks like a legitimate service you actually want — search-engine crawlers that obey robots.txt will never touch the trap, so every address in the log earned its place.
Keep the blocklist file outside the web root if your host allows it, so its contents are never directly downloadable.
Option C: Block at the edge with Cloudflare
If you use Cloudflare, you do not need to touch your server at all. Create a WAF custom rule (or a User Agent Blocking rule) that blocks requests to the trap path — Cloudflare documents the setup in its WAF docs. Better yet, pipe your trap log into a Cloudflare IP list via the API on a schedule, so trapped IPs are blocked at the edge before they ever reach your origin. Edge blocking also protects you against the one thing server-level blocks cannot: a botnet rotating through thousands of IPs faster than you can add firewall rules by hand.
Keeping false positives near zero
A honeypot is only useful if you trust its signal. These are the false positives that actually happen in practice, and how to handle each:
- Your own testing. The number one source of trap hits on a new setup is the site owner clicking around with dev tools open. Test the trap once, note your IP, then ignore or whitelist it.
- Uptime monitors and link checkers. Services that check every link on your site (uptime monitors, SEO audit tools, broken-link checkers) will follow the trap link because that is their job. Whitelist the IPs or user-agents of tools you use, or exclude the trap path from their scans.
- Browser prefetching. Some browsers prefetch links on a page to speed up navigation. Prefetch generally skips links hidden with CSS, but to be safe, do not auto-block on a single hit — require the trap hit plus suspicious behavior (high request rate, ignored robots.txt elsewhere) before blocking.
- Security scanners you hired. If you run penetration tests or use a security scanning service, tell the tester about the trap or whitelist their source IPs. Their scanner doing its job looks exactly like a bad bot.
- Shared and carrier-grade NAT IPs. One IP can represent hundreds of mobile users behind carrier-grade NAT. A permanent block on such an IP punishes innocent visitors. Prefer temporary blocks (24–72 hours) for trap hits, and escalate to permanent only for repeat offenders.
- Good bots with a misconfigured robots.txt. If Googlebot ever appears in your trap log, your robots.txt disallow is broken or not yet picked up. Fix the file — do not block Googlebot.
The practical policy: log everything, auto-block only IPs that hit the trap and show at least one more bad signal (request rate above a threshold, or hits on known vulnerability paths like /.env or /wp-config.php.bak). Everything else goes to a review queue. This two-signal rule keeps the trap's precision high without letting obvious scrapers linger.
Combine traps with rate limiting
A honeypot catches bots that follow links. Rate limiting catches bots that do not need to — the ones hammering your search endpoint or API directly. The two techniques cover each other's blind spots:
- Honeypot identifies bad actors with high confidence, even at low request volumes. A scraper that politely takes one page per minute still walks into the trap.
- Rate limiting stops high-volume abuse regardless of identity. A botnet rotating IPs and ignoring the trap still gets throttled per IP.
In Apache you can layer simple rate limiting with mod_ratelimit or use mod_qos/mod_evasive if your host provides them; on Cloudflare, rate-limiting rules take a few clicks. Set generous limits (real users rarely exceed 60 requests per minute to a content site) so the limiter only bites genuine abuse, and let the honeypot handle identification. For a broader blocking strategy, see our guide to blocking AI crawlers and bad bots with .htaccess and the complete list of AI crawlers to know which user-agents deserve scrutiny in your logs.
Legal and ethical notes
A honeypot is a defensive tool, but a few boundaries keep it on the right side of both ethics and the law:
- Collect the minimum. IP address, timestamp, and user-agent are enough to identify abuse. Do not fingerprint browsers, set tracking cookies on trapped visitors, or store anything you do not need for blocking decisions.
- Respond quickly and neutrally. Return a 403 or 404 immediately. Do not tarpit bots with endless redirects or infinite generated pages — aggressive retaliation can look like an attack itself and may violate your host's terms.
- Respect privacy rules. IP addresses count as personal data under regulations like the GDPR. Have a short retention policy (30–90 days for trap logs is plenty), document why you keep the log, and mention it in your privacy policy.
- Do not use the trap offensively. Logging an IP to protect your own site is defense. Publishing "wall of shame" lists of trapped IPs, or using trap data to attack someone else's infrastructure, is not.
- Keep humans out by design. The off-screen CSS,
tabindex="-1", andaria-hidden="true"are not just polish — they are how you make sure no real person, including people using assistive technology, ever lands in your enforcement pipeline.
FAQ
Will a honeypot trap catch Googlebot?
Not if you set it up correctly. Disallow the trap path in robots.txt and every compliant crawler — Googlebot, Bingbot, and the well-behaved AI crawlers — will stay away. If a major search crawler does appear in your trap log, treat it as a configuration bug in your robots.txt, not as a bot to block.
How is a honeypot different from just blocking bad user-agents?
User-agent blocking relies on the bot announcing itself, and bad bots lie about their identity constantly. A honeypot relies on behavior: only a bot that ignores robots.txt and crawls hidden links ends up in the trap. Behavior is much harder to fake than a header.
Can I put trap links on every page?
Yes — putting the trap in a shared footer or template is the standard approach, because it maximizes the chance a scraper encounters it. One trap path per site is enough; what matters is that the link appears wherever bots crawl.
Should trapped IPs be blocked forever?
Usually not. Temporary blocks of 24–72 hours handle most scrapers, since they move on to easier targets. Reserve permanent blocks for IPs that return to the trap repeatedly, and remember that IPs get reassigned — a permanent block today can hit an innocent user next month.
Do I still need robots.txt if I have a honeypot?
Absolutely. They do different jobs. Robots.txt gives honest bots clean instructions (which keeps your crawl budget healthy and your SEO intact), while the honeypot catches the dishonest ones. Run both — and keep the robots.txt correct with our robots.txt tester.
Will this stop AI crawlers specifically?
It stops the AI crawlers that ignore robots.txt, because ignoring robots.txt is exactly what springs the trap. Compliant AI crawlers never touch the trap path. If you want to manage AI crawler access declaratively as well, generate proper rules with our robots.txt generator and ai.txt generator first, then let the honeypot handle the ones that refuse to listen.
Bottom line: robots.txt asks, the honeypot proves. Plant an invisible link, disallow it for honest bots, log whatever walks in anyway, and block the repeat offenders. It is a small amount of code for a large upgrade in who gets to take your content.
