Free AI log file analyzer: see every AI bot that crawled your site
Drop a server or CDN log. Get AI bot hits by bot, day, URL and status code. The file is read on your device and never uploaded.
Drop a log to find problems
or ·
Up to 50 MB per file · .log .txt .json .gz
How to fix blocked AI crawlers
Rules
robots.txt
Allow the search and user bots you want, even if you block training bots. Changes take effect when each bot rereads the file.
Blocks
Firewall and CDN
Managed rules that block "AI scrapers" often catch search bots too. Allow the user agents you want, by name.
Limits
Rate limits
A 429 means the bot hit a limit. Give search bots you want to be cited by a higher one.
# Allow AI search and user bots, block training
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
User-agent: CCBot
Disallow: /Not sure a rule works? Test it with the AI Bot Access Checker →
Training, search and user-triggered bots
AI companies send three kinds of bot. Blocking one kind has a different effect from blocking another.
Training
Training crawlers
Collect pages to train models. GPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent. Blocking them keeps your pages out of future training data.
Search
Search crawlers
Index pages so an assistant can cite them in answers. OAI-SearchBot, Claude-SearchBot, PerplexityBot. Blocking them can keep you out of cited sources.
User-triggered
User-triggered fetchers
Fetch a page when a person asks the assistant about it. ChatGPT-User, Claude-User, Perplexity-User. These are a buyer reading you through AI.
What each chart and status code means
| Status | What happened | What to do |
|---|---|---|
| 200 | The bot got the page. | Nothing. |
| 301, 302, 304 | Redirected or not modified. | Fine in small numbers. Long redirect chains waste crawl. |
| 401, 403 | Blocked, usually by a firewall, bot rule or login wall. | Check your CDN or WAF bot settings for the bots you want. |
| 429 | Rate limited. | Raise the limit for search bots you want to be cited by. |
| 5xx | Your server failed. | Check the URLs in your error logs. Bots retry less after repeated errors. |
How the AI log file analyzer works
Drop a log file
Combined or common format (Apache, Nginx), Cloudflare Logpush JSON or JSON lines. Up to 50 MB.
Your browser reads it
A background worker on this device matches each line against known AI user agents. It keeps counts and the paths bots requested in its memory, and nothing leaves the device.
See the AI bots
Hits by bot, by day, by URL and by status code, split into training, search and user-triggered bots.
AI crawler monitoring over time
Export one week of logs each month and run it here. Compare three numbers from run to run: the share of requests from AI bots, the blocked count, and which pages the search bots reach.
What a log file can and cannot tell you
User agents as written
A bot can fake its name. This tool counts user agents as they appear and cannot tell a real bot from a fake one, so read a count as requests that said they were that bot.
One file at a time
50 MB per file. Split longer periods by day, or filter AI lines first.
Crawls, not answers
A crawl makes a page available to cite. Whether answers name you is measured separately.
After the crawl
The bots can read you. Now see if ChatGPT names you.
Crawling makes your pages available. It does not show whether answers recommend you. Firemark asks your buyer questions on up to five engines on a schedule and counts the answers that name you and those that name a competitor.
- 25 to 500 prompts, depending on plan
- Up to five engines: ChatGPT, Perplexity, Gemini, AI Overviews and AI Mode
- Daily, weekly or monthly runs against 1 to 10 competitors
- Every answer kept for 13 months

Questions people ask
How do I see AI crawlers in my server logs?
Download your access log from your host or CDN and drop it here. The tool finds lines whose user agent belongs to an AI bot and counts them by bot, day, URL and status.
Is my log file uploaded anywhere?
No. The file is read by your browser on your device, in a background worker. Nothing is sent to Firemark or anyone else. Close the tab and it is gone.
Which log formats work?
Combined and common format (Apache, Nginx), Cloudflare Logpush JSON and JSON lines. Lines it cannot read are counted and shown as skipped.
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
GPTBot collects pages for training. OAI-SearchBot indexes pages so ChatGPT can cite them in search answers. ChatGPT-User fetches a page when a person asks ChatGPT about it.
Can a bot fake its user agent?
Yes. Anyone can send a bot's name as their user agent. This tool counts user agents as written and cannot tell a real GPTBot from a fake one, so read each count as requests that said they were that bot.
My file is bigger than 50 MB. What do I do?
Split it by day, or keep only the AI bot lines first with a grep command (the page shows it when you drop a larger file). Then drop the smaller file.
Does more AI crawling mean AI names my brand more?
Not by itself. Crawling makes a page available to cite. How often answers name you is a separate measure.
More free tools
Find out if AI recommends you
Start tracking your brand. New accounts get one free run.