The Marsen toolbox / AI Visibility
Free AI Crawler Checker
Check whether your website tells major AI crawlers they can access your pages.
Start with your input
Run the AI Crawler Checker
Fetch a public page and its robots.txt, then evaluate declared rules for eight search and AI user agents.
What this tool checks
See what the result is based on.
Declared rules do not prove actual crawler access through a CDN or firewall, or confirm indexing.
Googlebot and Bingbot
Inspect declared rules for the two search crawlers.
OpenAI user-agent rules
Compare the declared OpenAI crawler rules; search and training are different uses.
Claude and Perplexity rules
Inspect the named AI user-agent rules in robots.txt.
Google-Extended rules
Review the separate Google-Extended declaration.
Meta and header noindex
Check for noindex declarations in page metadata and response headers.
Declared sitemap URLs
List sitemap locations declared by the website.
Methodology
How to use your result.
A trustworthy tool should show what it used, what it could not verify, and how each recommendation connects to the outcome.
- 01
Validate the input
Confirm the domain or values are usable before starting analysis.
- 02
Review what was found
Use only accessible sources and approved connections relevant to the check.
- 03
Check the details
Distinguish confirmed signals, likely explanations, and items requiring manual review.
- 04
Choose what to fix first
Order findings by impact, confidence, effort, risk, and ownership.
- 05
Measure the change
Check whether the change made the page or process better.
Use the result responsibly
What to keep in mind.
A tool can flag details worth checking. Use your knowledge of the business to decide which changes make sense.
Useful for
- Finding a structured starting point
- Surfacing evidence to verify
- Prioritizing a focused review
- Creating a shared action list
Still requires judgement
- Commercial and strategic decisions
- Legal, privacy, and provider constraints
- Implementation feasibility
- Whether a recommendation fits the business
Need to check something else?
Related free tools.
These tools can help you look at another part of the same problem.
Is the AI Crawler Checker live and free?
Yes. The runner is live and does not require an account or a sales call.
How does the AI Crawler Checker work?
Fetch a public page and its robots.txt, then evaluate declared rules for eight search and AI user agents.
What are its limits?
Declared rules do not prove actual crawler access through a CDN or firewall, or confirm indexing.
Are results saved?
Reports are not saved in a database. Public website checks send the supplied URL to Marsen’s server; hosting providers may retain operational logs. Answer samples, schema input and ROI values stay in the browser.
Does a passed check guarantee rankings or conversions?
No. It confirms an observed condition in this specific input. Review the evidence, context and limitations before making changes.
What if the website cannot be fetched?
The tool shows an error for blocked, private, oversized, unsupported or timed-out responses. It does not fill in a score when evidence is unavailable.
Can Marsen help with the next step?
Yes. Use the result to scope a focused implementation around your customer, work or search workflow.
You have a result. Now what?
Let’s turn it into a useful improvement.
Help me with the next stepBring us your result. We’ll help you decide what to fix first and make the changes.