Practical guide · ARC Scan
Crawler access is a starting point for agent readiness
Keep access, interpretation and observed actions separate when deciding what a website check proves.
Hoshiro Labs · Published 19 October 2026
Answer the access question first
A public page may be available in a human browser while a crawler receives a challenge, redirect or blocked response. Record the URL, response and collection conditions before interpreting the content. If collection is blocked, the unseen content remains unverified.
Do not use robots.txt as access control
Google explains robots.txt as a way to manage crawler access. It is not a mechanism for keeping a page out of every search result or protecting private material. Keep customer data behind real authentication and authorization. Decide crawler policy deliberately rather than opening every route to obtain a better scan result.
Separate three decisions
First, can the public content be collected? Second, does it explain the business and offer coherently? Third, was a permitted action actually observed? Passing the first condition does not establish the others. A descriptor or endpoint being present is weaker evidence than a successful, authorized action.
Retest the same boundary
After changing a crawler rule or challenge configuration, repeat the affected request and inspect the actual response. For an action such as adding a variant to a cart, define the intended outcome and a safe stop point before observation. Keep authorization and payment submission outside a public read-only scan.
Read the result in context
The free ARC Scan provides an initial public-site assessment. A detailed report adds evidence and assessment scope. Neither guarantees that an external AI system will cite, recommend or transact with your business. Use the findings to choose a concrete correction, then verify that correction.