About FaiBot, our page fetcher

Who is fetching

Requests that identify themselves with this User-Agent come from FaiV, the platform Fai Ventures runs for university venture programmes:

FaiBot/1.0 (+https://faiventures.com/bot)

Why

Our assessments help a university's venture programme review an early-stage company spun out of its research. While writing one, the assessment reads public pages about the venture, its market and its competitors, such as press releases, news articles and company reports, so that what it says rests on what those pages actually say. It reads the page, summarises it and cites it.

How often

Rarely. Pages are fetched on demand, one at a time, only when an assessment is being written, and only pages that came up in its own web searches or were linked from a page or document it has already read. We don't crawl or index sites. A summary of a page is kept for up to 30 days, so the same page usually isn't fetched again in that time.

What we don't do

• We don't get round CAPTCHAs, bot challenges, logins or paywalls, and we never pretend to be a person. When Cloudflare challenges a request, we ask once more with a Web Bot Auth signature proving the request is ours, and your settings decide. A page that still refuses us is left alone.
• We don't run JavaScript, submit forms or sign in.
• We only fetch public addresses over http and https.
• We don't fetch pages your robots.txt disallows. We read it before fetching and follow the rules for FaiBot, or those for every crawler (*) when none name us. To keep us off your site altogether, add User-agent: FaiBot and Disallow: /, or tell us and we'll stop.
• We don't fetch pages whose content signals say ai-input=no or use=immediate: we pass pages to an AI model and keep a summary and short quotes, with a link back, in the assessment. We never train models on your content.
• We don't hurry you: requests to one site are at least a quarter of a second apart, or further apart if your robots.txt sets a Crawl-delay.

Get in touch

If our fetcher is causing you a problem, or you'd like us not to fetch your site, email info@askfai.co and include the address of the page. We'll reply personally.