Skip to content

Atlastic front page

Kestrel, the web crawler Atlastic uses

If you see Kestrel in your server logs, this page explains who is crawling your site, why, and how to limit it or get in touch. When it crawls for Atlastic, its user agent is:

Kestrel/1.0 (Atlastic; +https://atlastic.ai/kestrel)

Who runs Kestrel

Kestrel is the web crawler used by Atlastic, a stock-market-anchored media analytics platform. It collects publicly available news and financial-media articles about listed companies for Atlastic's analytics.

What it fetches

  • Public news pages
  • The site's listings of new articles: RSS and Atom feeds, sitemaps, and front and section pages
  • robots.txt

Kestrel does not log in, submit forms, run JavaScript or keep cookies.

How it behaves

  • It obeys the rules in robots.txt for the user agent token Kestrel, and honours Crawl-delay.
  • By default it makes at most one request to each server every 5.5 seconds.
  • It backs off when a site answers with 429 or 403, or refuses access to its robots.txt.
  • It uses conditional requests (ETag and If-Modified-Since), so it does not fetch unchanged pages again.

How to limit or block it

Add rules for Kestrel to your robots.txt. This example keeps it out of /private/ and asks for at least 10 seconds between requests:

User-agent: Kestrel
Disallow: /private/
Crawl-delay: 10

To block Kestrel from the whole site:

User-agent: Kestrel
Disallow: /

Where requests come from

Kestrel sends its requests from these IP addresses:

  • 35.228.87.242
  • 35.228.235.236

The addresses can change. This page lists the current ones.

Contact

If you have questions or complaints about Kestrel, or want to make an arrangement for your site, email [email protected].