[{"data":1,"prerenderedAt":41},["ShallowReactive",2],{"page:\u002Fkestrel\u002F":3,"built-pages":20,"post:\u002Fkestrel\u002F":40},{"page":4,"path":5,"title":6,"description":7,"image":-1,"noindex":8,"demoTab":8,"sections":9,"posts":19},"kestrel","\u002Fkestrel\u002F","Kestrel, the web crawler Atlastic uses","Kestrel is the web crawler Atlastic uses for public news. What it fetches, how it behaves, how to limit or block it in robots.txt, its IP addresses and contact.",false,[10,16],{"type":11,"height":12,"heading":6,"background":13},"hero","medium",{"image":14,"overlay":15},"\u002Fuploads\u002F2023\u002F02\u002FSingle-network-v2-pipe-back-4.jpg","fade-soft",{"type":17,"body":18},"prose","If you see Kestrel in your server logs, this page explains who is crawling your site, why, and how to limit it or get in touch. When it crawls for Atlastic, its user agent is:\n\n```\nKestrel\u002F1.0 (Atlastic; +https:\u002F\u002Fatlastic.ai\u002Fkestrel)\n```\n\n### Who runs Kestrel\n\nKestrel is the web crawler used by Atlastic, a stock-market-anchored media analytics platform. It collects publicly available news and financial-media articles about listed companies for Atlastic's analytics.\n\n### What it fetches\n\n- Public news pages\n- The site's listings of new articles: RSS and Atom feeds, sitemaps, and front and section pages\n- `robots.txt`\n\nKestrel does not log in, submit forms, run JavaScript or keep cookies.\n\n### How it behaves\n\n- It obeys the rules in `robots.txt` for the user agent token `Kestrel`, and honours `Crawl-delay`.\n- By default it makes at most one request to each server every 5.5 seconds.\n- It backs off when a site answers with `429` or `403`, or refuses access to its `robots.txt`.\n- It uses conditional requests (`ETag` and `If-Modified-Since`), so it does not fetch unchanged pages again.\n\n### How to limit or block it\n\nAdd rules for `Kestrel` to your `robots.txt`. This example keeps it out of `\u002Fprivate\u002F` and asks for at least 10 seconds between requests:\n\n```\nUser-agent: Kestrel\nDisallow: \u002Fprivate\u002F\nCrawl-delay: 10\n```\n\nTo block Kestrel from the whole site:\n\n```\nUser-agent: Kestrel\nDisallow: \u002F\n```\n\n### Where requests come from\n\nKestrel sends its requests from these IP addresses:\n\n- `35.228.87.242`\n- `35.228.235.236`\n\nThe addresses can change. This page lists the current ones.\n\n### Contact\n\nIf you have questions or complaints about Kestrel, or want to make an arrangement for your site, email [cc@hypefactors.com](mailto:cc@hypefactors.com).\n",[],[21,22,23,24,25,26,27,28,4,29,30,31,32,33,34,35,36,37,38,39],"_home","about","asset-managers","contact","cookie-declaration","crypto","demo","hedge-funds","methodology","news-updates","perceived-esg","privacy-policy","private-capital","quantitative-trading","risk-management","terms-and-conditions","the-atlastic-philosophy","trading-signals","trust-value-tracking",null,1793105189345]