stancr-bot

Stancr shows how outlets across the political spectrum cover the same story. To do that, our crawler reads the RSS and Atom feeds of the outlets we list, and the article pages those feeds link to. Every story we show links back to the original reporting.

What it does

• It identifies itself as stancr-bot/0.1 (+https://stancr.com/bot).
• It checks feeds about every 15 minutes, with conditional requests (ETag and Last-Modified), so an unchanged feed costs you a 304.
• It fetches only the article pages your feed links to, and keeps few requests in flight per site, spaced apart.
• It follows your robots.txt (checked again every day) and never fetches a disallowed path.

Blocking it

Add this to your robots.txt and the crawler stops fetching your site at its next check:

User-agent: stancr-bot
Disallow: /