SwimBlocksBot

The web crawler for SwimBlocks, open-source tools for the competitive swimming community.

Why it visits

It reads public swimming information so that SwimBlocks tools can gather it in one place: meet schedules and meet packages from clubs, governing bodies and swimming.ca, and which team-management and registration platforms club websites use. It reads public pages only, and does not sign in or submit forms. Some of what it reads names people: a meet package, for example, can list the meet's officials and their email addresses. That is information the site has already published.

How it behaves

• It identifies itself with the User-Agent SwimBlocksBot/1.0 (+https://bot.lightning-snow.com/).
• It obeys robots.txt rules for SwimBlocksBot, rereading the file at least once a day. If robots.txt cannot be read because of a server error, it does not crawl the site.
• It sends one request at a time to a site, at least two seconds apart, or further apart if your robots.txt sets a longer Crawl-delay.
• When it has fetched a page before, it asks again conditionally (If-None-Match, If-Modified-Since), so an unchanged page costs your server a 304.
• It signs every request with Web Bot Auth, so a request that claims to be SwimBlocksBot can be verified. Its public keys are in its key directory, and its requests carry a Signature-Agent header naming https://bot.lightning-snow.com.

How to block it

Add this to your site's robots.txt:

User-agent: SwimBlocksBot
Disallow: /

It stops within a day, when it next reads the file. To block only part of your site, list those paths instead of /.

Contact

Questions or problems: bot@lightning-snow.com. The source code is at github.com/swimblocks/fetch-service.