Crawler

The 5od-profiler crawler

5od-profiler is the user agent 5od uses to read the pages of sites registered with 5od, so it can work out what each site and page is about.

It reads robots.txt first and follows it. It obeys rules for 5od-profiler, or the rules for all robots (*) when there are none for it, and it honours Crawl-delay.
It fetches one page at a time and waits between requests.
It reads the home page, a small sample of addresses from the sitemap, and pages where a 5od ad slot has appeared.
It doesn’t submit forms, log in, or follow links off the site.

To block it, add to robots.txt:

User-agent: 5od-profiler
Disallow: /