October 16, 2014 by Roman Ožana

Our Crawler Gets a New Name – Testomatobot

We’ve been hard at work improving our crawler this month. In addition to this work, we made another small but very important change.

Our crawler (i.e. user agent) is now called Testomatobot!

Easier to Locate in Logs

Testomatobot is the toolset we use to download page content, send web forms, and access other resources according to the tests you’ve configured for your projects.

We know that a lot of you need to be able to locate our crawler easily in your logs, so we decided to change our name to make the whole process easier.

Instead of the old minicrawler showing up in your server logs, the updated user agent HTTP header field will be the following:

Testomatobot/1.0 (%os%; +http://www.testomato.com/testomatobot) minicrawler/%version%

If for any reason you don’t want Testomatobot accessing your server (i.e. due to overloading), please contact our team at support@testomato.com.

Minicrawler v.s. Testomatobot

For those who don’t know, Minicrawler is our HTTP client and Testomatobot is the toolset used for scheduling tests, running user-defined tests, and processing errors.

Here’s how it works:

  • User-defined tests are sent from the Testomato web app to Testomatobot.
  • Testomatobot sends request to Minicrawler, which downloads HTML content from your page.
  • HTML content is sent back to the Minicrawler and then, processed by Testomatobot.
  • Final results are then sent and saved on Testomato.
a master plan (1)

Minicrawler is open-sourced under the AGPL License, and we encourage you to try it out for yourself! We love hearing your comments and feedback about ways we can make our crawler better.

Please share our feedback with us: support@testomato.com or you can join us on Facebook or Twitter to tell us more.

Is your website available? Testomato can check your website availability every 15 seconds from 10 different locations around the world and will send you an alert if the site is unavailable.

Keep track of the important parts of your website with simple checks that can run every minute. They can check plain words, HTML code, HTTP headers, redirects … and much more. Never miss anything.

Websites break, it happens all time. You'll be the first to know. Testomato has an extensive database of and will let you know if it finds any error on monitored url.