Our Crawler Gets a New Name – Testomatobot
We’ve been hard at work improving our crawler this month. In addition to this work, we made another small but very important change.
Our crawler (i.e. user agent) is now called Testomatobot!
Easier to Locate in Logs
Testomatobot is the toolset we use to download page content, send web forms, and access other resources according to the tests you’ve configured for your projects.
We know that a lot of you need to be able to locate our crawler easily in your logs, so we decided to change our name to make the whole process easier.
Instead of the old minicrawler showing up in your server logs, the updated user agent HTTP header field will be the following:
Testomatobot/1.0 (%os%; +http://www.testomato.com/testomatobot) minicrawler/%version%
If for any reason you don’t want Testomatobot accessing your server (i.e. due to overloading), please contact our team at support@testomato.com.
Minicrawler v.s. Testomatobot
For those who don’t know, Minicrawler is our HTTP client and Testomatobot is the toolset used for scheduling tests, running user-defined tests, and processing errors.
Here’s how it works:
- User-defined tests are sent from the Testomato web app to Testomatobot.
- Testomatobot sends request to Minicrawler, which downloads HTML content from your page.
- HTML content is sent back to the Minicrawler and then, processed by Testomatobot.
- Final results are then sent and saved on Testomato.
Minicrawler is open-sourced under the AGPL License, and we encourage you to try it out for yourself! We love hearing your comments and feedback about ways we can make our crawler better.
Please share our feedback with us: support@testomato.com or you can join us on Facebook or Twitter to tell us more.
