Web Crawlers

We have a bunch of sophisticated crawlers designed to scrape websites. Crawlers are configured via source records.

The heavy-lifting is done right out of hardcore testing applications: playwright. We have some tricks involving running headless (but actually not) in order to mitigate the arms race against bots and crawlers that our target sources array against us.

Crawling isn’t limited to the web - we can also crawl local documents, sending them to AI for ingesting into the suite too.