Web Crawlers

We have a bunch of sophisticated crawlers designed to scrape websites. Crawlers are configured via source records.

The heavy-lifting is done right out of hardcore testing applications: playwright. We have some tricks involving running headless (but actually not) in order to mitigate the arms race against bots and crawlers that our target sources array against us.

Crawling isn’t limited to the web - we can also crawl local documents, sending them to AI for ingesting into the suite too.

class lbn.aidoc.crawl.Crawler

Bases: object

The Governor: Manages source-specific adapters and the model-bound lifecycle.

__init__(source: Source)

commit(job_id: int)

Delegates the persistent strike to the adapter.

fetch(url: str, gatherer: str | None = None) → Tuple[str, Dict[str, Any]]

Stage 1: The Slurp. Delegates to the resolved adapter’s workflow.

class lbn.aidoc.crawl.core.Crawler

Bases: object

The Governor: Manages source-specific adapters and the model-bound lifecycle.

__init__(source: Source)

commit(job_id: int)

Delegates the persistent strike to the adapter.

fetch(url: str, gatherer: str | None = None) → Tuple[str, Dict[str, Any]]

Stage 1: The Slurp. Delegates to the resolved adapter’s workflow.

lbn.aidoc.crawl.fetchers.content_driver()

lbn.aidoc.crawl.gatherers.content_driver()

returns our configured content/data collectors

lbn.aidoc.crawl.gatherers.file_driver(mime_type) → Tuple[Dict[str, Any], Dict[str, Any]]

returns appropriate content, and metadata gatherers for the mime type

lbn.aidoc.crawl.gatherers.metadata_driver()

returns our configured meta data collectors

class lbn.aidoc.crawl.adapters.BaseAdapter

Bases: ABC

manages/degates to fetch and gather content, and return job, company, role dicts via it’s map_triad

__init__(source: Source, fetcher: Fetcher, debug: bool = True)

commit(job_id: int)

Hardware Strike: Flushes the internal _artifacts vault to a permanent job-specific directory. This is the only place filesystem IO happens.

execute_workflow(url: str, gatherer: str | None = None) → Tuple[str, Dict]

resolve_gatherer(nominee: str | None = None) → Tuple[str, str]

The Arbiter: Nominates a winner from the vault. Exact name or die if nominee is provided. Otherwise, the longest valid payload in the vault takes the crown.

class lbn.aidoc.crawl.adapters.BraveAdapter

Bases: BaseAdapter

calls brave.com and passes through query results without further adornment/structuring

__init__(source: Source, debug: bool = True)

class lbn.aidoc.crawl.adapters.JSONAdapter

Bases: BaseAdapter

__init__(source: Source, debug: bool = True)

class lbn.aidoc.crawl.adapters.LangChainHttpAdapter

Bases: BaseAdapter

__init__(source: Source, debug: bool = True)

class lbn.aidoc.crawl.adapters.LangChainSimpleAdapter

Bases: BaseAdapter

Uses LangChain fetchers - but without source

__init__(debug: bool = True)

class lbn.aidoc.crawl.adapters.LinkedInAdapter

Bases: BaseAdapter

use LinkedInAPI to crawl LinkedIn

__init__(source: Source, debug: bool = True)

class lbn.aidoc.crawl.adapters.LocalFileAdapter

Bases: BaseAdapter

Surgically inspects the file suffix on initialization to dynamically plug only the required content and metadata drivers—or fails fast with a ValueError.

__init__(source: Source)

execute_workflow(url: str, gatherer: str | None = None) → Tuple[str, Dict]

The Suffix Intercept Override: Forcefully swaps the web gatherer nominee for the explicit local driver key freshly mapped during the setup phase.

class lbn.aidoc.crawl.adapters.PlaywrightAdapter

Bases: BaseAdapter

uses underlying chrome/firefox to get past javascript, cloudflare turnstiles to download content

__init__(source: Source, debug: bool = True)

class lbn.aidoc.crawl.adapters.StaticAdapter

Bases: BaseAdapter

dumb adapter; uses requests; fails if javascript et al needs to be enabled

__init__(source: Source, debug: bool = True)

lbn.aidoc.crawl.adapters.driver(source)