AdFetch — Shared Fetch, Scrape and Search Infrastructure

Any product that works with the public web hits the same three problems, in the same order, regardless of what it's actually trying to accomplish.

A browser fleet is not optional. Modern pages render with JavaScript. Fetching raw HTML gets you an empty shell for a growing share of the sites worth fetching, so someone has to run headless browsers at scale.

Sites do not want to be fetched. Rate limits, fingerprinting and challenge pages are the default posture of the modern web, not the exception. Anti-bot handling is its own ongoing maintenance burden, unrelated to whatever the fetched page is actually going to be used for.

Egress has to come from somewhere real. Residential IP space with a metered, shared budget is what keeps a fetcher from looking like a data center — and somebody has to own that budget, the byte accounting, and the hard stop when it runs out.

None of this is specific to any one product. An ad-intelligence pipeline and a contact-discovery pipeline need identical answers to all three problems. Building the answer twice means maintaining two browser fleets, two anti-bot postures and two proxy relationships for one underlying capability.

Before this service existed, the answer was a vendor. A commercial scraping API was paid per request for page fetching — Lead Discovery is the pipeline with the clearest paper trail on this, because its own decision record still states the choice plainly: reimplement the paid API's exact request and response shapes in a self-hosted service, rather than keep paying for it or wrap a different vendor, specifically so that not one line of calling code had to change on the day of the switch.

The same reasoning generalizes past that one pipeline. Every additional product that wants to fetch a page either pays the same per-request vendor bill again, or quietly builds its own second browser fleet. Both are the same mistake at different price points.

Goals

1. One HTTP surface for fetching, rendering and searching the public web, reachable by any internal service.
2. Match the request and response shapes of the commercial API it replaced — specifically the Firecrawl shape — so a consumer's existing integration code needs no rewrite to switch.
3. Meter proxy egress centrally, with per-consumer budgets and hard stops, so the cost and the byte accounting live in one place rather than in every caller.
4. Stay honest about what is actually self-hosted: scrape and fetch run entirely on infrastructure this service owns; search and map proxy to the real, third-party Firecrawl API rather than reimplementing web search.

Non-goals

• No domain knowledge of what is being fetched. This service does not know what an ad is, does not know what a lead is, does not deduplicate, classify, or resolve identity. That logic belongs entirely to the consumer that requested the fetch — never to this one. A fetch service that starts making decisions about the content it fetches is a fetch service that has started coupling itself to one caller's domain, at the expense of every other caller.
• No consumer-specific code paths. AdSpy and Lead Discovery get the same API surface everyone else would get. There is no if adspy branch in this service, and there should never be one.
• Not a search engine. Search and map are proxied to a real search provider rather than built from scratch — reimplementing web search was never the point.

I own this service on my own: the API contract, the architecture decisions, the call to proxy search and map to real upstream Firecrawl rather than reimplement search, and production operations.

The two decisions I would point to as mine:

• Match an existing API shape instead of inventing one. Copying the Firecrawl request and response format for the scrape/search/map trio meant every consumer's integration code already worked against this service before it existed — the switch was a URL change, not a rewrite.
• Enforce the non-goals as a design constraint, not a policy note. The service has no schema, no table and no code path that knows what an ad or a lead is. That absence is deliberate: it's what makes a second, third or tenth consumer safe to add without ever touching this codebase's domain logic.

The result is an 8-endpoint surface across two shapes: a native job-based API for fetch, and a Firecrawl-compatible trio for scrape (served locally), search and map (proxied to the real Firecrawl). A consumer picks whichever shape matches its existing integration code.

Both consumers are real, independent, and unaware of each other. Neither shares a client library with the other — each was integrated separately, at a different time, for a different reason.

The absence of a shared client library between the two is a real gap, not a feature — see What's next.

There is no AI anywhere in this service's own request path. Every fetch, every job-queue claim, every proxy-budget check is deterministic code, on purpose — this is infrastructure two other products depend on being boring and predictable.

The one boundary worth naming honestly: what happens inside the real upstream Firecrawl API, on search and map requests, is outside this service's control and outside its visibility. This service proxies those two calls faithfully; it does not audit or constrain whatever Firecrawl itself does with them.

There is no screenshot of this service, and there will not be one. AdFetch is a headless API with no user interface — nothing renders that a browser could capture. The closest thing to a screenshot is the architecture diagram above; the card illustration on the case-studies index is the same kind of diagram, labeled on its face as an illustration rather than a capture, for the same reason.

What can be verified instead is the API surface itself:

What this service produces is visible secondhand, in its consumers' own evidence: the ad and landing-page captures in AdSpy's evidence gallery, and the captured pages in Lead Discovery's own case study. Every image in both of those galleries passed through this service first — this one just fetched what's on screen there.

There is no systemd unit for this service; it runs under Docker Compose. That is a real operational difference from every other flagship system in this portfolio, and it's named honestly in What's next rather than smoothed over.

• One fetch capability, two independent products. Neither AdSpy nor Lead Discovery had to build or maintain its own browser fleet, anti-bot posture, or proxy relationship. Both get it from one place.
• Zero rewrite on adoption. Matching the Firecrawl request and response shape meant a consumer's existing integration code worked against this service before the switch, not after a rewrite.
• The proxy bill lives in one place. Metered, budgeted egress with hard stops replaced what would otherwise be an unmanaged cost spread across every consumer that happened to need a browser.
• The boundary held. After two independent consumers and roughly a year of use, this service still has no code path that knows what an ad or a lead is. That absence is the actual product.

• Naming can silently smuggle ownership across a boundary. This portfolio's own case study on this exact system briefly attributed ad-specific ingestion logic to this service, because a package comment written from inside one consumer's codebase described this service in that consumer's own vocabulary. This service's own repository contains a binary literally named ad-ingestor as one of its nine components — a real, separate naming collision, unrelated to any consumer's domain logic, and exactly the kind of trap that makes "just read one file" an unsafe way to establish what a shared service actually does.
• A generic service earns trust by refusing scope, not by adding it. The hardest feature requests to say no to were the ones from a single consumer that would have made this service slightly better for them and permanently coupled for everyone else. Every one of those requests belonged in the consumer instead.
• Match the contract, not the implementation. Copying Firecrawl's request and response shapes — rather than designing a cleaner API from scratch — is what made the swap invisible to every caller. A better API nobody has to adopt is worth less than a compatible one everyone already has.
• Shared infrastructure needs its own case study. Describing this service only from inside a consumer's documentation is how its identity gets lost in the first place — which is the entire reason this page exists.

• Replace the four-plus-one bespoke per-consumer clients with one shared client library, so the next consumer doesn't write its own from scratch.
• Give this service a proper systemd unit instead of relying solely on Docker Compose for process supervision — the one operational gap that would matter most if this service went down.
• Onboard a third consumer, if one appears, as the real test of whether the non-goals above actually hold under pressure.