Lead Discovery and Enrichment

Building a prospect list was manual, slow, and produced data nobody could vouch for.

Discovery was a person with a browser. Searching, opening pages, copying addresses into a spreadsheet, checking whether the domain even accepts mail. Hours of work producing a list whose provenance existed only in that person's memory.

Quality problems surfaced after sending, which is the worst possible time. Addresses that do not resolve, role addresses that should have been filtered, records already on the suppression list. Each one costs sender reputation, and you find out afterwards.

Enrichment had a dangerous shape. The natural implementation — look up enrichment data while generating messages — makes an external data source a dependency of sending. When that source slows down, campaigns stop. The failure is remote, external, and takes the revenue path with it.

Paid search APIs made the cost scale with curiosity. Every exploratory query cost money, which meant people ran fewer of them and worked from worse data.

Nothing was reproducible. Re-running the same discovery produced a different result set, because query generation was ad hoc. There was no way to audit where a list came from.

Goals

1. Discovery runs as a repeatable job with recorded inputs and generated queries.
2. Results are filtered and reviewed before anything can become a sending list.
3. Retrying a job reproduces the same query set rather than a new random one.
4. Enrichment never sits in the sending path, under any configuration.
5. Discovery cost is not proportional to how many questions someone asks.
6. Repeated work is cached rather than repaid.

Non-goals

• No automatic import. Explicitly out of scope. A human reviews and creates the list.
• Not a contact database product. It supports the platform's own list building.
• No scraping behind authentication. Publicly published business contact information only.
• No enrichment guarantee. Records may have no enrichment data, and every consumer must handle that.

• Discovery — established that the expensive part was not searching but reviewing, which is why the review screen got the design attention rather than the crawler.
• Requirements — wrote the enrichment service specification, including its hard constraints: background-only, additive schema, no changes to the sending core, and tolerance of missing data everywhere.
• Safety design — made the no-auto-import rule a structural property rather than a policy: the service has no path into the sending pipeline at all.
• Pipeline architecture — the database-driven queue, its claim semantics, idempotent expansion and stale-lock reclamation.
• Cost architecture — moved discovery from a paid search API to self-hosted metasearch with an open-source fallback, keeping the paid backend selectable for comparison.
• Reproducibility — required query generation to be seeded by job identity, so a retry produces the same queries.
• Test planning and UAT — with the lead operations staff who use the review screen daily.
• Operational tuning — bandwidth budgeting across competing crawl consumers.

Discovery. An operator creates a job from keywords, optional manual search queries, seed domains, or direct URLs. For keywords, the service expands contact-focused query templates, optionally scoping some of the budget to specific public platforms. It runs those searches, fetches page content, extracts candidate addresses, and validates and classifies each one — does the domain accept mail, is this a role address, is it already suppressed.

The operator then reviews. The results screen filters by mail-exchange validity, confidence, role type and suppression status. From there they either create a list or export the data. There is no third option, and no automatic path.

Enrichment. A separate background service that owns three tables and nothing else. Jobs expand into a queue with set-based SQL and conflict handling that makes re-running a job a no-op rather than a duplicate. Workers claim rows with skip-locked semantics, heartbeat while working, and a reaper reclaims locks left behind by a worker that died. The service populates segmentation data and is architecturally incapable of delaying a send.

The first queue implementation could strand work. A worker that died while holding claimed rows left them marked as processing forever. Nothing errored — the job simply never completed, and its counters stopped moving. Fixed with heartbeats and a reaper that reclaims locks older than a threshold, but the lesson generalizes: any claim mechanism needs an answer for the claimant dying, and that answer has to be designed rather than assumed.

Idempotency was harder than expected. Re-running a job initially duplicated queue rows. The correct fix was conflict handling scoped to rows already queued or processing — which required being precise about what "already in flight" means, rather than deduplicating on identity alone.

Bandwidth is a shared, finite budget. Discovery competes with other crawling consumers for a metered monthly allowance. Raising concurrency to make discovery faster silently starves the others. This became an explicit split with a safe burn rate rather than a per-service tuning decision.

Result quality varies more without a commercial API. Self-hosted metasearch sometimes returns little for a query the paid backend handled well. The fallback chain covers most of it, and the honest position is that we traded some consistency for the ability to explore freely — which was the right trade for how the tool is actually used.

Review is still the bottleneck, by design. Discovery can produce candidates faster than people can review them. Every proposal to relax that has been a proposal to auto-import under a different name.

• Discovery job creation

  Keywords, manual queries, seed domains and platform scoping options.

• Results review screen

  Filtering by validity, confidence, role type and suppression before list creation.

• Enrichment queue state

  Job progress with claimed, done, error and dead row counts.

• Service specification excerpt

  The hard constraints section, including background-only and tolerated absence.

What worked. Making the safety property structural. "No auto-import" as a policy is a sentence someone can decide to make an exception to. "There is no code path from discovery into sending" is not.

What failed. My first queue design had no answer for a worker dying mid-claim. It failed silently — the job simply never finished. Any claim mechanism needs its failure case designed, not assumed.

What changed. The review screen became the product. I started out thinking discovery was the hard part; the operators made it clear within a week that finding candidates was easy and deciding about them was the work.

What I would do differently. I would have made query generation seeded from the start. Retrofitting reproducibility meant reworking the expander and accepting that earlier jobs are not auditable.

What this demonstrates about AI-product leadership. This is a system where the obvious AI application — "use a model to find and qualify leads" — is worse than the deterministic implementation on accuracy, cost and speed. The genuinely valuable judgment was recognizing that, keeping the model on the assistive side, and spending the design effort on the human review step that actually gates quality.

• Scheduled refresh of stale enrichment records. The job type is specified but not implemented; records currently age without a refresh cycle.
• Segment-scoped enrichment jobs, so a job can target a segment rather than a whole list or an explicit record set.
• Review-assist ranking. Ordering the review queue so the most likely decisions surface first — assisting the reviewer, explicitly not replacing the review gate.