Lead Discovery and Enrichment Pipeline
Find publicly published business contacts through reproducible searches, validate and classify them, then require a human to review before anything can become a sending list.
Runtime state: Both services run. Discovery is behind a feature flag that defaults off; enrichment runs continuously and is architecturally barred from the send path.
- Business objective
- Replace manual prospect research with a repeatable, auditable pipeline — without ever creating an automatic path from a search result into a campaign.
- Trigger
- An operator creates a discovery job, or an enrichment job is queued for a list.
- Inputs
- Keywords, manual search queries, seed domains or direct URLs
- Per-job search budget and platform scoping options
- Existing suppression list, for cross-checking
- Available crawl bandwidth from the shared monthly budget
- Outputs
- A filtered, reviewable candidate set with validity and confidence signals
- The generated queries, stored on the job for audit
- A sending list, only when an operator explicitly creates one
- Enrichment records supporting segmentation
The workflow
Discovery — job to reviewed list
The review step is not a formality. It is the only route from a candidate to a sending list.
- Human action
Operator defines the job
Keywords, manual queries, seed domains, direct URLs, optional platform scoping.
- Automated software
Query expansion
Contact-focused templates expanded and shuffled, seeded by job identity so a retry reproduces the same query set.
- Data store
Cache check
Decision point: Seen this query recently? Serve from cache rather than repaying the cost.
- External service
Web search
Decision point: Self-hosted metasearch first; an open-source harvester when it returns nothing usable.
- Automated software
Page fetch
Content retrieved through a metered proxy pool, drawing on a shared monthly bandwidth budget split across competing consumers.
- Automated software
Extract, validate, classify
Address extraction, mail-exchange validation, role-address detection and suppression cross-check.
- Approval gate
Operator review
Decision point: Filter by validity, confidence, role type and suppression. Create a list, or export. There is no automatic import — the service has no code path into the sending pipeline.
- Human action
- Automated software
- Data store
- External service
- Approval gate
Enrichment — background only
Structurally incapable of delaying a send.
- Automated software
Job expansion into the queue
Set-based SQL with conflict handling, so re-running a job is a no-op rather than a duplicate.
- Automated software
Worker claims a batch
Skip-locked claiming so parallel workers never contend for the same rows.
- External service
External enrichment sources
Decision point: On failure, mark for retry. The failure never propagates toward sending.
- Data store
Record written
Provider, geography, risk signals and extensible attributes.
- Monitoring / feedback
Heartbeat and stale-lock reaper
Locks held by a dead worker are reclaimed on a fixed cycle.
- Automated software
Available for segmentation
Consumers must tolerate a missing record. Absence is normal.
- Automated software
- Data store
- External service
- Monitoring / feedback
Decision points
- Search backend — metasearch, with a harvester fallback when results are unusable.
- Cache hit or fresh search — bounded lifetime per query.
- Per-address classification — valid, role address, already suppressed, low confidence.
- The review gate — create a list, export, or discard. Operator only.
- Enrichment retry or dead-letter — bounded attempts, then parked.
Responsibilities
AI / LLM
- Suggesting keyword and niche themes for an operator to consider before a job is created.
- Summarizing a result set so a reviewer knows what they are about to look at.
Deterministic
- Query generation — template-based and seeded, so it is reproducible.
- Address extraction from page content.
- Mail-exchange validation and role classification.
- Suppression cross-checking.
- Queue claiming, idempotent expansion and stale-lock reclamation.
- Bandwidth budgeting and crawl throttling.
- Every count and confidence value on the review screen.
Human approval points
- Job definition — what to search for, and how much budget it gets.
- The review gate — the mandatory step. No confidence threshold auto-imports.
- List creation — explicit, by a person, every time.
Failure handling
| Failure | Response |
|---|---|
| Search backend returns nothing usable | Fall back to the harvester; if that fails, the job completes with fewer candidates rather than erroring. |
| Page fetch fails | URL skipped and recorded; the job continues. |
| Bandwidth budget approaching its ceiling | Crawl concurrency throttled to a safe burn rate rather than starving other consumers. |
| Enrichment source unavailable | Row marked for retry. Sending is unaffected by construction. |
| Worker dies holding claimed rows | Reaper reclaims locks older than the threshold. |
| Job re-run | Conflict handling makes expansion idempotent; no duplicate queue rows. |
Monitoring
Job progress and completion rate, candidate yield per query, validation pass rate, queue depth with error and dead-row counts, stale-lock reclamation rate, and bandwidth burn against the monthly split.
Business value
Turns hours of manual research into minutes of review, while making list provenance auditable — which matters as much for compliance as it does for throughput.
Related projects
- Lead Discovery and EnrichmentStatus: Shipped
A discovery and enrichment pipeline that finds publicly published business contacts and enriches records for segmentation — with a mandatory human review step before anything enters a sending list.
- MailerClub — Email Marketing PlatformStatus: Production
A Go platform that takes a campaign from recipient list to delivered mail — unsubscribes and bounces enforced on every send, automatic delivery protection, and full event reporting.