Skip to content

Discovery & Enrichment

Every connector invocation runs two sequential phases:

  1. Discovery — scrape fresh listings from a third-party site and submit them to Mill.
  2. Enrichment — drain one batch of re-scrape requests from a Kafka topic and push updates back to Mill.

This dual-phase design means a single CronJob keeps the corpus both fresh and accurate without separate deployment concerns.

Discovery works the same way it always has. The connector scrapes paginated search results, extracts individual listing URLs, visits each one, maps the raw HTML/JSON into common.Property, and submits the batch to Mill via HTTP or Kafka.

All connectors implement the Connector interface:

type Connector interface {
GetName() string
GetSource() string
ScrapeProperties(opts ConnectorOptions) ([]Property, error)
GetStats() ConnectorStats
SetRateLimit(delay time.Duration)
HealthCheck() error
}

After discovery, the EnrichmentDispatcher reads enrichment requests from the property-enrichment Kafka topic and routes each message to the correct enricher by matching the message’s Source field to the connector that owns that domain. It runs one lane per country in parallel, so a backlog in one country never delays another.

Connectors opt into enrichment by implementing the Enricher interface:

type Enricher interface {
GetCountry() string
EnrichProperty(ctx context.Context, req EnrichmentRequest) (*Property, error)
}
type EnrichmentRequest struct {
PropertyID string
SourceURL string
Source string
Country string
MissingFields []string
Address string
City string
}

GetCountry() returns an ISO-3166-1 alpha-2 code (e.g. "NZ", "AU", "MT"). The dispatcher uses this to build a per-country index of AddressSearcher connectors for gap-fill.

EnrichProperty() receives the original listing URL and a list of fields that are known to be missing or stale. The implementation re-scrapes that URL and returns a *Property with fresh data. Returning (nil, nil) is safe — it signals that the property cannot be found or is not handled by this connector.

Connectors can optionally implement the AddressSearcher interface to participate in gap-fill enrichment:

type AddressSearcher interface {
Enricher
SearchByAddress(ctx context.Context, address, city, country string) (*Property, error)
}

When the primary enricher returns a result but fields like images, price, or room counts are still missing, the dispatcher picks up to 2 AddressSearcher connectors for the same country and asks them to look up the property by address. Fields returned by the gap-fill source are merged into the primary result before submission. This enables cross-source data filling — for example, an OSM property with no images can be enriched with photos from a real-estate listing site in the same country.

Searchers can return the nearest listing rather than the property itself, so a gap-fill result is merged only when its address matches the request (same street number and street, and the same town or nearby coordinates). Results for other addresses are dropped and counted as gap_fill_rejected in the batch stats. When the address cannot be decoded, nothing is merged.

A ListingLinker finds the property’s pages on its site so the property page can link to them. For example, raywhite-nz asks Ray White’s listing search API (POST https://www.raywhite.co.nz/api/rw/listings) for the 200 current, leased and sold listings nearest the property within 1 km, keeps those whose structured street number, unit and street match the property, and links each one’s page (/{region}/{district}/{suburb}/{source id}). Linkers include past listings as well as current ones, because expired listing pages keep their photos and details: harcourts-nz searches its buy, rent and sold map pins, but the sold pins only hold the 200 most recent sales around the property (a few months in Auckland), so older Harcourts sales are not linked. A linker that also knows about the property implements ListingDataLinker:

type ListingDataLinker interface {
ListingLinker
FindListingData(ctx context.Context, req EnrichmentRequest, match *AddressMatcher) ([]SourceLink, *Property, error)
}

The dispatcher merges the returned data after gap-fill. Rooms, price and images are only filled when Mill reported them missing, because an update’s images replace the property’s images in Mill. Mill copies them into its image bucket (images.gcs_bucket, at properties/<id>/img_NN.jpg), keeping the source URL for any image it cannot fetch. Car spaces, floor area, section area and the council valuation are filled when the update lacks them. homes.co.nz implements it: its property card supplies rooms, car spaces, floor and land area, listing photos and the council rating valuation (capital, land and improvements values, revision date, legal description and title).

A CouncilValuation on an update is published to the council-ratings topic: first the council’s registry metadata, then a rating record linked to the Mill property (match method lookup). The site’s property ID, without dashes, stands in for the valuation number, and source is the site’s domain, so the property page credits it (“via homes.co.nz”).

Mill’s EnrichmentScheduler runs as a background goroutine every hour. Each cycle it:

  1. Queries the properties table for candidates, taking them round-robin across countries so a large backlog in one country cannot crowd the others out of the hourly limit (500):
    • Priority — properties with missing images, zero price, or zero bedrooms are queued immediately.
    • Staleness — properties whose updated_at is older than two days are also included.
  2. Applies a cooloff — a property is skipped if its enrichment_queued_at timestamp is less than 24 hours ago.
  3. Publishes one EnrichmentRequest message per candidate to the property-enrichment Kafka topic, using property_id as the message key.
  4. Stamps enrichment_queued_at = NOW() on all published properties.

The enrichment_queued_at column is added by Migration 04 and is NULL for properties that have never been queued.

connectors/common/enrichment_dispatcher.go runs one lane per country that has a primary enricher, AddressSearcher or ListingLinker. Each lane is its own kafka-go Reader and consumer group (connector-enrichment-dispatcher-<country>, e.g. connector-enrichment-dispatcher-nz), so lanes keep independent offsets. Every lane reads the whole topic, skips messages owned by other lanes and enriches its own one at a time; a message belongs to the country of its source’s primary enricher, or to its own country when the source has none (e.g. OpenStreetMap). Within a lane, each message is dispatched to the correct enricher by looking up req.Source in an internal sourceToEnricher map.

Because lanes run concurrently, a large USA backlog or a slow, rate-limited source only slows its own country. Each source still sees one request at a time, as before.

// Simplified flow of one lane (runLaneBatch); RunBatch runs every lane in parallel
for ownMessages < cfg.BatchSize {
msg, err := reader.FetchMessage(readCtx) // short deadline
if err != nil { break } // timeout = done
req := deserialize(msg)
if laneFor(req) != country { // another country's lane owns it
reader.CommitMessages(ctx, msg)
continue
}
enricher := sourceToEnricher[req.Source] // route by source
// Phase 1: Primary enrichment
enriched, _ := enricher.EnrichProperty(ctx, req)
// Phase 2: Gap-fill missing fields via AddressSearcher connectors
if missing := remainingMissingFields(enriched, req.MissingFields); len(missing) > 0 {
for _, searcher := range countryToSearchers[req.Country] {
found, _ := searcher.SearchByAddress(ctx, req.Address, req.City, req.Country)
mergeGaps(enriched, found, missing)
}
}
submitter.SubmitProperty(*enriched)
reader.CommitMessages(ctx, msg)
}

Key behaviours:

  • Batch size: defaults to 200 of a lane’s own messages per run (configurable); skipped messages do not count.
  • Continuous mode (-mode enrichers -mode-loop): every lane loops on its own, starting its next batch immediately when the last one was full and otherwise -mode-interval after the last one started.
  • Read timeout: 5 s — a lane ends its batch after draining it or timing out.
  • Per-source rate limiting: adaptive exponential backoff per source domain. If a source hits HTTP 429, its lane backs off and eventually skips that source for the rest of the batch; other lanes are unaffected.
  • Gap-fill cap: at most 2 AddressSearcher connectors are tried per message (configurable via MaxGapFill).
  • Offset commits: messages are always committed after processing (including errors and skips) to avoid tight retry loops. Commits are flushed asynchronously every second and when the lane’s reader closes. Mill’s deduplication and validation layers guard against corrupt data.
connector invocation
│
├─ 1. Discovery
│ ScrapeProperties(opts) → submit to Mill
│
└─ 2. Enrichment dispatch (only if -kafka-brokers is set)
EnrichmentDispatcher.RunBatch(ctx)
├─ one lane per country, in parallel, each its own consumer group
├─ each lane skips other countries' msgs and handles up to 200 of its own
├─ route each msg to correct enricher by source domain
├─ Phase 1: EnrichProperty(req) — re-scrape original source
├─ Phase 2: gap-fill missing fields via AddressSearcher (up to 2 per msg, same address only)
├─ Phase 3: link listings via ListingLinker; merge ListingDataLinker data
├─ submit merged result to Mill
└─ return (exit)

Add GetCountry() and EnrichProperty() to an existing connector struct. The method should reuse the connector’s existing single-page scraping logic, targeting the specific req.SourceURL instead of a paginated search URL.

func (c *MyConnector) GetCountry() string { return "XX" }
func (c *MyConnector) EnrichProperty(ctx context.Context, req common.EnrichmentRequest) (*common.Property, error) {
if req.SourceURL == "" || !strings.Contains(req.SourceURL, "mysite.com") {
return nil, nil
}
detail, err := c.scrapeDetailPage(req.SourceURL)
if err != nil {
return nil, err
}
return detail, nil
}

Return (nil, nil) when the URL is unrecognised — the consumer commits the offset and moves on.

Connector Country Source
homes-co-nz NZ homes.co.nz
harcourts-nz NZ harcourts.co.nz
harcourts NZ harcourts.com
homes-nz NZ homes.co.nz
realestate-nz NZ realestate.co.nz
raywhite-nz NZ raywhite.co.nz
realestate-au AU realestate.com.au
harcourts-au AU harcourts.com.au
domain-au AU domain.com.au
homely-au AU homely.com.au
allhomes-au AU allhomes.com.au
property-au AU property.com.au
view-au AU view.com.au
onthehouse-au AU onthehouse.com.au
rent-au AU rent.com.au
reiwa-au AU reiwa.com.au
realcommercial-au AU realcommercial.com.au
commercialrealestate-au AU commercialrealestate.com.au
hausples-pg PG hausples.com.pg
marketmeri-pg PG marketmeri.com
property-pg PG property.com.pg
property-com-pg PG property.com.pg
property-com-fj FJ property.com.fj
housingsamoa-com WS housingsamoa.com
zillow US zillow.com
realtor US realtor.com
redfin US redfin.com
realtor-ca CA realtor.ca
zolo-ca CA zolo.ca
point2homes-ca CA point2homes.com
inmuebles24 MX inmuebles24.com
vivanuncios-mx MX vivanuncios.com.mx
lamudi-mx MX lamudi.com.mx
vivareal-br BR vivareal.com.br
zapimoveis-br BR zapimoveis.com.br
olx-br BR olx.com.br
zonaprop-ar AR zonaprop.com.ar
argenprop-ar AR argenprop.com
mercadolibre-ar AR mercadolibre.com.ar
homedy VN homedy.com
suumo-jp JP suumo.jp
99acres-in IN 99acres.com
zigbang-kr KR zigbang.com
rightmove-co-uk GB rightmove.co.uk
maltapark MT maltapark.com
immobilienscout24-de DE immobilienscout24.de
seloger-fr FR seloger.com
immobiliare-it IT immobiliare.it
idealista-es ES idealista.com
funda-nl NL funda.nl
homegate-ch CH homegate.ch
aqar-sa SA aqar.fm

Add -kafka-brokers to an existing CronJob’s args. No new jobs are required.

args:
- "-connector"
- "homes-co-nz"
- "-mill-api"
- "http://mill:4000"
- "-kafka-brokers" # adds enrichment phase
- "redpanda:9092"
- "-max"
- "2000"

Connectors that do not implement Enricher silently skip the enrichment phase even when -kafka-brokers is supplied.

connectors/sources/maltapark/ is the reference implementation for a connector that ships with both Connector and Enricher from the start.

  • Discovery scrapes the for-sale and for-rent category pages on maltapark.com, collects listing anchor links, visits each detail page, and extracts address, EUR price, bedrooms, bathrooms, images, and description.
  • Enrichment re-visits a known listing URL to refresh any of those fields.
  • Country code: MT (Malta).
  • Currency: EUR.
  • CronJob: k8s/connectors/maltapark-connector-cronjob.yaml, scheduled hourly at minute 30.
Mill EnrichmentScheduler (hourly)
│
├─ SELECT candidates WHERE
│ (missing images/price/rooms OR updated_at < now-2d)
│ AND enrichment_queued_at < now-1d
│ round-robin across countries, LIMIT 500
│
├─ PUBLISH to "property-enrichment" topic
│ key = property_id, includes address + city + country
│
└─ UPDATE enrichment_queued_at = now
Connector CronJob (e.g. hourly)
│
├─ Discovery: scrape → submit to Mill (via Kafka or HTTP batch)
│
└─ Enrichment Dispatcher (one lane per country, in parallel)
├─ each lane reads "property-enrichment" and handles up to 200 of its country's msgs
├─ route msg → enricher by source domain
├─ Phase 1: EnrichProperty(sourceURL) — primary re-scrape
├─ Phase 2: gap-fill via AddressSearcher (same country, up to 2)
├─ merge gap-fill fields into primary result
└─ POST /connectors/properties/single