One core, two shells.
The rules of matching and merging live in a small core library with no database, web server or Spark dependency. The Job and the App are thin shells around it, and Unity Catalog tables are the contract between them.
The picture
Dashed line: the Job replays people's decisions and rules on every run.
What happens in a run
- Read every configured source table from Unity Catalog.
- Clean values: names, phones, addresses and identifiers are standardized so they can be compared.
- Find candidates using blocking keys (tax id, DUNS, website, name plus postal code) so it does not compare every record to every other.
- Score each candidate pair. Above the auto-merge threshold the records merge; between the review and auto-merge thresholds they go to the review queue; below it they stay separate. Defaults are 0.92 and 0.55 and both are configurable per domain.
- Apply decisions: a person's confirm or reject always beats the engine.
- Survive: pick the best value for each field. By default the most recently updated source wins, using the source's own timestamp, and you can set priorities per field.
- Write golden, crosswalk, review queue, lineage, rejects and run history, keeping
mdm_ids stable by overlap with the previous run.
The tables
| Table | Written by | Holds |
|---|---|---|
<domain>_golden | Job | One row per real-world entity |
_crosswalk | Job | Every source row mapped to its mdm_id |
_review_queue | Job | Pairs the engine would not decide alone |
_lineage | Job | Which source supplied each golden value, and why |
_rejects | Job | Source rows too incomplete to use |
_runs, _meta, _rules_preview | Job | Run history, contract version, and the effect of a proposed rule change |
_decisions | People | Confirm or reject, per pair |
_rules | People | Rule overrides |
Every table has exactly one writer, which is why the Job and the App can share them safely. Tables carry a contract version, and a run refuses to write over tables made under a different version.
Configurable by your team
Matching and survivorship rules are configuration, not code. They layer: product defaults, then the YAML next to your job, then a rules table in Unity Catalog. Later layers win, and the merged result is validated before anything is written. Before changing a rule, run a preview to see how many groups would merge, split or change a golden value.
# mdm_step.yaml (excerpt)
domain: account
output:
catalog: main
schema: mdm_account
sources:
- name: sfdc
table: main.gold.sfdc_account
id_column: account_id
updated_at_column: last_modified_date
columns: # canonical field: this table's column
account_name: name
website: website
- name: sfmc
table: main.gold.sfmc_account
id_column: subscriber_account_id
updated_at_column: modified_at
columns:
account_name: company_name
rules:
auto_merge_threshold: 0.94