Normalizing Company Names with Phonetic Algorithms and AI

Company names are the messiest field in most databases, and phonetic algorithms plus a small amount of AI will clean them up better than either approach alone.

I first worked this problem at AdDaptive Intelligence against roughly 2.3 million company names collected from ad exchanges, CRM exports and form fills. Nothing validated them on the way in, so the same company arrived a dozen different ways. The phonetic approach I built then still does the heavy lifting today, and the piece that was impossible in 2016 is now cheap: a second pass that hands each ambiguous bucket to a language model and lets it settle the cases phonetics gets wrong. I wrote up the original phonetic approach on the AdDaptive blog in 2016, and this is that method brought current.

Why Company Names Resist Normalization

Every one of these is a legitimate way to write the same company:

Wal Mart
Walmart
Wal*Mart
Wal-Mart Stores, Inc.
WAL MART STORES INC.

Unless you can validate against an authoritative dataset at write time, and almost nobody can, inconsistency is the default state. Search and replace, whitelists and regex patterns handle the first few hundred cases and then become a maintenance burden that grows faster than the data.

The reason exact matching fails is that it compares spellings. What you actually want to compare is how a name sounds, because human-entered variants of the same name almost always survive as homophones.

Pass One: Phonetic Bucketing

A phonetic algorithm converts a string into a code representing its pronunciation, so words that sound alike collapse to the same code regardless of spelling. That single property turns normalization from an impossible pairwise comparison into a grouping operation you can run in one pass over the data.

The Algorithms Worth Knowing

Algorithm	Notes
Soundex	The oldest and most widely available, built into most databases. One letter plus three digits. Start here.
Metaphone	Handles English spelling irregularities far better than Soundex, variable length output.
Double Metaphone	Returns a primary and an alternate code, which is useful for names with more than one plausible pronunciation.
NYSIIS	New York State Identification and Intelligence System, tuned for surnames and slightly more precise than Soundex.
Caverphone	Built for matching names in New Zealand electoral rolls, aggressive at collapsing variants. This is what worked best on our data.
Cologne Phonetic	Designed for German, the right choice if your data is German language rather than English.
Match Rating Approach codex	Comes with its own comparison rating rather than a simple equality check.

There is no universally correct choice here. Start with Soundex because it is everywhere, measure how many duplicates it collapses on your data, then try Metaphone and Caverphone and keep whichever reduces your record count the most without merging companies that are genuinely different.

Cleanup Comes First

Phonetic algorithms expect a pronounceable string, so the quality of your results is set by what you feed them. The cleanup step strips everything that carries no identifying signal: punctuation, accents, and above all legal suffixes, since “Inc” and “LLC” tell you nothing about which company you are looking at.

import { createReadStream } from 'node:fs';
import { createInterface } from 'node:readline';
import caverphone from 'talisman/phonetics/caverphone.js';

// Legal suffixes and filler that carry no identifying signal
const SUFFIXES = /b(inc|incorporated|llc|ltd|limited|corp|corporation|co|company|plc|gmbh|ag|sa|nv|bv|pty|holdings|group)b/g;

function clean(name) {
  return name
    .normalize('NFKD')                // separate accents from their letters
    .replace(/[u0300-u036f]/g, '')  // drop the accent marks
    .toLowerCase()
    .replace(/&/g, ' and ')
    .replace(/[^a-z0-9s]/g, ' ')     // Wal*Mart and Wal-Mart converge here
    .replace(SUFFIXES, ' ')
    .replace(/s+/g, ' ')
    .trim();
}

function bucketKey(name, length = 8) {
  const cleaned = clean(name);
  if (!cleaned) return null;

  // Coding each word separately and sorting makes the key word-order
  // independent, so "Stores Wal Mart" lands with "Wal Mart Stores"
  const codes = cleaned.split(' ').map(word => caverphone(word)).sort();

  // Phonetic specificity grows with length, so truncating the key
  // loosens the match. Tune this number against your own data.
  return codes.join('').slice(0, length);
}

Grouping is then a single streaming pass, which matters when the file is millions of lines long:

const buckets = new Map();

const input = createInterface({
  input: createReadStream('companies.txt'),
  crlfDelay: Infinity,
});

for await (const line of input) {
  const name = line.trim();
  if (!name) continue;

  const key = bucketKey(name);
  if (!key) continue;

  if (!buckets.has(key)) buckets.set(key, new Set());
  buckets.get(key).add(name);
}

console.log(`${buckets.size} buckets from the input file`);

The Same Thing in PHP

PHP is the easiest version of this to write because soundex() and metaphone() are core language functions with no dependency to install:

<?php

function company_clean(string $name): string {
    $name = iconv('UTF-8', 'ASCII//TRANSLIT', $name);
    $name = strtolower($name);
    $name = str_replace('&', ' and ', $name);
    $name = preg_replace('/[^a-z0-9s]/', ' ', $name);
    $name = preg_replace('/b(inc|llc|ltd|limited|corp|corporation|co|company|plc|gmbh|holdings|group)b/', ' ', $name);

    return trim(preg_replace('/s+/', ' ', $name));
}

function company_bucket_key(string $name, int $length = 8): string {
    $words = array_filter(explode(' ', company_clean($name)));
    $codes = array_map('metaphone', $words);
    sort($codes);

    return substr(implode('', $codes), 0, $length);
}

$buckets = [];
foreach ($names as $name) {
    $buckets[company_bucket_key($name)][] = $name;
}

The Same Thing in Python

Python needs one library, jellyfish, which implements Soundex, Metaphone, NYSIIS and the Match Rating codex behind a consistent interface:

import re
import unicodedata
from collections import defaultdict

import jellyfish

SUFFIXES = re.compile(
    r"b(inc|incorporated|llc|ltd|limited|corp|corporation|co|company"
    r"|plc|gmbh|ag|sa|nv|bv|pty|holdings|group)b"
)

def clean(name: str) -> str:
    name = unicodedata.normalize("NFKD", name)
    name = "".join(c for c in name if not unicodedata.combining(c))
    name = name.lower().replace("&", " and ")
    name = re.sub(r"[^a-z0-9s]", " ", name)
    name = SUFFIXES.sub(" ", name)
    return re.sub(r"s+", " ", name).strip()

def bucket_key(name: str, length: int = 8) -> str:
    words = clean(name).split()
    codes = sorted(jellyfish.metaphone(word) for word in words)
    return "".join(codes)[:length]

buckets: dict[str, list[str]] = defaultdict(list)
for name in names:
    buckets[bucket_key(name)].append(name)

And Directly in SQL

If the data already lives in a database, you can bucket it without moving it anywhere:

-- MySQL, SQL Server, Oracle and DB2 all ship SOUNDEX in core
SELECT SOUNDEX(company_name) AS bucket_key,
       COUNT(*) AS variants,
       MIN(company_name) AS sample
FROM companies
GROUP BY bucket_key
HAVING COUNT(*) > 1
ORDER BY variants DESC;

-- PostgreSQL needs one extension, and gains metaphone and levenshtein with it
CREATE EXTENSION IF NOT EXISTS fuzzystrmatch;

SELECT dmetaphone(company_name) AS bucket_key, COUNT(*) AS variants
FROM companies
GROUP BY bucket_key
HAVING COUNT(*) > 1;

What Pass One Achieved

Running the original version of this against a 20,000 record sample produced these numbers:

Records:        20001
Reduced:        10398
Execution Time: 6310ms
Algorithm:      caverphone

That is a 50% reduction in distinct company values, and the full 2.3 million names took roughly 12 minutes end to end. For a deterministic pass with no external calls and no per record cost, that is an excellent trade, and it is still the right first move on any dataset of this shape.

Where Phonetics Get It Wrong

The remaining half is where the interesting failures live, and they fall into two categories.

Names that should match but do not. Phonetic codes compare sound, so an acronym and its expansion are unrelated strings: “IBM” and “International Business Machines” never land in the same bucket. Neither do rebrands like “Facebook” and “Meta”, parent and subsidiary pairs like “Alphabet” and “Google”, or the same company written in two languages.

Names that match but should not. Aggressive truncation of the key is what makes the pass collapse variants, and it also drags in unrelated companies that happen to sound similar. “Delta Airlines” and “Delta Dental” are different businesses. So are “Apple” and “Apple Valley Dental”.

Then there is the canonical name problem. Picking the shortest string in each bucket is a fast heuristic and it produces “wal mart”, not “Walmart Inc.” It is fine for grouping and poor for anything a human will read.

Every one of these needs judgment about what a name refers to in the real world, which is precisely what a language model is good at and what no string algorithm can do.

Pass Two: Let AI Consolidate Each Bucket

The naive way to apply AI here fails on arithmetic. Comparing 2.3 million names pairwise is about 2.6 trillion comparisons, so any per comparison cost, model or otherwise, makes it impossible.

The hybrid works because pass one already did the expensive part. After bucketing you no longer have millions of loose names, you have a few hundred thousand buckets holding 10 to 20 names each, and each bucket is a small, self contained question. So you never ask the model to compare two names. You hand it a whole bucket and ask it to consolidate the bucket in one shot: split it into the real companies it contains and name each one properly.

That reframing is the entire efficiency story. One call resolves an entire bucket, and batching several buckets per call takes it further still.

Triage: Most Buckets Never Need the Model

The next saving is realizing that most buckets are not ambiguous at all. A bucket holding “walmart”, “wal mart” and “wal-mart stores inc” needs no intelligence to resolve, and cheap string similarity can tell you so. Score each bucket first, auto accept the confident ones, and send only the messy remainder onward:

from rapidfuzz import fuzz

def bucket_confidence(names: list[str]) -> float:
    """Mean pairwise similarity within a bucket, on cleaned strings."""
    if len(names) < 2:
        return 1.0

    cleaned = [clean(n) for n in names]
    scores = [
        fuzz.token_sort_ratio(a, b) / 100
        for i, a in enumerate(cleaned)
        for b in cleaned[i + 1:]
    ]
    return sum(scores) / len(scores)

AUTO_ACCEPT = 0.86

confident, ambiguous = {}, {}
for key, names in buckets.items():
    target = confident if bucket_confidence(names) >= AUTO_ACCEPT else ambiguous
    target[key] = names

print(f"{len(confident)} resolved by similarity, {len(ambiguous)} need review")

On our data this left well under 10% of buckets for the model. That is the number that makes the whole approach affordable: you are paying for intelligence only where the deterministic passes actually disagree.

Consolidating a Bucket

Each remaining bucket goes to the model as a unit, with several buckets batched into one request. Asking for structured JSON keeps the output parseable and lets you batch aggressively:

import json
from anthropic import Anthropic

client = Anthropic()

SYSTEM = """You consolidate lists of company name variants.

For each bucket you receive, split the names into sets that refer to the same real
company, and give each set the correct legal name as its canonical form.

Rules:
- An acronym and its expansion are the same company (IBM, International Business Machines).
- A former name and its current name are the same company (Facebook, Meta).
- A parent and a subsidiary are different companies (Alphabet, Google).
- Companies that merely share a word are different (Delta Airlines, Delta Dental).
- If you are not confident a set of names is one company, keep them separate.

Return JSON only, shaped as:
[{"bucket_id": "...", "companies": [{"canonical": "...", "variants": ["..."], "confidence": 0.0}]}]
"""

def consolidate(batch: dict[str, list[str]]) -> list[dict]:
    payload = [{"bucket_id": k, "names": v} for k, v in batch.items()]

    message = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=8192,
        system=SYSTEM,
        messages=[{"role": "user", "content": json.dumps(payload)}],
    )
    return json.loads(message.content[0].text)

def batched(items, size=25):
    items = list(items)
    for i in range(0, len(items), size):
        yield dict(items[i:i + size])

resolved = []
for batch in batched(ambiguous.items()):
    resolved.extend(consolidate(batch))

The same call in Node.js, for a JavaScript pipeline:

import Anthropic from '@anthropic-ai/sdk';

const client = new Anthropic();

async function consolidate(batch) {
  const payload = Object.entries(batch).map(([bucketId, names]) => ({
    bucket_id: bucketId,
    names,
  }));

  const message = await client.messages.create({
    model: 'claude-haiku-4-5-20251001',
    max_tokens: 8192,
    system: SYSTEM,
    messages: [{ role: 'user', content: JSON.stringify(payload) }],
  });

  return JSON.parse(message.content[0].text);
}

Use the smallest model that holds up on a sample of your own buckets. This is classification against world knowledge, not reasoning, so a fast cheap model is usually the right call and the cost difference at this volume is substantial.

Cache Every Decision

Bucket contents are stable between runs, so a decision made once should never be paid for twice. Hash the bucket and look it up before calling anything:

import hashlib

def bucket_signature(names: list[str]) -> str:
    return hashlib.sha256("|".join(sorted(names)).encode()).hexdigest()

Write the results into a lookup table that your application reads directly, so normalization at runtime is a single indexed query rather than any kind of computation:

CREATE TABLE company_aliases (
  raw_name     VARCHAR(255) NOT NULL,
  canonical    VARCHAR(255) NOT NULL,
  method       ENUM('phonetic', 'similarity', 'ai', 'manual') NOT NULL,
  confidence   DECIMAL(3, 2) NOT NULL,
  decided_at   DATETIME NOT NULL,
  PRIMARY KEY (raw_name),
  KEY canonical (canonical)
);

Keeping the method and confidence columns matters more than it looks. When a grouping turns out to be wrong six months from now, they tell you which pass made the call, and they let you re run just the low confidence decisions when you change models or thresholds.

The Pipeline End to End

1. Clean each name into a pronounceable string, stripping punctuation, accents and legal suffixes.
2. Bucket by phonetic code, truncating the key to control how loose the match is. Cheap, deterministic, one pass over the data.
3. Score each bucket with string similarity and auto accept the confident majority.
4. Consolidate only the ambiguous buckets with a language model, a whole bucket at a time, batched and cached.
5. Store the result as an alias table with the method and confidence behind every decision.

The division of labor is what makes it work. Phonetics are free and handle the bulk mechanical reduction, similarity scoring settles the obvious cases for nothing, and the model only sees the small slice of genuinely hard decisions where its judgment is worth paying for. The 2016 version of this reduced our name count by half; adding the second pass fixes the acronyms, rebrands and false merges that the first pass structurally cannot, and gives you canonical names a person can actually read.

If you are doing this work at warehouse scale, the same pattern applies directly inside BigQuery, and my notes on importing large CSVs into BigQuery and querying it efficiently cover the surrounding plumbing. The related problem of figuring out which company a website visitor belongs to in the first place is covered in web visitor company identification.

Any Language Will Do This

Nothing above is specific to JavaScript, PHP or Python. Every mainstream language and database gives you the same building blocks, usually under the same name, because they all implement the same small set of published algorithms:

Platform	What to reach for
PHP	soundex(), metaphone(), similar_text(), levenshtein(), all in core
Python	jellyfish, phonetics, rapidfuzz
Node.js	talisman, natural, double-metaphone
Ruby	text gem (Soundex, Metaphone, Double Metaphone, NYSIIS)
Java	Apache Commons Codec (Soundex, Metaphone, DoubleMetaphone, Nysiis)
C# and .NET	Phonix, FuzzyString
Go	go-phonetics, smetrics
MySQL, SQL Server, Oracle, DB2	SOUNDEX() in core
PostgreSQL	fuzzystrmatch extension: soundex(), metaphone(), dmetaphone(), levenshtein()
BigQuery and Snowflake	SOUNDEX() plus edit distance functions

They are all called phonetic functions, and they all do the same job: turn a word into a code that represents how it sounds so you can compare pronunciation instead of spelling. Pick whichever your stack already has, get the bucketing pass working there, and add the AI consolidation pass on top of it in whatever language is most convenient. The architecture travels; only the function names change.