Brand Name Normalization Rules

Company names are the messiest field in most databases, and cleaning them takes three things working together: clear brand name normalization rules, phonetic algorithms, and a small amount of AI.

I first worked this problem at AdDaptive Intelligence against roughly 2.3 million company names collected from ad exchanges, CRM exports and form fills. Nothing validated them on the way in, so the same company arrived a dozen different ways. The phonetic approach I built then still does the heavy lifting today, and the piece that was impossible in 2016 is now cheap: a second pass that hands each ambiguous bucket to a language model and lets it settle the cases phonetics gets wrong. I wrote up the original phonetic approach on the AdDaptive blog in 2016, and this is that method brought current.

Why Company Names Resist Normalization

Every one of these is a legitimate way to write the same company:

Wal Mart
Walmart
Wal*Mart
Wal-Mart Stores, Inc.
WAL MART STORES INC.

Unless you can validate against an authoritative dataset at write time, and almost nobody can, inconsistency is the default state. Search and replace, whitelists and regex patterns handle the first few hundred cases and then become a maintenance burden that grows faster than the data.

The reason exact matching fails is that it compares spellings. What you actually want to compare is how a name sounds, because human-entered variants of the same name almost always survive as homophones.

Brand Name Normalization Rules

Before any code runs, you need brand name normalization rules: an agreed list of what counts as noise. These are the terms that carry no identifying signal, grouped the way I maintain them in practice. Build your replacement arrays or regex alternations straight from these lists.

US Legal Entities

• Corporations: Inc, Inc., Incorporated, Corp, Corp., Corporation
• Limited liability: LLC, L.L.C., LC, L.C., Limited Liability Company, Limited Liability Co., PLLC
• Partnerships: LP, L.P., Limited Partnership, LLP, L.L.P., LLLP
• Professional: PC, P.C., Professional Corporation, PA, P.A.

UK, Commonwealth and Global Entities

• Limited: Ltd, Ltd., Limited, Lte, Ltee
• Public/private: PLC, P.L.C., Public Limited Company, Pty, Proprietary, Pte, Private
• Other: CIC, ULC

European and International Entities

• Germanic: GmbH, AG, KG, OHG, UG
• Francophone/Latin: SA, S.A., SARL, S.A.R.L., SAS, Cia, Cia., Compañia, Ltda
• Italian: SpA, S.p.A., Srl, S.r.l., Snc
• Dutch/Belgian: BV, B.V., NV, N.V., VOF
• Nordic: AB, AS, A/S, OY, OYJ, ApS
• Asian: KK, K.K. (Kabushiki Kaisha), YK, Sdn Bhd, Bhd

Generic Corporate and Structural Words

These do not denote legal status, but they get appended to core brand names constantly:

• Company, Co, Co.
• Group, The Group
• Holdings, Holding
• Partners, Partner
• Associates, Associate
• Enterprises, Enterprise
• International, Intl, Int’l
• Global, Worldwide
• Brands, Brand
• Technologies, Tech, Solutions, Systems, Services

Stop Words and Connectors

These disrupt exact matching more often than they help it:

• The, usually at the start (“The Walt Disney Company”)
• And, &, +, or
• Of

Order of Operations

Order of operations is where most implementations go wrong, so follow these three rules regardless of language:

1. Standardize casing and punctuation first. Lowercase everything and strip commas and periods before any term matching, so “Apple, Inc.” becomes “apple inc” and one pattern catches every punctuation variant.
2. Replace from longest to shortest. Strip “limited liability company” before “company” or “limited”. Strip “company” first and you are left with “limited liability”, which no pattern in your list will ever match again.
3. Use word boundaries. Anchor every term with b in regex so it only matches as a standalone word. You want to remove “Co” from “Ford Motor Co”, but you do not want to turn Costco into “stco”.

Pass One: Phonetic Bucketing

A phonetic algorithm converts a string into a code representing its pronunciation, so words that sound alike collapse to the same code regardless of spelling. That single property turns normalization from an impossible pairwise comparison into a grouping operation you can run in one pass over the data.

Phonetic Algorithms

Algorithm	Notes
Soundex	The oldest and most widely available, built into most databases. One letter plus three digits. Start here.
Metaphone	Handles English spelling irregularities far better than Soundex, variable length output.
Double Metaphone	Returns a primary and an alternate code, which is useful for names with more than one plausible pronunciation.
NYSIIS	New York State Identification and Intelligence System, tuned for surnames and slightly more precise than Soundex.
Caverphone	Built for matching names in New Zealand electoral rolls, aggressive at collapsing variants. This is what worked best on our data.
Cologne Phonetic	Designed for German, the right choice if your data is German language rather than English.
Match Rating Approach codex	Comes with its own comparison rating rather than a simple equality check.

There is no universally correct choice here. Start with Soundex because it is everywhere, measure how many duplicates it collapses on your data, then try Metaphone and Caverphone and keep whichever reduces your record count the most without merging companies that are genuinely different.

String Cleanup

Phonetic algorithms expect a pronounceable string, so the quality of your results is set by what you feed them. The cleanup step strips everything that carries no identifying signal: punctuation, accents, and above all legal suffixes, since “Inc” and “LLC” tell you nothing about which company you are looking at.

import { createReadStream } from 'node:fs';
import { createInterface } from 'node:readline';
import caverphone from 'talisman/phonetics/caverphone.js';

// Legal suffixes and filler that carry no identifying signal
const SUFFIXES = /\b(inc|incorporated|llc|ltd|limited|corp|corporation|co|company|plc|gmbh|ag|sa|nv|bv|pty|holdings|group)\b/g;

function clean(name) {
  return name
    .normalize('NFKD')                // separate accents from their letters
    .replace(/[\u0300-\u036f]/g, '')  // drop the accent marks
    .toLowerCase()
    .replace(/&/g, ' and ')
    .replace(/[^a-z0-9\s]/g, ' ')     // Wal*Mart and Wal-Mart converge here
    .replace(SUFFIXES, ' ')
    .replace(/\s+/g, ' ')
    .trim();
}

function bucketKey(name, length = 8) {
  const cleaned = clean(name);
  if (!cleaned) return null;

  // Coding each word separately and sorting makes the key word-order
  // independent, so "Stores Wal Mart" lands with "Wal Mart Stores"
  const codes = cleaned.split(' ').map(word => caverphone(word)).sort();

  // Phonetic specificity grows with length, so truncating the key
  // loosens the match. Tune this number against your own data.
  return codes.join('').slice(0, length);
}

Grouping is then a single streaming pass, which matters when the file is millions of lines long:

const buckets = new Map();

const input = createInterface({
  input: createReadStream('companies.txt'),
  crlfDelay: Infinity,
});

for await (const line of input) {
  const name = line.trim();
  if (!name) continue;

  const key = bucketKey(name);
  if (!key) continue;

  if (!buckets.has(key)) buckets.set(key, new Set());
  buckets.get(key).add(name);
}

console.log(`${buckets.size} buckets from the input file`);

PHP Implementation

PHP is the easiest version of this to write because soundex() and metaphone() are core language functions with no dependency to install:

<?php

function company_clean(string $name): string {
    $name = iconv('UTF-8', 'ASCII//TRANSLIT', $name);
    $name = strtolower($name);
    $name = str_replace('&', ' and ', $name);
    $name = preg_replace('/[^a-z0-9\s]/', ' ', $name);
    $name = preg_replace('/\b(inc|llc|ltd|limited|corp|corporation|co|company|plc|gmbh|holdings|group)\b/', ' ', $name);

    return trim(preg_replace('/\s+/', ' ', $name));
}

function company_bucket_key(string $name, int $length = 8): string {
    $words = array_filter(explode(' ', company_clean($name)));
    $codes = array_map('metaphone', $words);
    sort($codes);

    return substr(implode('', $codes), 0, $length);
}

$buckets = [];
foreach ($names as $name) {
    $buckets[company_bucket_key($name)][] = $name;
}

Python Implementation

Python needs one library, jellyfish, which implements Soundex, Metaphone, NYSIIS and the Match Rating codex behind a consistent interface:

import re
import unicodedata
from collections import defaultdict

import jellyfish

SUFFIXES = re.compile(
    r"\b(inc|incorporated|llc|ltd|limited|corp|corporation|co|company"
    r"|plc|gmbh|ag|sa|nv|bv|pty|holdings|group)\b"
)

def clean(name: str) -> str:
    name = unicodedata.normalize("NFKD", name)
    name = "".join(c for c in name if not unicodedata.combining(c))
    name = name.lower().replace("&", " and ")
    name = re.sub(r"[^a-z0-9\s]", " ", name)
    name = SUFFIXES.sub(" ", name)
    return re.sub(r"\s+", " ", name).strip()

def bucket_key(name: str, length: int = 8) -> str:
    words = clean(name).split()
    codes = sorted(jellyfish.metaphone(word) for word in words)
    return "".join(codes)[:length]

buckets: dict[str, list[str]] = defaultdict(list)
for name in names:
    buckets[bucket_key(name)].append(name)

SQL Implementation

If the data already lives in a database, you can bucket it without moving it anywhere:

-- MySQL, SQL Server, Oracle and DB2 all ship SOUNDEX in core
SELECT SOUNDEX(company_name) AS bucket_key,
       COUNT(*) AS variants,
       MIN(company_name) AS sample
FROM companies
GROUP BY bucket_key
HAVING COUNT(*) > 1
ORDER BY variants DESC;

-- PostgreSQL needs one extension, and gains metaphone and levenshtein with it
CREATE EXTENSION IF NOT EXISTS fuzzystrmatch;

SELECT dmetaphone(company_name) AS bucket_key, COUNT(*) AS variants
FROM companies
GROUP BY bucket_key
HAVING COUNT(*) > 1;

Pass One Results

Running the original version of this against a 20,000 record sample produced these numbers:

Records:        20001
Reduced:        10398
Execution Time: 6310ms
Algorithm:      caverphone

That is a 50% reduction in distinct company values, and the full 2.3 million names took roughly 12 minutes end to end. For a deterministic pass with no external calls and no per record cost, that is an excellent trade, and it is still the right first move on any dataset of this shape.

Where Phonetics Get It Wrong

The remaining half is where the interesting failures live, and they fall into two categories.

Names that should match but do not. Phonetic codes compare sound, so an acronym and its expansion are unrelated strings: “IBM” and “International Business Machines” never land in the same bucket. Neither do rebrands like “Facebook” and “Meta”, parent and subsidiary pairs like “Alphabet” and “Google”, or the same company written in two languages.

Names that match but should not. Aggressive truncation of the key is what makes the pass collapse variants, and it also drags in unrelated companies that happen to sound similar. “Delta Airlines” and “Delta Dental” are different businesses. So are “Apple” and “Apple Valley Dental”.

Then there is the canonical name problem. Picking the shortest string in each bucket is a fast heuristic and it produces “wal mart”, not “Walmart Inc.” It is fine for grouping and poor for anything a human will read.

Every one of these needs judgment about what a name refers to in the real world, which is precisely what a language model is good at and what no string algorithm can do.

Pass Two: AI Consolidation

The naive way to apply AI here fails on arithmetic. Comparing 2.3 million names pairwise is about 2.6 trillion comparisons, so any per comparison cost, model or otherwise, makes it impossible.

The hybrid works because pass one already did the expensive part. After bucketing you no longer have millions of loose names, you have a few hundred thousand buckets holding 10 to 20 names each, and each bucket is a small, self contained question. So you never ask the model to compare two names. You hand it a whole bucket and ask it to consolidate the bucket in one shot: split it into the real companies it contains and name each one properly.

That reframing is the entire efficiency story. One call resolves an entire bucket, and batching several buckets per call takes it further still.

Bucket Triage

The next saving is realizing that most buckets are not ambiguous at all. A bucket holding “walmart”, “wal mart” and “wal-mart stores inc” needs no intelligence to resolve, and cheap string similarity can tell you so. Score each bucket first, auto accept the confident ones, and send only the messy remainder onward:

from rapidfuzz import fuzz

def bucket_confidence(names: list[str]) -> float:
    """Mean pairwise similarity within a bucket, on cleaned strings."""
    if len(names) < 2:
        return 1.0

    cleaned = [clean(n) for n in names]
    scores = [
        fuzz.token_sort_ratio(a, b) / 100
        for i, a in enumerate(cleaned)
        for b in cleaned[i + 1:]
    ]
    return sum(scores) / len(scores)

AUTO_ACCEPT = 0.86

confident, ambiguous = {}, {}
for key, names in buckets.items():
    target = confident if bucket_confidence(names) >= AUTO_ACCEPT else ambiguous
    target[key] = names

print(f"{len(confident)} resolved by similarity, {len(ambiguous)} need review")

On our data this left well under 10% of buckets for the model. That is the number that makes the whole approach affordable: you are paying for intelligence only where the deterministic passes actually disagree.

Bucket Consolidation

Each remaining bucket goes to the model as a unit, with several buckets batched into one request. Asking for structured JSON keeps the output parseable and lets you batch aggressively:

import json
from anthropic import Anthropic

client = Anthropic()

SYSTEM = """You consolidate lists of company name variants.

For each bucket you receive, split the names into sets that refer to the same real
company, and give each set the correct legal name as its canonical form.

Rules:
- An acronym and its expansion are the same company (IBM, International Business Machines).
- A former name and its current name are the same company (Facebook, Meta).
- A parent and a subsidiary are different companies (Alphabet, Google).
- Companies that merely share a word are different (Delta Airlines, Delta Dental).
- If you are not confident a set of names is one company, keep them separate.

Return JSON only, shaped as:
[{"bucket_id": "...", "companies": [{"canonical": "...", "variants": ["..."], "confidence": 0.0}]}]
"""

def consolidate(batch: dict[str, list[str]]) -> list[dict]:
    payload = [{"bucket_id": k, "names": v} for k, v in batch.items()]

    message = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=8192,
        system=SYSTEM,
        messages=[{"role": "user", "content": json.dumps(payload)}],
    )
    return json.loads(message.content[0].text)

def batched(items, size=25):
    items = list(items)
    for i in range(0, len(items), size):
        yield dict(items[i:i + size])

resolved = []
for batch in batched(ambiguous.items()):
    resolved.extend(consolidate(batch))

The same call in Node.js, for a JavaScript pipeline:

import Anthropic from '@anthropic-ai/sdk';

const client = new Anthropic();

async function consolidate(batch) {
  const payload = Object.entries(batch).map(([bucketId, names]) => ({
    bucket_id: bucketId,
    names,
  }));

  const message = await client.messages.create({
    model: 'claude-haiku-4-5-20251001',
    max_tokens: 8192,
    system: SYSTEM,
    messages: [{ role: 'user', content: JSON.stringify(payload) }],
  });

  return JSON.parse(message.content[0].text);
}

Use the smallest model that holds up on a sample of your own buckets. This is classification against world knowledge, not reasoning, so a fast cheap model is usually the right call and the cost difference at this volume is substantial.

Decision Caching

Bucket contents are stable between runs, so a decision made once should never be paid for twice. Hash the bucket and look it up before calling anything:

import hashlib

def bucket_signature(names: list[str]) -> str:
    return hashlib.sha256("|".join(sorted(names)).encode()).hexdigest()

Write the results into a lookup table that your application reads directly, so normalization at runtime is a single indexed query rather than any kind of computation:

CREATE TABLE company_aliases (
  raw_name     VARCHAR(255) NOT NULL,
  canonical    VARCHAR(255) NOT NULL,
  method       ENUM('phonetic', 'similarity', 'ai', 'manual') NOT NULL,
  confidence   DECIMAL(3, 2) NOT NULL,
  decided_at   DATETIME NOT NULL,
  PRIMARY KEY (raw_name),
  KEY canonical (canonical)
);

Keeping the method and confidence columns matters more than it looks. When a grouping turns out to be wrong six months from now, they tell you which pass made the call, and they let you re run just the low confidence decisions when you change models or thresholds.

Scaling: Pure AI vs Pre-Clustering

Once an LLM enters the pipeline, the architecture around it matters as much as the prompt. Modern models have no trouble understanding that “Apple, Inc.” and “Apple Incorporated” are the same entity. The problem is economic: feed them millions of rows the naive way and you get a massive API bill and days of processing time. Which architecture you want comes down to the size of the dataset.

Pure AI with Array Batching (Under 1M Records)

For moderately sized datasets, say a list of one million company names, the simplest viable approach is to let the model do all the cleaning and standardizing itself. The trick that makes it affordable is batching.

Send one million individual API requests and you pay for your system instructions (“You are a data cleaner. Remove LLC, Inc…”) one million times over. Network latency drags the job out for hours, and you will hit provider rate limits long before it finishes. Instead:

1. Chunk the data. Group the raw names into arrays of 50 to 100 items.
2. Send the batch as one prompt. Pass the whole array and instruct the model to return a cleanly mapped JSON array of normalized names.
3. Reconstruct. Receive the batch and update the database records from the mapping.

Batching divides your network requests and your instruction-token overhead by 100. On modern low-cost models, a million records goes from a slow, expensive operation to a job that finishes quickly and costs a few dollars.

Phonetic Pre-Clustering (10M+ Records)

At 10 or 50 million names, pure AI breaks down, and the reason is specific: output tokens. Output tokens cost several times more than input tokens and dictate the model’s generation latency. Asking a model to generate 10 million distinct strings is slow and expensive, and millions of those strings are near-identical variations of the same company anyway.

The fix is the hybrid architecture this whole post describes: use fast, free, local compute to group similar names first, and only then bring in the model.

1. Exact deduplication. A simple GROUP BY collapses every perfectly identical string at zero cost, which alone removes millions of rows.
2. Phonetic clustering. Run the remaining unique strings through Soundex, Metaphone or Double Metaphone so “Acme Corp”, “Akme Corporation” and “Acme Corp.” land in the same cluster.
3. Batched canonicalization. Send the model clusters, not rows. Pass the array of variations and ask one question: return the single best canonical version of this company name.

Pre-clustering routinely compresses 10 million raw rows into something like 1.5 million distinct clusters, so the model outputs 1.5 million canonical names instead of 10 million individual ones. In practice that cuts total token cost by 60 to 80 percent and speeds the pipeline up by an order of magnitude, because you stopped paying the model to repeat itself.

The rule of thumb: under a million records, batched pure AI is simpler and cheap enough that the pipeline is not worth building. Past that, output tokens become the bill, and every row you resolve locally is a row the model never has to generate.

The Pipeline End to End

1. Clean each name into a pronounceable string, stripping punctuation, accents and legal suffixes.
2. Bucket by phonetic code, truncating the key to control how loose the match is. Cheap, deterministic, one pass over the data.
3. Score each bucket with string similarity and auto accept the confident majority.
4. Consolidate only the ambiguous buckets with a language model, a whole bucket at a time, batched and cached.
5. Store the result as an alias table with the method and confidence behind every decision.

The division of labor is what makes it work. Phonetics are free and handle the bulk mechanical reduction, similarity scoring settles the obvious cases for nothing, and the model only sees the small slice of genuinely hard decisions where its judgment is worth paying for. The 2016 version of this reduced our name count by half; adding the second pass fixes the acronyms, rebrands and false merges that the first pass structurally cannot, and gives you canonical names a person can actually read.

If you are doing this work at warehouse scale, the same pattern applies directly inside BigQuery, and my notes on importing large CSVs into BigQuery and querying it efficiently cover the surrounding plumbing. The related problem of figuring out which company a website visitor belongs to in the first place is covered in web visitor company identification.

Phonetic Functions by Language

Nothing above is specific to JavaScript, PHP or Python. Every mainstream language and database gives you the same building blocks, usually under the same name, because they all implement the same small set of published algorithms:

Platform	What to reach for
PHP	soundex(), metaphone(), similar_text(), levenshtein(), all in core
Python	jellyfish, phonetics, rapidfuzz
Node.js	talisman, natural, double-metaphone
Ruby	text gem (Soundex, Metaphone, Double Metaphone, NYSIIS)
Java	Apache Commons Codec (Soundex, Metaphone, DoubleMetaphone, Nysiis)
C# and .NET	Phonix, FuzzyString
Go	go-phonetics, smetrics
MySQL, SQL Server, Oracle, DB2	SOUNDEX() in core
PostgreSQL	fuzzystrmatch extension: soundex(), metaphone(), dmetaphone(), levenshtein()
BigQuery and Snowflake	SOUNDEX() plus edit distance functions

They are all called phonetic functions, and they all do the same job: turn a word into a code that represents how it sounds so you can compare pronunciation instead of spelling. Pick whichever your stack already has, get the bucketing pass working there, and add the AI consolidation pass on top of it in whatever language is most convenient. The architecture travels; only the function names change.