Lab 1: Cognate Detection with LingPy

Naxos Summer School 2026 · Phylogenetic Methods in Historical Linguistics

Nothing to install

This page runs Python inside your browser. There is no software to install, no account to create, and nothing you write is sent to a server. (Python itself is fetched from a public CDN the first time the page loads.) Edit any cell and press Run Code.

The first cell takes about ten seconds while Python loads. Everything after that is fast.

Why this lab exists

Every phylogenetic tree of a language family rests on a character matrix, and for lexical data every column of that matrix is a cognate set: a claim that these words in these languages descend from one ancestral word. Lectures 3 and 4 are concerned with what happens after that matrix exists, i.e. with the substitution model, the likelihood, and the MCMC.

This lab is about where the matrix comes from. That is not a technicality. A tree can only be as good as the cognate judgements underneath it, and those judgements are precisely the part that historical linguists argue about.

Traditionally a specialist produces them by hand, over years. The question here is how far an algorithm gets on its own, and where exactly it breaks.

Setup

We load LingPy (List 2012, 2017), the standard Python toolkit for quantitative historical linguistics.

1. What does “similar” mean?

Before cognates can be found automatically, a linguist’s intuition, these two words look related, has to be turned into a number. The crudest such number is edit distance: the number of insertions, deletions and substitutions required to turn one word into the other.

Two things have to be right before we may even compute it. The words must be given in phonetic transcription, not in spelling, since orthography is a poor guide to sound. And they must be cut into segments, which is a job for LingPy’s tokenizer, ipa2tokens(), and not for a naive split on characters. A character split would treat the length mark of /iː/ as a segment in its own right, and would tear the German diphthong /aɪ/ and the affricate /t͡s/ in half.

Note that ipa2tokens() keeps /iː/, /aɪ/ and /t͡s/ together, exactly as the wordlist we load in §3 does. Now the distances mean something.

Each of these four is a different kind of case, and edit distance cannot tell them apart. Worse, it ranks them in almost exactly the wrong order.

/hænd/ ~ /hant/ are genuine cognates, and they look it. Distance 2.
/tuː/ ~ /t͡svaɪ/ are genuine cognates that look nothing alike. Distance 3, the largest of the four. The correspondence English /t/ : German /t͡s/ is entirely regular (ten/zehn, tongue/Zunge), but no string metric will discover that from a single pair.
/biːf/ ~ /bœf/ is a loan, borrowed by English from Norman French. Distance 1, the smallest of the four. The words are related, but not by descent, and putting them in one cognate set would poison the tree, which would read the resemblance as shared inheritance.
/mʌt͡ʃ/ ~ /mut͡ʃo/ are not related at all. English much goes back to Old English myċel, Spanish mucho to Latin multus. They converged by accident, at distance 2, which is exactly the score of the true cognate hand / Hand.

Read that list again. The loan is the most similar pair, a true cognate is the least similar, and the chance pair is indistinguishable from a real one.

The central problem of this lab

Surface similarity is neither necessary nor sufficient for cognacy. Two and zwei are cognate and look distant; much and mucho look nearly identical and are unrelated.

Any method built on raw string similarity will therefore make both errors. What distinguishes real cognates is not that the sounds are similar, but that they correspond regularly: the same substitution, in the same environment, across many words. This is the Neogrammarian insight from Lecture 1, and LexStat is an attempt to learn it from data.

2. Alignment: keeping the where

Edit distance collapses everything into one number, and thereby discards the most valuable information, namely where the differences are. Alignment keeps it. The aligned columns are exactly where a sound correspondence lives, since a correspondence is nothing more than a column that keeps recurring.

Read the columns, not the score. Each cell is coloured by the segment’s sound class, so a column of one colour holds sounds of the same kind.

Now look at column 1, which comes out in two colours. θ and d are not similar sounds, and the colouring says so. The column is nevertheless real: θ : d is the Germanic reflex of PIE *t (Grimm’s Law, followed by the High German shift). If the same column recurred in thou/du, thick/dick and thorn/Dorn, we would have a correspondence, and that is evidence.

Column 2 (r : r) is the trivial case. Column 3 pairs the vowels, /iː/ against /aɪ/, which is the regular English-German reflex of Middle English /iː/ (compare mine/mein, time/Zeit). Had we split the diphthong, that column would have been nonsense.

This is worth holding on to, because it is the whole lab in one picture. What makes a correspondence is not that the sounds resemble each other, but that the column keeps coming back. Column 2 is the easy case, which any similarity score gets right. Column 1 is the case that similarity gets wrong, and it is the interesting one. A single alignment is an anecdote; a recurring column is data.

Recurrence, not similarity, is what LexStat will try to learn.

3. The data

We use a subset of IECor (Indo-European Cognate Relationships), a curated database in which specialists have coded cognate sets by hand. That hand-coding is our gold standard: it allows us to score an algorithm against expert judgement, rather than admire its output and hope.

Every row is one word, with five fields that concern us:

column	what it is
`DOCULECT`	the language
`CONCEPT`	the meaning slot (hand, two, water …)
`VALUE`	the form, as written
`TOKENS`	the form as phonetic segments, i.e. what the algorithm sees
`COGSET`	the expert’s cognate set ID, i.e. the answer key

The TOKENS column matters more than it looks. Feeding an algorithm orthography is hopeless, since English ough is six different sounds. Segmented phonetic transcription is what makes any of this tractable, and producing it is real work.

The composition of that language list shapes everything that follows. There are 13 languages from five branches: Germanic (Dutch, German, Swedish), Romance (French, Italian, Spanish, Latin), Slavic (Polish, Russian), Hellenic (Ancient Greek, Greek) and Indo-Iranian (Hindi, Vedic Sanskrit). Three of them are ancient (Latin, Ancient Greek, Vedic Sanskrit), which matters: ancient languages are closer to the proto-language and preserve correspondences that their modern descendants have obscured.

There is no Anatolian and no Tocharian, hence no deep outgroup. This is worth remembering in Lab 2, where we have to root the tree and find that we have nothing to root it on.

Look at one concept

This is the raw material: one meaning, thirteen languages, and the expert’s verdict on which of them descend from the same word.

Change CONCEPT and re-run. Some concepts are clean, with one cognate set per branch. Others are a mess of many small sets, because the word has been replaced repeatedly, and replacement is precisely the signal that phylogenetics feeds on. A concept for which every language has the same cognate is beautiful and tells us nothing, since it distinguishes no tree from any other. The information lives in the changes.

4. LexStat: learning the correspondences

Now the real method (List 2012). Its core idea is a direct implementation of the Neogrammarian insight:

Align every pair of words for the same concept, across every pair of languages.
Count which segments actually end up aligned with which.
Compare that to how often they would align by chance, by shuffling the data to destroy any real relationship and re-counting.
A pair of segments that aligns far more often than chance predicts is a correspondence, and is scored highly. A pair that aligns no more often than chance is noise, and is scored near zero.

Step 3 is the expensive one, since it is a permutation test, and it is why the next cell takes a few seconds. What comes out is a scorer that is specific to each language pair. It has learned from the data alone that English /t/ corresponds to German /ts/, without being told.

From scores to sets: the clustering step

That gives us scores, though, not sets. Turning the one into the other is the second half of the algorithm, and it works one concept at a time: the words for hand are clustered among themselves, the words for water among themselves, and whether hand and water are related is a question never asked. The meaning slot does that work for us, which is precisely why a wordlist is organised by concept in the first place.

Within a slot, the alignment scores are converted to distances, and the words are grouped by a flat variant of UPGMA. Every word starts alone. The two nearest sets are merged, where the distance between two sets is the average distance between their members; then the next two, and so on. Ordinary UPGMA keeps going until everything has collapsed into one set, which is no use to us. This variant stops the moment the nearest remaining pair lies further apart than a threshold τ. Whatever is still separate when it stops is a cognate set.

So τ decides how much phonetic distance we are willing to call inheritance. Below it words merge, above it they stay apart. The cell uses threshold=0.55; §5 is about why that number is a choice and not a setting.

We use runs=1000 rather than the usual 10000. In the browser that is about 12 seconds instead of roughly two minutes, and it costs almost nothing in accuracy.

The permutation test is randomised, so these numbers move slightly from run to run. Do not read the third decimal.

What those three numbers mean

They are B-cubed scores, and they are not accuracy. Cognate detection is a clustering problem, since we are partitioning words into sets, and a partition has to be scored as a partition. There is no list of right answers to tick off: there are two ways of carving up the same words, and we want to know how far apart they are.

B-cubed does this one word at a time. Take Latin manus. LexStat has placed it in some set, the expert has placed it in another, and we compare the two sets from the point of view of that single word. The share of LexStat’s set that the expert also put with manus is its precision. The share of the expert’s set that LexStat managed to include is its recall. Do that for all 2,159 words, average, and you have the numbers above.

Scoring per word rather than per set is the whole point of the measure. Every word contributes exactly once, so a single enormous cluster cannot dominate the result, and neither can a long tail of singletons, which matters here, because LexStat produces a great many singletons.

Precision (about 0.97): of the words LexStat puts alongside a given word, almost all genuinely belong there. When LexStat says two words are cognate, it is nearly always right.
Recall (about 0.70): of the words the expert puts alongside a given word, LexStat finds about seven in ten. It misses roughly a third of the real cognates.
F (about 0.81): the harmonic mean of the two, and the single number usually quoted.

The error is thus lopsided, and lopsided in a revealing direction. LexStat is cautious. It splits real cognate sets into fragments rather than lumping unrelated words together. It fails on cases like two / zwei, i.e. on true cognates that no longer look alike, because the phonetic distance has grown too great for the evidence in 170 concepts to bridge.

This is not a bug that can be tuned away. It is the time-depth ceiling from Lecture 5, showing up here as a recall number.

Now the same concept as in §3, but grouped by LexStat’s clusters, with the expert’s answer beside it.

Read the two right-hand columns against each other. Wherever one gold set is spread over several LexStat blocks, that is a split: the recall problem, in a single picture. Wherever one LexStat block contains two different gold numbers, that is a merge, and those are far rarer, which is exactly what precision 0.97 against recall 0.70 means.

Did it actually learn the correspondences?

An F-score is a summary, and a summary can hide anything. Let us look at what LexStat produced. We take one of its cognate sets, align the whole set, and read the columns.

Nobody told LexStat which segments correspond. It inferred that from the data alone, and the columns are the proof.

Column 1 is a single colour: the reflexes of PIE *t. Dutch and German have d, while the rest have t, and that includes their Germanic cousin Swedish. So the column is not simply “Germanic against the rest”, and it is worth pausing on why. Proto-Germanic *þ (Grimm’s Law, Lecture 1) became d in Dutch and German, but t in Swedish. The column is real; the naive reading of it is wrong. Read the correspondences, not the language labels.

(There is no English in this dataset, which is why column 1 comes out in one colour here and in two in §2. English kept the fricative θ, and a fricative is not the same sound class as a stop. Same correspondence, different reflex.)

Column 2 is the one to look at. The rhotics are not the same segment from language to language: a trill in one, a uvular fricative in another, a tap in a third. The column is nevertheless a single colour, because all of them are rhotics. This is exactly what §2 promised: what makes a correspondence is not that the sounds resemble each other, but that the column keeps coming back.

The vowels are a different matter, and come out in several colours. Vowels are less stable than consonants, and the algorithm shows this rather than hiding it.

Count the languages in the title, and note who is missing. Several languages have a perfectly good cognate of three, yet LexStat has left them out of this set and put them in one of their own. That is the recall problem of the next section, visible in a single picture: LexStat splits rather than lumps.

Because the permutation test is randomised, the exact membership shifts a little from run to run. Press Run Code again and watch it change. A method whose output moves when you re-run it is telling you something about how much evidence it really had.

Now change CONCEPT and look again. Some sets are this clean, others are ragged, and the ragged ones are where the missing recall comes from.

5. The threshold is a choice, not a setting

τ is the distance cutoff for clustering. Everything above it goes into separate sets, everything below it is merged. There is no true value of τ. We are choosing where to sit on a trade-off, and the honest thing is to know which way we are leaning, and why.

LingPy prints its progress bars in red under WebAssembly, and it also emits a red TqdmMonitorWarning. Both are cosmetic. Nothing has gone wrong.

Precision and recall move in opposite directions. Low τ is conservative: high precision, poor recall, and cognate sets shattered into fragments. High τ is permissive: recall climbs, and precision gives way as unrelated look-alikes are swept in.

Now note where F is highest. It is not at 0.55, the value we clustered at above, but at 0.65. We are apparently leaving performance on the table, and doing so deliberately.

The question that matters, then, is not which τ maximises F. It is which error hurts the tree more. These are different questions, and the second one has a real answer. Exercise 1 works it out, and the reasoning there is worth more than any number in the table.

6. What learning buys, and what it costs

LexStat learns correspondences from the data. SCA does not: it scores sounds with a fixed, universal sound-class matrix, so it knows that /p/ and /b/ are similar in general, but nothing whatever about these languages. Running both at the same threshold is the cleanest way to see what the learning is actually worth.

The overall F-scores are almost identical, about 0.81 for LexStat against about 0.82 for SCA, so on this data the learning has bought nothing at all in aggregate. The picture shows why that summary is worthless.

The picture is one alignment of all thirteen words, with three classifications of it side by side in the row labels: the expert’s (A, B, C…), LexStat’s (1, 2, 3…) and SCA’s (a, b, c…). Rows are in the expert’s order, so each gold class is a contiguous block.

Read the label columns downwards. A method is right exactly where its symbol changes at the same places the gold symbol does. Now change CONCEPT to "two" and draw it again.

Two different things are on display here, and it is worth naming them. The columns of the picture are the alignment: which segment is judged to correspond to which. The letters and numbers beside it are the classification: which words are judged to be cognate. This section holds the alignment fixed and varies only the classification, so that you can watch one thing at a time. Section 7 lets the alignment move as well.

dig is what learning buys. The expert recognises eight separate sets here, and SCA collapses them into four. It puts Ancient Greek skáptō and Greek skávō together with Polish kopać, Russian kopat’, Italian scavare and Spanish cavar, because all of them carry a velar and a labial stop in roughly the right order. It merges Germanic graven, graben, gräva with French creuser. It merges Vedic khan(i)- with Hindi khodanā. Every one of those is a false shared innovation, the kind that puts spurious structure into a tree with complete confidence. LexStat, having learned what Germanic-to-Romance and Slavic-to-Hellenic correspondences actually look like in this data, keeps them apart, and on most runs recovers the concept exactly.

two is what learning costs. Here the expert puts all thirteen languages in a single set, and they really are all reflexes of one Proto-Indo-European word. SCA sees the surface similarity across dýo, twee, deux, zwei, dva, do, and agrees. LexStat breaks the set into roughly five fragments. Exactly which languages end up isolated shifts between runs, because the permutation test is randomised, but the shattering itself is entirely stable. This is the two / zwei problem from §1, now with a number attached: LexStat demands recurrent evidence before it will merge, and across 170 concepts there is not enough of it to bridge a distance that large.

So the two methods are not better and worse. They are differently wrong, and the direction of the error is what matters. SCA over-merges, which fabricates shared history. LexStat over-splits, which discards real history but invents none. Which of those you prefer is Exercise 1, and it is not a question the F-score can answer.

7. Learning changes the alignment too

In §2 we aligned two words by hand with nw_align, using a fixed idea of which sounds resemble which. That is precisely the point at which LexStat’s learned scores are consulted, so once it has learned them, the same two words can come out aligned differently. align_pairs is LexStat’s pairwise aligner, and it takes a method argument, so we can ask for both answers and set them side by side.

bitter is what learning buys. Latin amārus and Spanish amargo are cognate, and the Spanish word carries a velar ɣ with no counterpart in the Latin one. SCA cannot know that. It does the only thing an aligner working from general similarity can do, and matches Latin s against Spanish ɣ, on the grounds that both are consonants standing in about the right place. LexStat leaves the ɣ unmatched and lines Latin -us up with Spanish -o instead, which is the correspondence every Romance handbook opens with. It could only do that because it had counted how Latin endings actually turn up in Spanish across the other 169 concepts. This is what “language-pair-specific” buys you, in one line.

Now set the pair to "three", "Latin", "VedicSanskrit" and run it again.

three is what learning costs. Latin trēs against Vedic tráyaḥ. SCA puts the final s opposite the final ḥ, which is exactly right: the visarga is the regular Vedic reflex of an original final *-s, and it is one of the most reliable correspondences in the family. LexStat pulls the Latin s back onto the Vedic y and leaves the ḥ dangling past the end of the Latin word. Here the evidence was there and it failed to use it.

Notice also that the SCA distance comes out the same on every run, while the LexStat one moves around. SCA’s matrix is fixed; LexStat’s was estimated from a randomised permutation test, so its distance is a measurement, with the wobble that implies. The alignments themselves are steadier than the numbers: both of these came out identical on every run we tried.

Be careful about the general lesson here. These two were chosen to show one error of each kind, and that is not the proportion in which they occur. The two methods disagree about a couple of hundred of the cognate pairs in this wordlist, and among the ones we inspected SCA’s alignment was usually the better one. That fits the F-scores you have already seen, about 0.82 for SCA against 0.81 for LexStat. With 13 languages and 170 concepts there is simply not enough evidence to estimate a good scorer for every language pair, and a learned score based on thin evidence is worse than a sensible universal guess. LexStat earns its keep here on the precision of its clustering, 0.97, not on its alignments. Give it a few hundred languages and the balance changes; that is the finding the literature reports, and it is not one this wordlist can show you.

Exercises

1. Which error does the tree tolerate?

There are two ways to be wrong. Over-splitting breaks one true cognate set into two, which creates a character that no language shares. Over-merging puts unrelated words into one set, which creates a false shared innovation.

Consider what each does downstream, in the terms of Lecture 3. An over-split set contributes a character that is uninformative, since it supports no grouping. An over-merged set contributes a character that actively supports a grouping which never existed, and it will do so with complete confidence.

So which way should τ be set, and does that match where the F-score is highest? It does not, as the sweep above shows. This is why “we tuned τ to maximise F” is a sentence to be suspicious of in a methods section.

2. What does learning buy you?

§6 has the comparison: identical F-scores, opposite failure modes, and dig and two as the two concepts that show it.

Now change CONCEPT in the §6 cell and hunt for more of both kinds. Good places to look for SCA over-merging are short, pull, head and round; for LexStat over-splitting, try name, new, tooth and four.

Two things to consider afterwards. These are 13 Indo-European languages with 170 concepts, which is a lot to learn from. Would you expect learning to help more, or less, for a family whose languages are more distantly related? And for one where only 40 concepts are available?

3. Find a failure, and diagnose it

Pick a concept in §3, look at the gold sets, and compare them with what LexStat produced (lexstatid). Find one that it gets wrong, then classify the error. Be strict about which of the three it is:

a false friend: chance similarity, as in much / mucho
a loan: genuinely related, but horizontally rather than by descent
a true cognate it missed: the sounds have drifted too far apart, as in two / zwei

Only the first is really the algorithm’s fault. The second requires information that the wordlist does not contain. The third is the time-depth ceiling, which no amount of cleverness will remove. Which kind did you find?