TangoTube
From the desk

Clips are here

Create short practice clips from any video. Loop movements, tag techniques, and build your personal library.

Try Clips
Engineering notesVolume 1 of 8 · The pipeline
Volume 1

How TangoTube Knows

TangoTube holds four and a half million YouTube videos, collected from channels that post tango. Most of them are not tango performances at all, and YouTube cannot tell which are. This is how one application decides what is worth showing, and puts names on it, without ever guessing.

videos in the database
4,566,186videos in the database
shown to visitors
116,404shown to visitors
channels admitted by a person
2,947channels admitted by a person
entries in the notebook
299,869entries in the notebook
guesses
0guesses

Part 01

Most of it is not a tango performance

4,566,186 videos, from channels that post tangoa performancea classan interviewa folklore nighta concert, nobody dancinga festival announcementWhich are the performances?Nothing on the way in says. The title is the only evidence there is.A wrong answer is not a blank page. It is someone’s name on a dance they were not in.
A channel films everythingFollow one that posts tango and you get its classes, interviews and concerts too.
No label arrives with itYouTube knows these are videos. It does not know which are a dance.
And guessing costsAt 400,000 new videos a month, a rule that is wrong 2% of the time invents 8,000 falsehoods a month.

Strip the tango away and this is an entity resolution problem: unstructured text written by thousands of strangers, a catalogue of real people and real recordings to match it against, and a hard requirement that every match be explainable in one sentence. The rest of this essay is the five things that make that affordable.

Part 02

Decide who is allowed to speak

138,923 channels encountereda person looks once2,947 admittedasked every night for new uploadseverything else is never asked, and never arrives3,000 decisions cover 4.5 million videos.Curating videos one at a time would be 4.5 million decisions. This is the trade that makes the rest affordable.
The cheapest doorTangoTube never searches YouTube for videos. It follows channels a person has admitted.
Then a set differenceEach night: what the channel lists, minus what we hold, is new. The rest is the whole algorithm.
One judgement, thousands of rowsPut the human decision where it is cheapest and covers the most.

An import then writes down everything YouTube said, including the parts nothing can read yet. A music panel captured on every import sat unread for two years; when someone finally wrote the reader, 20,435 videos already held their own answer. So re-imports merge into that record and never replace it. Evidence you cannot use yet is still evidence, and disk is cheaper than a second chance.

Part 03

Read the title with rules a person could check

the title, as the uploader typed itYanina Quiñones and Neri Piliu – La milonga de Buenos Airesevery rule carries a grade: how much of a name it had to seea whole name in the title0.95a curated alias0.90a near spelling0.75a lone nickname0.45found only in the description−0.15the line: assert above, ask below0.950.900.750.45A lone nickname is the only rule under the line, deliberately: one token in a title names whoever shares it.It once gave one dancer 1,237 credits that belonged to three other men with the same first name.
A title has a grammarTwo names, a song, a festival, a year. Four matchers read it against the catalogue.
Every rule is gradedAnd the grade travels with the credit, so precision can be measured per rule.
Below the line, askThe weakest rule is written down as a question, never asserted as a fact.

The line sits at 0.50, and it was chosen by measurement rather than taste. Running the real matcher over 600 random videos and 1,417 credits showed what each candidate line would cost: at 0.50, 6% of credits are withheld and two videos lose every credit they had; the next rung up costs ten times as many videos for the same kind of doubt. That table is written in the code beside the number.

Part 04

No single source is allowed to be right on its own

what is playing in this video?the title says: “La milonga de Buenos Aires”the stored panel says: the samefive linesverifiedeach confirms the otherone confident source, aloneapplied, but unverified — a second voice can still supersede itone unsure source, alonewritten down as a question for a persona person already saiduntouched. No machine overwrites a human answer.
Sources disagree constantlyA title, a stored panel, a fingerprint, a person. All sound equally sure.
Agreement makes factsNot confidence. Two independent sources that agree verify each other.
Doubt is written downA weak source becomes a question, not a silent guess and not a discarded clue.

This rule came from a measurement. Audio fingerprinting, the technology behind Shazam, was 85% precise on videos nothing else had named, which sounds like a success. Then we looked only at the conflicts, where it disagreed with the title: it was right 32% of the time. A source that loses two conflicts in three cannot be allowed to overwrite anything, however confident it sounds. That number is why the judge is five lines long and has not changed since.

Part 05

Write down the evidence, and hand the rest to a person

the notebook299,869 entries: who said it, how sure,which version of the rule, what became of itnever edited, only added toappliedproposedconflicts · 885panel leftovermatcher missnot tango?piles, by reason, not one queue by agedecides oncepinnedas a testthe tests run on every change, so a fixed mistake cannot come back122,059 questions are waiting. That pile is the honest size of what the machine does not know.
The answer is a projectionThe fast column on the video is a cache of the notebook, not the truth.
Cluster by reasonA queue sorted by age is a landfill. Piles sorted by impact are work.
Decisions outlive the sessionEvery human answer becomes a test case, so the next person does not rediscover it.

Because the notebook keeps everything, the system is correctable rather than merely accurate. When one rule turned out to be wrong about half the time, nobody had to guess which credits it had made: the notebook said 111,804 of them, on exactly which videos, and they were routed to review the same afternoon. An answer without its evidence is a liability. An answer with its evidence is a dataset.

Take-aways, without the tango

  1. 01Put the human decision where it is cheapest and covers the most ground.
  2. 02Store what the source said, raw, and merge on re-read. You will need it later.
  3. 03Prefer rules a person can check to models a person cannot, and grade every rule.
  4. 04Choose thresholds by measuring their cost, and print the measurement beside the constant.
  5. 05Trust comes from independent agreement, never from a single confident voice.
  6. 06Keep an append-only record of evidence, so every answer is a dataset and every mistake is traceable.
  7. 07Make “I do not know” a first-class output, route it to a person, and turn their answer into a test.

The Cabeceo

We've caught your eye across the milonga. Like any good tango, this starts with consent — nothing optional runs until you say so. What we store