TangoTube
From the desk

Clips are here

Create short practice clips from any video. Loop movements, tag techniques, and build your personal library.

Try Clips
Engineering notesVolume 7 of 8 · Running it
Volume 7

One Server

Four and a half million videos, a 37-gigabyte database, eight search indexes, twenty nightly jobs, and a matcher that reads every title, on one rented machine. This is how it runs, deploys, recovers, tests itself, and knows when it is lying, with no platform team.

machine
1machine
boxes on it
4boxes on it
of data
37 GBof data
nightly jobs
20nightly jobs
test workers
8test workers

Part 01

Four boxes on one machine

one rented machinethe websiteanswers visitorsthe page painterfirst paint has contentthe night workerimports, harvests, mailthe readermatches titles, alonethe database37 GB · the job list toothe search indexa fast copya newversionhealthyA deploy starts the new box beside the old, waits until it answers, switches traffic, then stops the old.Built only from committed work, so an unsaved change can never ship. Old tabs keep working.Every non-obvious setting has its cost beside it: “this delay was seven seconds and bought nothing.”
Four boxesWebsite, page painter, night worker, and one box that does nothing but read titles.
Two storesThe database is the truth and the job list. The search index is a fast copy.
Deploy by swapNew beside old, health check, switch, stop. Nobody sees it happen.

The fashionable way to run this is a fleet of services with a team to keep them alive. That solves problems this system does not have: traffic is modest, data is large but slow-moving, the hard work is nightly. So the complexity budget goes into the data, where the value is, not the infrastructure.

Part 02

Lanes, not services

the job listlives in the databaselane: reading titles · heavy thinkinglane: measurements · must never waitlane: everything elsethe reader boxone core, all to itselfthe night worker boxone thread reserved for measurementsRuby gives one box one core.So reading titles, the heaviest job, gets a box of its own and can never starve the imports.Need more? Add a box, not a service. There is no message bus; there is a column that says which lane.
Jobs in the databaseNo separate queue to keep alive. A job and the change it makes are saved together.
Three lanesHeavy thinking, measurements, and everything else. A lane cannot block another.
One core eachThe reader gets its own box because it needs the whole core. Concurrency is boxes and lanes.

Twenty nightly jobs are scheduled in the same list, each with a unique key, so two boxes holding the schedule cannot run the same job twice. Measurement jobs give up rather than retry when the database is busy, because a statistic is never worth a slow page.

Part 03

A night's work, in short steps that can be dropped and picked up

02:00still on YouTube?03:30rebuild search05:00ask the channels05:45find partners06:00redraw couples06:15draw the map07:15read descriptions07:30read music panelseach job finds what the one before it madeInside every job: the same short walk.Read 500 rows past a bookmark. Do the work. Hand the bookmark to the next step. Stop on an empty page.Killable at any second, resumable from the bookmark, and always scoped to the 116,404 videos anyone will see, never the 4.5 million.
In orderChannels before partners, partners before couples, couples before the map.
Short walksNo job runs for hours. Each is a chain of small steps passing a bookmark.
The right scopeOver the browsable set a walk finishes before breakfast. Over everything it is a six-year job.

The bookmark is a row id, not a page number, so a walk that changes the rows it is reading never skips any. A crash at three in the morning loses one step of five hundred rows, and the next run resumes from the bookmark. By eight the index is truer than at midnight, and the heartbeat says by how much.

Part 04

Work on real data without touching production

productiononly ever readsecure copynames, emails: blacked outa laptop123k · 968k · or all 4.5M videoscatalogs always wholethen the testseight at once, each with its own database and indexthe mistake bookpast mistakes, with the right answerevery change must pass itNever measure the matcher on a sample of its own output. A small checked-in sample serves interface work only.
Read-only mirrorProduction is copied, never written to. People's details are scrubbed on the way.
Eight workersThe suite runs in parallel; the inner loop is one file and takes seconds.
The mistake bookEvery past production mistake is a test. A rule that repeats one cannot ship.

Tests name what a person gets, not what a function returns. The suite is measured before it is changed, and the goal is tests per second, not test count. Ninety-one world champions and eleven real search queries live in the mistake book alongside every case a reviewer pinned at the desk in Volume 5.

Part 05

Know when it is lying

every 10 minthe notebooksensorsa recording dated 2031? · 0the same person credited twice? · 0new dancers minted today · 74two confident sources disagreeing · 885a work pileclick the light, do the worksaved every day“did it get truer?”now has a historyA catalog that is wrong is worse than one that is down, because nobody notices.A lit sensor is a regression with a date. A dark one is a class of mistake that stopped arriving.A quiet day, with nothing applied and nothing new wrong, is a good day.
Watch the dataNot only the servers. Sensors read the notebook for things that cannot be true.
Lights are doorsEach lit sensor opens the pile of rows that lit it. Nothing is a number with no verb.
A historyThe daily answer is kept, so “truer than yesterday” can be shown, not felt.

Crashes go to an error tracker with an allow-list so a known noisy error cannot drown a new one. And because every nightly step can be dropped and picked up, the fix for most three-in-the-morning problems is to let the next step run.

Take-aways, without the tango

  1. 01Spend the complexity budget on the data, not the infrastructure. One machine and a boring stack are a feature.
  2. 02Concurrency is boxes and lanes. Add a box before you add a service. Keep the job list in the database you already have.
  3. 03Every long job is a chain of short steps passing a bookmark. Killable any second, resumable, and scoped to what anyone will see.
  4. 04Measure against real data mirrored read-only, never against a sample of your own output. Turn every mistake into a test.
  5. 05Watch the data, not just the process, and keep the daily answer to “did it get truer?”

The Cabeceo

We've caught your eye across the milonga. Like any good tango, this starts with consent — nothing optional runs until you say so. What we store