Sixty cycles

A tour, with pictures, of the largest machine ever built and the dumber, tougher one buried next to it; in which your correspondent follows a wire out of his kitchen wall, learns why the thing is so hard to kill, watches it cut off its own limbs on purpose, and comes home with a theory about toast.

60.000 Hz

The toaster problem

In which a question is asked about breakfast, the premise of the question is found to be half wrong, and the half that is right turns out to be the interesting half.

Start with the toaster, because the toaster is where the mystery actually lives. You push the lever, a ribbon of nichrome wire turns orange, and at no point in this transaction do you stop to consider that the energy reddening your bread did not exist a fraction of a second ago. It is not stored anywhere. There is no tank of electricity out back. The power you are using right now is being made right now, by a spinning piece of metal that may be four hundred miles away, and the instant you pushed that lever, that piece of metal felt it, the way you feel someone step onto the far end of a dock you happen to be standing on.

This is the sort of fact that, once absorbed, makes it difficult to look at a wall socket the same way. A socket is not a tap on a reservoir. It is a handle on a machine: one machine, continental in scale, with thousands of rotors and some millions of miles of wire, all of it moving in unison, and you are holding one end of it while waiting for breakfast.

Most people come to this subject the way your correspondent did, which is by noticing that the thing doesn't break. Or rather by believing that it doesn't break, which turns out to be different. The news shows you a country being shelled nightly, and behind the reporter the streetlights are on. It shows you a capital city carved up by gangs, and somebody in the frame is scrolling a charged phone. How?

The first answer is that you are being fooled, a little. In Port-au-Prince the public grid delivers a few hours of power on a good day and none on a bad one, and the lights in the background are running on diesel generators, rooftop solar panels, and car batteries: thousands of private workarounds that, seen from a distance, resemble a functioning utility about as much as a flotilla of rowboats resembles a ferry. In Ukraine, more than half of the generating capacity has been blown up at one time or another, and ordinary people organize their lives around a published outage schedule. Four hours on, four off, laundry at three in the morning. These grids are not unbroken. They are very badly broken.

But here is the second answer, and the reason for this essay: they are broken the way a starfish is broken, not the way a wine glass is broken. They lose pieces. They run degraded, for years, on a fraction of their hardware. What they almost never do is the thing you would naively expect a tightly coupled continental machine to do when someone starts shooting at it, which is stop, everywhere, for good. The grid was built by people who assumed, correctly, that something is always going wrong somewhere, and over a hundred and forty years they have turned it into a machine whose chief talent is failing in small, deliberate, survivable ways. To see how, you have to go up the wire.

Up the wirepage 1

Push the lever. The energy toasting this bread did not exist a moment ago.
OK, but from where?
Behind the wall: a gray can on a pole, stepping the street's voltage down for the house.
7,200 volts up here240 down here
Then a substation, humming a slightly flat B. Up to here, the system is a tree.
!hmmmmmmm
Above the substation the shape changes. Lattice towers over the ridgelines, many routes to everywhere, all the way to the thing that makes the power.
Which is this: a couple hundred tons of steel, turning exactly sixty times a second.
3,600 rpmsixty turns a second, exactly
It felt you push the lever, the way you feel someone step onto the far end of a dock.
Felt that.

59.98 Hz

A web, not a tree

In which we climb from the kitchen to the high-voltage network, meet a rule with an ugly name, and learn what happened when somebody spent nineteen minutes shooting at a substation.

Behind the socket is a breaker panel. Behind that, a gray can bolted to a pole, which is a transformer taking the seven thousand or so volts running down your street and stepping it down to the 240 your dryer wants. And behind that, a mile or ten away, is a substation: a gravel yard full of gray-painted equipment behind chain link, humming at a pitch that a musician would identify as a slightly flat B.

Up to this point the system is a tree. Your house is a leaf, your street a twig, the feeder a branch. Trees are cheap, and trees are fragile: cut a branch and everything past the cut dies, which is why a squirrel with poor judgment can black out your neighborhood but not your state.

Above the substation the architecture changes, and this is the first secret. The high-voltage network, the lattice towers marching over ridgelines at 115 or 345 or 765 thousand volts, is not a tree. It is a web. Any two points on it are joined by many paths, and electricity, which is not so much routed as poured, divides itself among all of them at once according to the laws of physics and nobody's instructions. Cut one strand and the flow rearranges itself across the others at very nearly the speed of light, before any human being knows there is a problem to have an opinion about.

The people who run the web hold themselves to a rule with the unlovely name of N-minus-one. It says that at every moment the system must be able to lose its single most important component (the largest power plant, the most heavily loaded line, whichever would hurt most) and carry on, with nothing overloaded and no customer the wiser. Not recover from the loss. Carry on through it. Which means the grid is at all times being driven with the spare tire already mounted, and that the job of a control-room operator consists largely of sitting in front of a wall-sized map while a computer simulates, every few minutes, a long list of catastrophes that have not happened, and reports whether any of them would be embarrassing. When the answer is yes, the operators shuffle power around until the answer is no. And on the day the worst thing really does happen, a clock starts: they have about half an hour to get the system back into a condition where it could survive the next worst thing.

In April of 2013, persons unknown crept up to the Metcalf substation outside San Jose, California, cut the phone cables in a nearby vault, and spent nineteen minutes methodically putting rifle rounds through the cooling fins of seventeen large transformers, which bled out some fifty thousand gallons of oil and cooked themselves off line. It was about as thorough a physical attack as the American grid has suffered. The number of customers in Silicon Valley who lost power was zero. Operators routed around the hole, asked people to go easy on the air conditioning, and spent a month and fifteen million dollars on repairs. The shooters were never caught. The web did not much care.

A web, not a treepage 2

A tree is cheap and fragile. Cut a branch and the leaves beyond it die.
everything past the cut goes dark
A web has many paths. Cut a strand, and power pours around the hole before any human knows.
The N-minus-one rule: at every moment, be able to lose your single largest piece and carry on.
What if we lostour biggest plantright now?You'd be fine. Ask meagain in five minutes.
San Jose, 2013. Nineteen minutes of rifle fire; seventeen transformers bled dry.
the city never noticed

Customers blacked out: zero.

59.94 Hz

The spin

In which a power plant dies at 2:14 in the morning, and several thousand machines that have never met come to its aid without being asked.

Follow the towers far enough and you arrive at the thing making the electricity, which is, more often than not, a rotor: a forged steel shaft weighing a couple of hundred tons, turning sixty times a second. Not roughly sixty. Sixty. And here is the second secret, the one I find hard to stop thinking about: every other rotor on the network is turning at the same speed, in step, locked together through the wires by electromagnetic forces as firmly as if they were all bolted to a common axle. A steam turbine in Georgia and a gas turbine in Ontario are, in every sense that matters, parts of the same machine. (A hydroelectric generator has more magnetic poles and turns more slowly, the way a big gear turns more slowly than the small one it meshes with, but it is meshed all the same.)

The eastern two-thirds of North America is one such machine. The West is another. Texas, being Texas, has its own. So does Québec, which is why power coming south into New England has to be converted to direct current and back again, at places like Highgate, Vermont, in the manner of a train changing wheels at a border where the track gauge changes.

Now kill a big plant. It is 2:14 in the morning, a feedwater pump fails, and a thousand megawatts vanish: a million toasters' worth of demand that is suddenly being served by nobody. What happens in the first second?

Nothing that anyone decides. The energy comes out of the spin. Every rotor on the interconnection is a flywheel, and the missing power is drawn, instantly and automatically, from the momentum of all those thousands of tons of whirling steel, each of which slows by a hair. Sixty cycles a second becomes 59.97, then 59.94. The system is quite literally coasting, burning its own inertia to cover the gap, and it can keep that up for a few seconds.

But the slowing is also a message, and as a piece of engineering it is the most elegant thing I know of. Because every rotor is locked to every other, the frequency is the same everywhere, and so every power plant on the continent learns at the same moment, from the speed of its own shaft, that the system is short of power. There is no alarm, no phone call, no network packet to be lost or forged. Each machine has a governor, a direct descendant of the whirling brass weights on James Watt's steam engines, that feels the shaft slowing and opens the throttle a little, in proportion. Thousands of plants each contribute a sliver. Within ten or fifteen seconds the fall is arrested; within a few minutes a slower, centralized system hands out new assignments and the frequency climbs back to sixty. In the control room, a line on a chart dips and recovers. At no point did anyone have to be brave.

If you have spent time around distributed computer systems you will recognize how enviable this is. It is a consensus protocol with no messages. The shared state is a physical quantity, available to every participant for free, impossible to spoof, with latency bounded by the speed of light. We have been trying to build things this robust out of software for fifty years.

The spinpage 3

2:14 a.m. A pump fails, and a thousand megawatts, a million toasters' worth, is suddenly nobody's job.
trip!2:14 a.m.
They all slow together, because they are all one machine. And the slowing is the message.
60.0059.960.159.94hertz, everywhere at once
First second: nobody decides anything. The missing power comes out of the spin.
GeorgiaOntarioOhioMaineFloridagoneeach one slows by a hair, and pays the difference out of its own momentum
Every plant has a governor, a descendant of the whirling brass weights on James Watt's engines. No phone call required.
shaft slows,weights drop,valve openssteam
Ten seconds later the fall is arrested. A few minutes later: sixty again. Nobody had to be brave.
60.0059.94about ten secondsThere it goes…and back.

59.30 Hz

The amputation

In which the machine, finding itself in real trouble, begins cutting off its own limbs, and we are obliged to revise our definition of failure.

Suppose the hole is too big. Not one plant but six, or a volley of cruise missiles through the switchyards. The governors open all the way and it isn't enough; the frequency keeps sinking. This is dangerous in a way that is not obvious. A big steam turbine is tuned, like a bell, to run at one speed, and a few percent below that speed its longest blades begin to resonate and work themselves toward destruction. So each plant carries a relay that, below a certain frequency, will disconnect it from the grid to save it. Which removes more generation. Which drops the frequency further. Which trips more plants. This is the cascade, the grid's one truly catastrophic failure mode, and once it gets going it can black out fifty million people in a matter of minutes, as it did across the Northeast in August of 2003, beginning with some sagging power lines brushing trees in Ohio and a control-room alarm system that had quietly crashed without telling anyone.

The defense is brutal and simple. Scattered through the substations are relays, preset and unattended, watching the frequency. At a threshold (59.3 hertz is typical in North America) the first tier of them opens breakers and drops a preassigned slice of customers, five or ten percent of the load, with no warning and no human involved. If the frequency keeps falling, a second tier drops more. Then a third. The machine amputates until what's left of demand fits what's left of supply, and the fall stops.

This is where the definition of failure needs adjusting. When you are sitting in the dark because your feeder was on the sacrificial list, it feels as though the grid has failed. It hasn't. It has done exactly what it was designed to do, which is to trade your evening for the survival of the whole, on the reasoning that a neighborhood that has been switched off can be switched back on in an hour, while an interconnection that has collapsed may take days or weeks to coax back to life. A rolling blackout is the same trade made slowly and by hand: operators rotating the pain around the map on a schedule, so that everyone has power sometimes rather than no one having it ever.

That is what you are looking at in Ukraine. Not a grid that is mysteriously unbroken, but a grid being run, with great skill, at the edge of what its remaining hardware can carry, by dispatchers who decide every hour who goes dark so that the frequency holds. It helps that they began with advantages. The Soviet Union built that network to feed steel mills and tank factories that no longer exist, so it had far more capacity than modern Ukraine needed. About half its electricity came from nuclear plants, which even this war has mostly declined to hit directly. And then there is a matter of timing that any competent novelist's editor would strike as too convenient: in the small hours of 24 February 2022, Ukraine disconnected its grid from Russia's and Belarus's for a long-planned, three-day test of whether it could run alone, a rehearsal for someday joining the European system. The invasion began a few hours later. They never reconnected. They ran as an island, under bombardment, for three weeks, and on 16 March synchronized with continental Europe about a year ahead of schedule, so that today a rotor in Portugal and a rotor in western Ukraine turn in step.

The Russians, for their part, learned which organ to aim for. Wires and towers are cheap, and can be restrung in a day by crews who have become, by grim necessity, the best in the world at it. The thing that cannot be replaced quickly is the big substation transformer: a custom-built object weighing a few hundred tons, with a lead time measured in years even in peacetime, and in Ukraine's case built to Soviet voltage standards, 750 and 330 kilovolts, that hardly anyone else uses. Those are the true currency of the campaign. Every one destroyed is a strand of the web that stays cut.

The amputationpage 4

Now make the hole too big. Six plants, or a volley of missiles. Throttles wide open, and the frequency keeps sinking.
59.61
Run a turbine too slowly and it shakes itself apart, so each plant will disconnect to save itself. Which makes things worse. This is the cascade.
each plant that saves itself dooms the next
The defense: unattended relays, set in advance. At 59.3 hertz the first tier drops a slice of the city. No warning. No human.
59.30click.
In Ukraine the same trade is made by hand, on a schedule taped to the fridge. Everyone has power sometimes, so that no one has it never.
06 – 10 on10 – 14 off14 – 18 on18 – 22 off22 – 02 on
The machine amputates until what's left of demand fits what's left of supply.

The dark blocks are not the failure. They are the fix.

0.00 Hz

Dark start

In which the worst happens anyway, and we find out how to start a machine that needs electricity in order to make electricity.

Sometimes all the defenses fail, and a grid goes to zero: Italy in 2003, Venezuela in 2019, Spain and Portugal on an April afternoon in 2025. Texas came within about four and a half minutes of it in February 2021, when the frequency sagged to 59.3 and hung there while operators shed load by hand as fast as they could, knowing that if it stayed that low for nine minutes the remaining plants would begin to disconnect themselves, and that putting the state back together afterward might be a matter of weeks, in a hard freeze.

The reason it takes so long is an awkward fact about power plants, which is that they need power. A big thermal plant has pumps, fans, control systems, and lubricating-oil systems that eat a noticeable fraction of its own output, and it cannot start without them. A dead grid is therefore a bootstrapping problem, of just the kind familiar to anyone who has ever wondered how a computer loads the program that loads programs.

The answer is called black start, and every grid keeps a few plants designated for it. Usually they are hydroelectric dams, since all a dam really needs is for someone to open a gate, which can be done with a diesel generator the size of a van, or a bank of batteries, or in extremis a hand crank. The dam comes up and energizes a single line chosen years in advance, the cranking path, to the next plant, whose pumps spin up on the borrowed power. That plant starts. Now there are two. The operators add a little load, a town, to give the generators something to push against, because a generator with too little load is nearly as twitchy as one with too much. Then another plant. Then another town. It is slow, fussy work, done out of binders, and it produces not one grid but several small ones: islands of light, growing outward through the dark.

Joining two islands is the delicate bit. Each is turning at very slightly its own speed, and if you close a breaker between them when they are out of step, electromagnetic forces will try to yank a few hundred tons of rotor into alignment in about a hundredth of a second, with results that resemble dropping the clutch at redline. So there is an instrument called a synchroscope, a dial with a single needle that rotates at the difference between the two frequencies. You nudge one island's speed until the needle is creeping slowly clockwise, and as it comes up on twelve o'clock you close the breaker, and two machines become one.

Dark startpage 5

Sometimes every defense fails. Zero hertz. And here is the awkward part: a power plant needs power to start.
0.00 Hz
So every grid keeps a few black-start plants. Usually dams. A diesel the size of a van is enough to open a gate.
dieselOpen the gate.Water does the rest.
One line, chosen years in advance, carries borrowed power to the next plant. Then a town. Then another plant. Islands of light, growing outward.
1. the dam2. the cranking path3. the next plant4. a town, to push against
Joining two islands: each turns at very slightly its own speed. Watch the synchroscope creep toward twelve o'clock…
slowfastwait for it…
…and close the breaker. Get it wrong and it's like dropping the clutch at redline, with a two-hundred-ton flywheel.
60.00 Hz, one machine

62 psi

Uphill

In which we leave electricity for an older and dumber network, which turns out to be tougher, and also to be lying to us.

Water, by comparison, is a relaxed business, for a reason that fits in one sentence: you can store water, and you can't (much, yet) store electricity. The grid has to balance supply and demand every instant. A water system has to balance them every day or so, and it holds the difference in the most reliable battery ever devised, which is a tank on a hill.

That is what a water tower is. It isn't there to hold the town's water; it holds perhaps a day's worth. It is there to hold the town's pressure. Every foot of elevation is worth 0.43 pounds per square inch at the tap, so a tank 140 feet up gives you 60 psi, indefinitely, with no moving parts. The pumps run at night, when electricity is cheap, and push water uphill, and gravity does the rest. If the power goes out the pumps stop and the taps run anyway, for hours or days, because a hill does not require a power supply. The pipes are buried, which protects them from weather, traffic, and most ordnance. And the whole apparatus tolerates a degree of abuse that would be unthinkable on the electrical side: plenty of large cities lose a third of their water through leaks before it ever reaches a meter, every day, for decades, and carry on.

But water has a nasty property that electricity lacks. Electricity fails honestly: the light is on or it is off. Water fails quietly. As a system degrades, quality goes first and quantity goes last, and the reason is pressure. A pressurized pipe leaks outward, which is wasteful but clean. Let the pressure drop, because the pumps have been off too long or the tanks have drained or the mains are broken, and the same cracks leak inward, drawing in whatever the pipe happens to be lying in, which in a city is frequently sewage. This is why American regulators treat 20 psi as the line below which you get told to boil your water. The tap still runs. The water looks fine. It is now the most dangerous object in the house.

That is what the failure of a water system looks like in Haiti: not dry taps, but cholera, a disease of contaminated water that arrived in 2010, killed something like ten thousand people, was beaten back nearly to nothing, and returned in 2022 as fuel for pumps and treatment plants ran short. It is also why the two networks in this essay are not really independent. Pumping and treatment run on electricity. A one-hour grid outage is nothing to a water system. A one-week outage is the beginning of a public health emergency. The tank on the hill is a battery, and batteries run down.

Uphillpage 6

A water tower doesn't hold the town's water. It holds the town's pressure. Pumps fill it at night; gravity works the day shift.
pumps run at night
Power out? The taps run anyway, for hours or days. A hill does not need electricity.
140 feet up= 60 psi at the tap,no moving parts
Pressurized pipes leak outward. Plenty of cities lose a third of their water this way and carry on for decades.
60 psi
Let the pressure fall and the same cracks leak inward, drawing in whatever the pipe is lying in.
18 psisewer
Electricity fails honestly: the light is on or it is off. Water fails quietly. The tap still runs. Quality goes first, quantity last.
Looks fine to me.

60.000 Hz

A theory about toast

In which your correspondent attempts a summary, tours the basement where the real vulnerabilities are kept, and returns to the kitchen.

So why is the thing so hard to kill? Boil the tour down and I count four ideas, none of them peculiar to electricity.

The first is redundancy where failure is expensive and thrift where it isn't. The core is a web, run at all times with a spare already in hand; the edges are cheap trees, because losing a leaf is tolerable. The second is a signal that cannot be cut. Frequency on the grid, pressure in the pipes: each tells every part of the system how the whole is doing, locally, for free, with no central brain to decapitate. The third is planned amputation. The system decided long ago what it would sacrifice and in what order, and it does so automatically, quickly, and reversibly, preferring many small honest failures to a single large one. And the fourth is that the parts are simple, standardized, and fixable, and that there exist people who fix them.

None of which makes it invulnerable, and it would be a poor tour that skipped the basement. There are the big transformers, already discussed. There is the sun: in March of 1989 a geomagnetic storm took down the whole of Québec's grid in about ninety seconds. There is software: the 2003 blackout owed a great deal to an alarm bug, and in December 2015 Ukraine became the first country to have part of its grid switched off by hackers. (It was back within hours, in part because its old-fashioned breakers could be closed by hand, by people who got in trucks and drove to the substations.) And there is the newest worry, which is inertia itself. Solar panels and wind turbines reach the grid through power electronics and bring no heavy spinning mass with them. A grid with fewer big rotors falls faster when something trips, which leaves less time for everything described above to work. Engineers are teaching inverters to imitate the old behavior, and installing enormous flywheels that generate nothing and exist only to spin. It looks solvable. But it means that the grid's oldest safety feature, which for a century came free with the way we made power, now has to be built on purpose.

The last item on the list is not hardware. After a hurricane, convoys of bucket trucks roll in from six states away, under agreements that utilities signed with each other decades ago. In Kharkiv and Odesa, crews go out after each strike knowing that the second wave is sometimes aimed at them. Every society, in whatever shape it happens to be in, fixes the power and the water first, with whatever it has. The final layer of redundancy wears a hard hat.

The toaster pops. Four hundred miles away, a couple of hundred tons of steel, relieved of the burden of your breakfast, speeds up by an amount too small to measure, and a few thousand others speed up with it.

The basement, and the peoplepage 7

The part that can't be fixed in a day: the big transformer. Wires are cheap. These are the true currency of a war on a grid.
a few hundred tonsbuilt to orderyears to replace(person,for scale)
The last layer of redundancy wears a hard hat and drives toward the storm.
The toast pops. Four hundred miles away a rotor, relieved of your breakfast, speeds up by an amount too small to measure.
Thanks, Ontario.