When most of Fermilab went dark Wednesday, non-essential support staff went home, but many engineers, technicians, and physicists picked up flashlights and buckled down for work.
Instead of their normal jobs--running the cutting-edge machinery in an international race to discovery--the men and women were switching to the role of a high-tech pit crew: inspecting and returning to life the highest-energy particle accelerator currently operating.
At 2:30 p.m. Wednesday, a power arc at a power line on site had brought Fermilab’s accelerator complex and its key component the Tevatron--a 4-mile particle raceway--to a standstill. Two nearly 6000-ton particle detectors reading data from an average of 2.5 million proton-antiproton collisions per second ground to a halt as the engine powering bunches of particles racing toward them silenced.
The outage wasn’t a monumental problem. Outages of up to a few seconds occur several times a year. Outages of up to 10 minutes occur occasionally. They cause little concern because the temperature of the superconducting magnets in the Tevatron particle collider rises slowly.
But an outage of an hour, which occurs only once every one or two years, creates some challenges. Such outages push the system past a temperature threshold that usually requires a day or two of downtime. It’s a speed bump, but it’s also a chance to test how well the divisions of the laboratory work together.
First steps to recovery
Fermilab employees took on their repair roles like a well-oiled machine.
"When the lights went out,” said Bob Mau, head of Accelerator Division Operations, “within minutes, 40 people from all divisions were in the Control Room to find out what was going on and talking about what to do.”
The Accelerator Division’s Controls Department shut down non-essential equipment to conserve backup power, to avoid overheating the now non-air conditioned rooms and to keep key control room systems operating after the power outage.
In the event of an outage, the beams racing through the accelerator complex are aborted to prevent damage to accelerator components.
The helium refrigeration system that cools the magnets to make their coils superconducting shuts down. The helium in the magnet’s cryogenic system warms and some of it escapes as it turns from liquid to gas.
Getting a giant race back on track
To restore lab-wide electricity, Fermilab crews rerouted power from the downed main power station on site to the Kautz Road substation within an hour of the power outage. But that didn’t mean the Tevatron could be turned back on as easily as flipping a light switch.
First, operating systems for the United States’ last large atom smasher have to come back online: computers for use by the control room and cryogenics team, the cryogenic system, the vacuum system, the water must be restored. Then the beam can be turned back on. Beam returned to the neutrino experiments, the test beam area, and neutron therapy by Monday. The Tevatron could return to operation along with the CDF and DZero detectors as early as this evening.
"We have got a lot of dedicated people working a lot of hours right now,” Accelerator Division head Roger Dixon, told a standing-room only crowd of supervisors, technicians, and physicists working to coordinate efforts. "I know they are willing to work until the job is done but it is more important to not to let them work beyond their limits. Getting the job done safely is more important than getting the machine up early."
What happened?
An anchor-shaped bracket connects the electric lines to the pi-shaped power poles north of Wilson Hall. A metal pin on one of the brackets became loose, causing a static line that serves as a lightening rod to fall and hit the insulator and power lines. The ensuing arc of electricity triggered a shut down of the master power station. Power to everywhere except the Village and Main Injector vanished.
Crews from Fermilab, Commonwealth Edison, and the city of Batavia, which maintains the lines, inspected all the poles Wednesday night, finding two other loose pins. Experts say the wind combined with either a faulty spring, oversized pinhole, or improper installation likely caused the failures.
"It was like a football game by the poles Wednesday,” said site engineer Prem Mattappally about the large number of bucket and lighting trucks and inspectors. ComEd used a helicopter to further inspect the area Thursday morning.
As soon as power to the site was stabilized, operations people from several divisions and sections huddled together to report on their respective tire kicking of their parts of the laboratory’s machinery. People were assigned repair jobs. Shifts were set up to allow everyone a chance t
o get into the cramped tunnels and detector halls to tinker under the hoods of the complex systems.
Some items were damaged, but so far, everything has been easily fixable with spare parts on site. Extended outages always cause some damage and the laboratory plans for that.
Sixty percent of the helium inventory was lost but enough remained to refill the Tevatron for operation before reordering more. A water pipe split, a couple leaks formed, a couple computer hard drives were lost, and a turbine sustained damage--all anticipated occurrences.
Rather than waiting for ComEd to finish repairs to the master power station Friday morning, the DZero and CDF collaborations used stored program runs Wednesday to test their detectors’ parts and slowly applied power to sections of their detectors to make sure they could withstand a full startup scheduled for Monday. The MiniBooNE neutrino experiment performed similar checks. So far, there appears to be no damage.
By Thursday morning, the Booster accelerator had cooling water circulating and the ion pumps were back on. The collider detectors were under inspection, and the chiller was restarted for the NuMI target hall.
At the Tevatron, kicker magnets and beam separators were being conditioned, the vacuum system restored and back ups for the computer systems recharged.
Friday started the complicated process of slowly cooling down the Tevatron to minus 450 degrees Fahrenheit—close to absolute zero. It typically takes a minimum of 24 to 48 hours to purify the helium system and then restore the cryogenic system.
Like a fine-tuned race car engine, you can’t just throw the throttle on the Tevatron without warming it up, or in this case cooling it down, but with patience, skill, and dedicated employees you can get even town-sized raceways and three-story particle detectors humming again.
