A joint Fermilab/SLAC publication

24/7: Labs that never sleep


Here they measure the time not in minutes or hours. Instead they think in terms of how many antiprotons are ready to stack and how soon the Tevatron will be ready to accept new beam. Or how fast they need to fix something, any time of the day or night. Or how long they can stay awake.

The scenes are played out around the clock and around the world in control rooms at CERN in Switzerland; at DESY in Germany; at KEK in Japan; at SLAC in California; and accelerators at many other research institutions. These 24 hours at Fermilab provide a glimpse of the work taking place at these labs.

24/7: Labs that never sleep
By Dawn Stanton


Geneva, Switzerland
Tsukuba, Japan
Menlo Park, California, USA
Batavia, Illinois, USA
Hamburg, Germany

Fermilab at night: view from the 15th floor of Wilson Hall.
Photos: Reidar Hahn, Fermilab

Here they measure the time not in minutes or hours. Instead they think in terms of how many antiprotons are ready to stack and how soon the Tevatron will be ready to accept new beam. Or how fast they need to fix something, any time of the day or night. Or how long they can stay awake.

The scenes are played out around the clock and around the world in control rooms at CERN in Switzerland; at DESY in Germany; at KEK in Japan; at SLAC in California; and accelerators at many other research institutions. These 24 hours at Fermilab provide a glimpse of the work taking place at these labs.

High-energy particle physics is a lifestyle where the mundane and the marvelous meet. When everything’s working well, particles and antiparticles accelerate to near the speed of light and make collisions that produce spectacular energy events. Discoveries that revolutionize human understanding of the subatomic world can happen, as discovery of the top and bottom quarks did, in 1995 and 1977, respectively.

When people and machines run 24 hours a day, seven days a week, a lot of the mundane is needed to produce the marvelous. In the Main Control Room for Fermilab’s Tevatron, no one can exactly predict when new beams of protons and antiprotons will be ready for the next "shot setup" in the four-mile ring. Day or night, operators wait out the tedium of sitting in front of several assigned computer consoles, awaiting the moment to activate and make a contribution. To keep alert, many drink cola, tea, or coffee–anything with caffeine. When their moment does come, they need to be ready.


Main Control Room
Around the clock, seven days a week, operators monitor the Fermilab accelerator complex. During shot setup, the room has the noise level of a cocktail party.

Stashes, stores and shots

It’s Monday morning, 9 a.m., the week before a scheduled 3-month maintenance shutdown for the Tevatron. Researchers are anxious to gather data before operations cease. If something breaks before then, the Tevatron might even shut down early.

After attending the usual Monday morning debriefing session, Main Control Room crew chief Marty Murphy returns to the Main Control Room. By 11:30 a.m., it’s clear to Murphy that the next shot setup to the Tevatron won’t be ready for another hour. A "shot setup" means transferring the particle beams from one machine to another. All around Murphy, groups of people clump together, talk, and disband. The room has the noise level of a cocktail party with a background accompaniment of a steady beep, which rises or falls in pitch based on beam intensity. Like the monitor in a hospital room that informs doctors of a patient’s heartbeat, this beep alerts operators of their "patient’s" condition. An intense beam is a healthy beam.

One of the operators this shift, Aria Meyhoefer monitors antiproton beam in the Recycler storage ring. Her job is to cool the beam, which means lessening the "jiggling" of antiprotons so she can pack more into a single bunch. Cool the beam too early before its launch into the Tevatron, and you’ll shorten the life of the antiprotons. Cool the beam too late, and the proton beam will pass through the loosely-packed antiparticles without making collisions.

At consoles across the room, operators make corrections to keep the antiproton beam stable during its vulnerable transfer from the Accumulator, the machine that collects antiprotons, to the Recycler. They could lose the entire beam with one mistake. The first round of corrections ends successfully at 12:30 p.m., and the latest estimate for shot setup into Tevatron now moves to 1:30 p.m.

Instead, a quench happens. "Anytime a superconducting magnet gets warmed up, it’s called a quench," explains Meyhoefer. The accelerator’s superconducting wire and magnets operate at -269°C, just below the boiling point of helium. When the beam sprays a magnet, the liquid helium used to cool the equipment warms up and becomes a gas. Operators must vent the helium gas before resuming normal operations.

Operators finish transferring the last of the 36 antiproton bunches into the Tevatron around 6 p.m. They dub the fresh set of colliding beams "store 4658."


Working in the CDF control room, physicists Ted Liu, Kirsten Tollefson (center), and Veronica Sorin test new software that triggers the recording of particle collisions.

Trigger happy

During an Illinois winter, the 15 minutes it takes to walk the half mile from the Main Control Room to the CDF building, site of one of two collision detectors, can seem much longer. Inside, operators at CDF measure store 4658’s initial luminosity at 6:19 p.m., before collisions start. With an average value of 142.59E30, it’s only 20 percent off the record.

When the proton and antiproton beams collide, CDF searches the resulting "collision wreckage" for interesting information. The greatest amount of debris comes when luminosity is highest at the outset of a store. "We have a half hour of excitement followed by 24 hours of boredom," jokes the shift scientific coordinator, Alvin Laasanen, of Purdue University. "As experimenters we have no control over the quality of the store, but we are responsible for our detector. We have to monitor the beam conditions to determine when it is safe to turn on. Our silicon-based components in particular are very sensitive to high radiation levels and can be damaged if the levels are too high." Once the detector is on and data-taking begins, operators settle into a monitoring mode.

Bunches of protons and antiprotons are now making collisions, but CDF can record only 50 events from the millions of collisions per second. A programmed "trigger table" determines which collisions are the most interesting to record.

A group of young physicists, men and women, huddle near two consoles, talking and gesturing. "They’re testing a new trigger table," says Brian Mohr of UCLA, expert for the CDF Data Acquisition system. "We have to rescale trigger tables as luminosity changes, so different events will trigger recording." The new trigger tables will go into use when the Tevatron restarts after its scheduled shutdown.


Main Control Room
A constant beep, which rises or falls in pitch based on beam intensity, alerts crew chief Duane Newhart and his colleagues of the status of the Tevatron collider.


Scientists in the CDF control room monitor the steady stream of data produced by proton- antiproton collisions and work on upgrades to the system.


Packed with monitors, the DZero control room allows scientists to keep an eye on the many subsystems of the DZero particle detector.

From the other side

Off a prairie road lined with pine trees, the DZero building sits opposite CDF, across the 1.3-mile-wide Tevatron ring. By 7:40 p.m., Marco Verzocchi has nearly finished taking data to test DZero’s new trigger table. He works alone at a monitor next to the shift crew chief.

"I was hoping for a luminosity record," says Verzocchi, looking weary. "When we have high enough luminosity, we collect special data to simulate the new trigger system offline." Over the previous weeks leading up to the shutdown, he has been collecting such data, which means some days coming to the control room in the middle of the night and some, staying late.

Nearby, Marie-Claude Cousinou of France’s Centre de Physique des Particules de Marseille views a constantly changing 3D chart of red, blue, and yellow bars: the detector’s interpretations of electromagnetic energy, hadronic energy, and missing energy, respectively. Sometimes the amount of missing energy, compared to the bars of known energy, is astounding: a yellow skyscraper amidst blue and red tents. Cousinou offers a possible explanation: "Often a W [boson] will decay to a lepton and neutrinos. The neutrinos escape detection here, and so the missing energy could be due to neutrinos." She then slips away from her console to the break room to heat up a frozen dinner in a microwave. It’s 9 p.m.


The MINOS detector only records a few neutrino events per day. Working a shift in the MINOS control room, Sergey Kotelnikov makes sure that none of these precious events evades detection.


Sitting in a control room on the 10th floor of Wilson Hall, Michel Sorel has a good view of the Fermilab prairie as well as the data recorded by the MiniBooNE neutrino experiment.

Midnight running

After a shift change at midnight, and one small false alarm, the new CDF crew looks comfortably settled by 12:30 a.m. The four operators on this shift are physicists who hail from Japan, Italy, and Germany. A hushed tone has replaced the exuberance of the previous CDF group, which worked on the trigger table. This group monitors the detector and its incoming data.

Back in Fermilab’s main building, Wilson Hall, a lone operator keeps watch at the MINOS experiment. MINOS receives proton beam diverted from the Main Injector accelerator, a machine that also can "inject" protons into the Tevatron or send them toward a tungsten-rhenium target to make antiprotons. MINOS’ fraction of proton beam slams into a carbon target to produce neutrinos.

At CDF and DZero, data inundate the control rooms during the collisions of protons and antiprotons–composite particles containing three quarks–with plenty of collision debris. In contrast, MINOS operators record only 4 to 6 neutrino candidates per day. Neutrinos’ infinitesimal mass and their ultra-rare interactions with matter make them incredibly hard to detect.

At 1 a.m., Panos Stamoulis, from the University of Athens, is one hour into his shift at MINOS and everything has been running smoothly. A beam neutrino candidate occurred around 11 p.m. on an earlier shift; otherwise, it has been a quiet night. "I’ve been interested in the kinds of questions asked in physics since high school," says Stamoulis. "In high-energy physics, we ask some of the most basic questions about the universe."

And as anyone who works in the control room of a high-energy particle physics lab knows, the universe doesn’t take a holiday.

Breakfast with neutrinos

At 5:30 a.m. on Tuesday, Main Control Room operations report that store 4658 aborted. While CDF and DZero must now await the injection of the next store into the Tevatron, Fermilab’s neutrino experiments keep plugging along.

By 9 a.m., the sun has been up more than two hours, warming a cloudy, chilly Illinois day. On the 10th floor of Wilson Hall, the MiniBooNE control room has windows that look out over the prairie. Michel Sorel, a native of Italy, now affiliated with the University of Valencia in Spain, sits alone in the room. "See the flag pole there, without the flag, in the middle of the woods? That’s where our detector is underground," he says, pointing to a tall white pole amidst leafless trees. Protocol requires MiniBooNE operators to go down to the detector site once per shift to check equipment.

MiniBooNE tests for neutrino mass by studying neutrino oscillations. MiniBooNE’s neutrinos stem from a beam of protons from the Booster accelerator. The proton beam from the Booster has an energy of 8 GeV (while MINOS gets proton beam at 120 GeV). Using proton beam of a lower energy allows MiniBooNE to be sensitive to hints at neutrino oscillations that the LSND experiment at Los Alamos detected in the 1990s.

MiniBooNE researchers are currently finishing analysis from the experiment’s first run, which lasted nearly three years. The experiment collaborators expect to have final results in summer 2006. Will they announce signs of a fourth neutrino, a revolutionary discovery? Such an assertion would dramatically change particle physicists’s view of the microcosm.

The 24 hours have ended without a luminosity record, but there has been a decision: The Tevatron will shut down a few days earlier than planned. Weeks of the mundane will follow: repairs, maintenance, installations of upgrades. But perhaps at some later time, sifting through the data collected on this day, the mundane will produce the marvelous.

1 Main Control Room
2 MINOS and MiniBooNE control rooms
3 MINOS detector
4 MiniBooNE detector
6 DZero
7 Tevatron collider

Click here to download the pdf version of this article.