12/2/09 power loss and many systems down

The place to discuss the LHC. Commissioning, operation, issues, events ....
LHC_Insider
LHCPortal Guru
Posts: 15
Joined: Sat Nov 28, 2009 11:05 pm

Re: *** MAJOR power loss and many systems down

Post by LHC_Insider » Wed Dec 02, 2009 9:42 am

Xymox wrote: I dont think it will cause almost any delay. It appears The LHC stayed cold and wont need to be re chilled. I would think mostly rebooting and such..
It takes weeks to warm up a sector if cryo goes. A few hours with tripped power is nothing. The power cut was into the Meyrin site. The induced effects in the power grid tripped the cryo stuff. These things happen.

With regards to the energy consumption, LHC and experiments is around 120 MW

daavery
Posts: 12
Joined: Sun Nov 29, 2009 3:40 am

Re: *** MAJOR power loss and many systems down

Post by daavery » Wed Dec 02, 2009 9:48 am

there are some pictures of the damaged wire ends in the morning briefing pptx file

http://lhc-commissioning.web.cern.ch/lh ... 21209.pptx

User avatar
Xymox
Site Admin
Posts: 1022
Joined: Wed Nov 11, 2009 12:55 pm
Location: Phoenix, Az USA
Contact:

Re: *** MAJOR power loss and many systems down

Post by Xymox » Wed Dec 02, 2009 3:00 pm

Man judging from the black soot in those photos that must have been a pretty good POOF.. 18KV and those connections look like high current. Must have been quite a bit of power going through the wire at the time.

I would not have wanted to be in that electrical closet at the time of the failure :shock:

Im sure thats most likely 3 phase power, so it looks like only one phase blew out. That would explain the reference to a breaker tripping. Loosing 1 phase of a 3 phase connection can be nasty.

Well... The LHC was almost not effected at all. Maybe 1 day of delay ? They are already well ahead of schedule anyway.

The good news is its good that the failure happened now rather then later during science.

A fairly exciting event, but no bad impact at all.

Excellent..

User avatar
March_Hare
Posts: 135
Joined: Tue Dec 01, 2009 6:09 pm
Location: The Netherlands

Re: *** MAJOR power loss and many systems down

Post by March_Hare » Wed Dec 02, 2009 5:07 pm

Lexa wrote:I have seen the power converter of the PS (well, the old one, and the new one we are currently working on it).
If already 50 years ago they were able to do such machine that resilient to power cut, I believe for the LHC it is really not something to bother about.

Protection, redundancy and backup circuit everywhere in large scale :shock:
That's what I figured. I was actually wondering how it all works (and that's what my question was about). I mean... if I try to imagine the scale of the machine... and then the need to ensure safety in case of unpredictable events... :?
Nothing travels faster than the speed of light with the possible exception of bad news, which obeys its own special laws.
~Douglas Adams

publius
Posts: 3
Joined: Thu Nov 19, 2009 2:33 pm

Re: *** MAJOR power loss and many systems down

Post by publius » Wed Dec 02, 2009 5:29 pm

A couple of things:

Backup power is relatively straight forward. Critical control systems will be powered through many uninterruptible power supplies (UPS) running on batteries. The UPS batteries have to last just long enough for the diesel generators to spin up and stabilize, about 30 seconds in most cases. After that, its a matter of fuel supplies. The control rooms of some nuclear power plants run directly from huge banks of batteries, just to be sure they stay running no matter what happens.

Since CERN is a real power hog, many separate utility power sources are used, so not all equipment powers down when just one feeder is lost. However, keeping everything stable when this happens is really challenging. Obviously, the designers are very, very good.

Typically, a power conductor is limited in the maximum current it can carry without excessive heating. For modern aluminum/steel cables, the largest size is around 28mm diameter and the maximumn current is about 1000 amperes. So, very roughly, 17MW were flowing through the breaker when it tripped. Stopping this power flow is akin to blocking the most powerful of lightning bolts. Ouch!

User avatar
March_Hare
Posts: 135
Joined: Tue Dec 01, 2009 6:09 pm
Location: The Netherlands

Re: *** MAJOR power loss and many systems down

Post by March_Hare » Wed Dec 02, 2009 5:59 pm

Ah, ok thx!
Nothing travels faster than the speed of light with the possible exception of bad news, which obeys its own special laws.
~Douglas Adams

User avatar
Xymox
Site Admin
Posts: 1022
Joined: Wed Nov 11, 2009 12:55 pm
Location: Phoenix, Az USA
Contact:

Re: *** MAJOR power loss and many systems down

Post by Xymox » Wed Dec 02, 2009 6:46 pm

The forum has made Scientific American .. They botched the link to the story. Then I posted the correction 4 times on accident.

http://www.scientificamerican.com/blog/ ... 2009-12-02

User avatar
DCWhitworth
LHCPortal Guru
Posts: 581
Joined: Mon Nov 30, 2009 8:13 am
Location: Norwich, UK

Re: *** MAJOR power loss and many systems down

Post by DCWhitworth » Wed Dec 02, 2009 11:06 pm

I love this entry in hte LHCb Logbook from the time of the power loss. Someone's keen for a break it seems !

18 kV drop on the Meyrin site....
6 LHC sectors tripped.
I guess this will give some rest.
DC

The LHC - One ring to rule them all !

User avatar
Xymox
Site Admin
Posts: 1022
Joined: Wed Nov 11, 2009 12:55 pm
Location: Phoenix, Az USA
Contact:

Re: *** MAJOR power loss and many systems down

Post by Xymox » Thu Dec 03, 2009 2:29 am

Well the guys working at CERN have been working around the clock. They have been working really hard..

I am sure a rest was a good thing..

User avatar
Xymox
Site Admin
Posts: 1022
Joined: Wed Nov 11, 2009 12:55 pm
Location: Phoenix, Az USA
Contact:

Re: *** MAJOR power loss and many systems down

Post by Xymox » Thu Dec 03, 2009 2:35 am

I am going to leave this up for another day because of the press coverage. Then it will get dated and stored under the collider forum

Harbles
LHCPortal Guru
Posts: 314
Joined: Sat Nov 28, 2009 10:22 pm
Location: Mississauga, Ontario. Canada

Re: *** MAJOR power loss and many systems down

Post by Harbles » Thu Dec 03, 2009 3:24 am

Final verdict on this incident at the morning commissioning meeting [url]http://lhc-commissioning.web.cern.ch/lh ... ws.htm/url] in the power point (07) .pptx file are photos of the damage to hi-voltage switch gear (transformer or breaker?)
All better now, except for a rocky restart plagued by power fluctuations as filters were reconnected to power feeds causing system glitches. They had beam for a while then a faulty data item from the cryo in 4-5 caused a prob.
Could be worse.
It's pretty good actually, probably push collisions back to Saturday though.

Post Reply