Page 2 of 2

Re: *** MAJOR power loss and many systems down

Posted: Wed Dec 02, 2009 9:42 am
by LHC_Insider
Xymox wrote: I dont think it will cause almost any delay. It appears The LHC stayed cold and wont need to be re chilled. I would think mostly rebooting and such..
It takes weeks to warm up a sector if cryo goes. A few hours with tripped power is nothing. The power cut was into the Meyrin site. The induced effects in the power grid tripped the cryo stuff. These things happen.

With regards to the energy consumption, LHC and experiments is around 120 MW

Re: *** MAJOR power loss and many systems down

Posted: Wed Dec 02, 2009 9:48 am
by daavery
there are some pictures of the damaged wire ends in the morning briefing pptx file

http://lhc-commissioning.web.cern.ch/lh ... 21209.pptx

Re: *** MAJOR power loss and many systems down

Posted: Wed Dec 02, 2009 3:00 pm
by Xymox
Man judging from the black soot in those photos that must have been a pretty good POOF.. 18KV and those connections look like high current. Must have been quite a bit of power going through the wire at the time.

I would not have wanted to be in that electrical closet at the time of the failure :shock:

Im sure thats most likely 3 phase power, so it looks like only one phase blew out. That would explain the reference to a breaker tripping. Loosing 1 phase of a 3 phase connection can be nasty.

Well... The LHC was almost not effected at all. Maybe 1 day of delay ? They are already well ahead of schedule anyway.

The good news is its good that the failure happened now rather then later during science.

A fairly exciting event, but no bad impact at all.

Excellent..

Re: *** MAJOR power loss and many systems down

Posted: Wed Dec 02, 2009 5:07 pm
by March_Hare
Lexa wrote:I have seen the power converter of the PS (well, the old one, and the new one we are currently working on it).
If already 50 years ago they were able to do such machine that resilient to power cut, I believe for the LHC it is really not something to bother about.

Protection, redundancy and backup circuit everywhere in large scale :shock:
That's what I figured. I was actually wondering how it all works (and that's what my question was about). I mean... if I try to imagine the scale of the machine... and then the need to ensure safety in case of unpredictable events... :?

Re: *** MAJOR power loss and many systems down

Posted: Wed Dec 02, 2009 5:29 pm
by publius
A couple of things:

Backup power is relatively straight forward. Critical control systems will be powered through many uninterruptible power supplies (UPS) running on batteries. The UPS batteries have to last just long enough for the diesel generators to spin up and stabilize, about 30 seconds in most cases. After that, its a matter of fuel supplies. The control rooms of some nuclear power plants run directly from huge banks of batteries, just to be sure they stay running no matter what happens.

Since CERN is a real power hog, many separate utility power sources are used, so not all equipment powers down when just one feeder is lost. However, keeping everything stable when this happens is really challenging. Obviously, the designers are very, very good.

Typically, a power conductor is limited in the maximum current it can carry without excessive heating. For modern aluminum/steel cables, the largest size is around 28mm diameter and the maximumn current is about 1000 amperes. So, very roughly, 17MW were flowing through the breaker when it tripped. Stopping this power flow is akin to blocking the most powerful of lightning bolts. Ouch!

Re: *** MAJOR power loss and many systems down

Posted: Wed Dec 02, 2009 5:59 pm
by March_Hare
Ah, ok thx!

Re: *** MAJOR power loss and many systems down

Posted: Wed Dec 02, 2009 6:46 pm
by Xymox
The forum has made Scientific American .. They botched the link to the story. Then I posted the correction 4 times on accident.

http://www.scientificamerican.com/blog/ ... 2009-12-02

Re: *** MAJOR power loss and many systems down

Posted: Wed Dec 02, 2009 11:06 pm
by DCWhitworth
I love this entry in hte LHCb Logbook from the time of the power loss. Someone's keen for a break it seems !

18 kV drop on the Meyrin site....
6 LHC sectors tripped.
I guess this will give some rest.

Re: *** MAJOR power loss and many systems down

Posted: Thu Dec 03, 2009 2:29 am
by Xymox
Well the guys working at CERN have been working around the clock. They have been working really hard..

I am sure a rest was a good thing..

Re: *** MAJOR power loss and many systems down

Posted: Thu Dec 03, 2009 2:35 am
by Xymox
I am going to leave this up for another day because of the press coverage. Then it will get dated and stored under the collider forum

Re: *** MAJOR power loss and many systems down

Posted: Thu Dec 03, 2009 3:24 am
by Harbles
Final verdict on this incident at the morning commissioning meeting [url]http://lhc-commissioning.web.cern.ch/lh ... ws.htm/url] in the power point (07) .pptx file are photos of the damage to hi-voltage switch gear (transformer or breaker?)
All better now, except for a rocky restart plagued by power fluctuations as filters were reconnected to power feeds causing system glitches. They had beam for a while then a faulty data item from the cryo in 4-5 caused a prob.
Could be worse.
It's pretty good actually, probably push collisions back to Saturday though.