12/2/09 power loss and many systems down

The place to discuss the LHC. Commissioning, operation, issues, events ....
User avatar
Xymox
Site Admin
Posts: 1022
Joined: Wed Nov 11, 2009 12:55 pm
Location: Phoenix, Az USA
Contact:

12/2/09 power loss and many systems down

Post by Xymox » Wed Dec 02, 2009 2:32 am

It appears CERN is completely off the internet.

Thats fairly remarkable

Might be some large scale internet network failure... NOT at CERN...

Lexa
Posts: 30
Joined: Fri Nov 27, 2009 2:25 pm
Location: Thoiry/Geneva (CERN)

Re: *** Complete internet access failure

Post by Lexa » Wed Dec 02, 2009 2:34 am

Refer to this:
viewtopic.php?f=4&t=108

We are actually not completely off the internet. I reckon that cern network is really great because while Meyrin is off, Prevessin has still access to the world ^^.

A few, very few web servers are responding (but useless ones :D)

User avatar
Xymox
Site Admin
Posts: 1022
Joined: Wed Nov 11, 2009 12:55 pm
Location: Phoenix, Az USA
Contact:

Re: *** Complete internet access failure

Post by Xymox » Wed Dec 02, 2009 2:36 am

Things are back i think

wow that was rather amazing..

Serious loss of internet connection

http://network-statistics.web.cern.ch/n ... %20Traffic

Lexa
Posts: 30
Joined: Fri Nov 27, 2009 2:25 pm
Location: Thoiry/Geneva (CERN)

Re: *** Complete internet access failure

Post by Lexa » Wed Dec 02, 2009 2:55 am

Not quite completely :D
I think just the monitoring service are back online.

And it looks like the magnet did not warm up, phew!

User avatar
Xymox
Site Admin
Posts: 1022
Joined: Wed Nov 11, 2009 12:55 pm
Location: Phoenix, Az USA
Contact:

Re: *** Complete internet access failure

Post by Xymox » Wed Dec 02, 2009 3:05 am

This is going to become a news story quickly...

User avatar
Xymox
Site Admin
Posts: 1022
Joined: Wed Nov 11, 2009 12:55 pm
Location: Phoenix, Az USA
Contact:

Re: *** Complete internet access failure

Post by Xymox » Wed Dec 02, 2009 3:14 am


User avatar
Xymox
Site Admin
Posts: 1022
Joined: Wed Nov 11, 2009 12:55 pm
Location: Phoenix, Az USA
Contact:

Re: *** MAJOR power loss and many systems down

Post by Xymox » Wed Dec 02, 2009 3:25 am

Once they have power back and all the systems are back online,,, I really dont think this will create much delay at all..

I think that is worth repeating..

I don't think this will create much of a delay at all...

EVERYTHING is still way ahead of schedule.

Man A bird better not have dropped a baguette into the power systems again !

Lexa
Posts: 30
Joined: Fri Nov 27, 2009 2:25 pm
Location: Thoiry/Geneva (CERN)

Re: *** MAJOR power loss and many systems down

Post by Lexa » Wed Dec 02, 2009 3:59 am

Page 1 just get updated.
Likely no beam before 12am.

User avatar
Xymox
Site Admin
Posts: 1022
Joined: Wed Nov 11, 2009 12:55 pm
Location: Phoenix, Az USA
Contact:

Re: *** MAJOR power loss and many systems down

Post by Xymox » Wed Dec 02, 2009 4:34 am

Well things are slowly coming back up...

The public site is back..

Man I feel bad for the poor IT guys that happened to be on duty.... You can bet they are having a headache..

User avatar
Xymox
Site Admin
Posts: 1022
Joined: Wed Nov 11, 2009 12:55 pm
Location: Phoenix, Az USA
Contact:

Re: *** MAJOR power loss and many systems down

Post by Xymox » Wed Dec 02, 2009 5:49 am

AGAIN

I dont think it will cause almost any delay. It appears The LHC stayed cold and wont need to be re chilled. I would think mostly rebooting and such.

They are WAY ahead of schedule anyway.

I just feel so sorry for everybody on duty today. LOTS of unexpected work to do.

User avatar
March_Hare
Posts: 135
Joined: Tue Dec 01, 2009 6:09 pm
Location: The Netherlands

Re: *** MAJOR power loss and many systems down

Post by March_Hare » Wed Dec 02, 2009 7:01 am

Is there public info about safety measures at LHC in case of power loss?
Come to think of it, is something known about the LHC's power consumption? Of the accelerator? Of the experiments?

I mean, power loss is obviously an eventuality you need to be prepared for. A power loss just seconds before high energy collisions start could wreck the place, so my guess is that you'd want sufficient beam control and monitoring equipment to continue functioning to ensure (and verify) that both beams are safely dumped.
I don't think this can come from generators because of slow response time of such equipment. So where DOES the power come from in case of an outage?
Nothing travels faster than the speed of light with the possible exception of bad news, which obeys its own special laws.
~Douglas Adams

pcatom
LHCPortal Guru
Posts: 68
Joined: Fri Nov 27, 2009 8:33 pm
Location: Geneva

Re: *** MAJOR power loss and many systems down

Post by pcatom » Wed Dec 02, 2009 7:16 am

There was a fault on an 18kV circuit breaker close to point 1 which caused a power cut on the Meyrin site. The power cut affected the main computer centre and switched off the pre-injector machines (linac, PSB,PS). LHC an SPS were not seriously affected. The electrical network has been reconfigured and the injectors should be restarted in the coming hours.

User avatar
Xymox
Site Admin
Posts: 1022
Joined: Wed Nov 11, 2009 12:55 pm
Location: Phoenix, Az USA
Contact:

Re: *** MAJOR power loss and many systems down

Post by Xymox » Wed Dec 02, 2009 7:32 am

YES !

See no serious delay !

GO CERN !

I admit to being a bit worried.

neutralino
Posts: 4
Joined: Sun Nov 22, 2009 11:08 am

Re: *** MAJOR power loss and many systems down

Post by neutralino » Wed Dec 02, 2009 7:49 am

Yes, me too.
Last year’s incident is still fresh in our memories.

Lexa
Posts: 30
Joined: Fri Nov 27, 2009 2:25 pm
Location: Thoiry/Geneva (CERN)

Re: *** MAJOR power loss and many systems down

Post by Lexa » Wed Dec 02, 2009 9:31 am

I have seen the power converter of the PS (well, the old one, and the new one we are currently working on it).
If already 50 years ago they were able to do such machine that resilient to power cut, I believe for the LHC it is really not something to bother about.

Protection, redundancy and backup circuit everywhere in large scale :shock:

Post Reply