problems 10/26/09 - 10/28/09

The place to discuss the LHC. Commissioning, operation, issues, events ....
User avatar
Xymox
Site Admin
Posts: 1022
Joined: Wed Nov 11, 2009 12:55 pm
Location: Phoenix, Az USA
Contact:

problems 10/26/09 - 10/28/09

Post by Xymox » Fri Nov 27, 2009 5:08 am

They lost Cryo lock on section 3-4

As I write this temps are still rising. Im not sure why.

I have seen this before and it look about 1/2 hrs to cool it back down again. Everything should be fine.. I would just feel better seeing the temps start to go down again rather then up.

Its odd how section 3-4 seems to warm up like this. With one small section of magnets getting hot quickly. This section is just past the ones they replaced after the incident last year..

Overall I dont think this is any big deal.. But worth watching to make sure.

http://hcc.web.cern.ch/hcc/cryo_main/cr ... n=Sector34

User avatar
Xymox
Site Admin
Posts: 1022
Joined: Wed Nov 11, 2009 12:55 pm
Location: Phoenix, Az USA
Contact:

Re: ** Current problem **

Post by Xymox » Fri Nov 27, 2009 2:11 pm

Things are cooling back down now...

We will never know what caused the problem most likely. The log mentions "Automat" whatever that is.

These kinds of events can be a good thing as well. They allow underground access to the machine directly and workers can do stuff that would normally have to wait until the machine was turned off.

I doubt this will effect the schedule at all.

They expect cooling will be done around 11PM to midnight CERN time.

These types of things are completely normal. These are just part of the normal process. All part of the checkout process of a new machine.

Lexa
Posts: 30
Joined: Fri Nov 27, 2009 2:25 pm
Location: Thoiry/Geneva (CERN)

Re: ** Current problem **

Post by Lexa » Fri Nov 27, 2009 2:44 pm

The "Automat" term refer to PLC.

The problem already shows up on wednesday in sector 12. And might happen again randomly if not solved.

User avatar
Xymox
Site Admin
Posts: 1022
Joined: Wed Nov 11, 2009 12:55 pm
Location: Phoenix, Az USA
Contact:

Re: ** Current problem **

Post by Xymox » Fri Nov 27, 2009 3:02 pm

So some form of software problem then ?

Well certainly understandable. I wonder how much custom written software code there is at LHC ? If you include all the firmware, PLC stuff and higher level stuff... Must just be a enormous amount.

Working out bugs would just be normal and expected. Im sure its annoying for the LHC team.. It is also fun however because its like a puzzle that needs to be solved. Debugging software/hardware can be fun..

Lexa
Posts: 30
Joined: Fri Nov 27, 2009 2:25 pm
Location: Thoiry/Geneva (CERN)

Re: ** Current problem **

Post by Lexa » Fri Nov 27, 2009 3:20 pm

I don't want to spoil/leak anything, but... responsibility might not be on our custom stuffs in that case :].

Danny252
LHCPortal Guru
Posts: 169
Joined: Mon Nov 23, 2009 4:45 pm

Re: ** Current problem **

Post by Danny252 » Fri Nov 27, 2009 3:27 pm

Cryo was acting up 1200/0000 CERN time last night - I saw a couple of the sector 34 cryo displays go red for about 5 minutes, as well as a few other yellows and warning signs. Weren't they having problems with beam loss at position 4 (and RF, situated there, was acting up all day as well) earlier too?

User avatar
Xymox
Site Admin
Posts: 1022
Joined: Wed Nov 11, 2009 12:55 pm
Location: Phoenix, Az USA
Contact:

Re: ** Current problem **

Post by Xymox » Fri Nov 27, 2009 3:46 pm

Lexa.. Hehehe... So its a manufacturer's glitch... Well thats OK too. There are a tremendous number of systems at the LHC.. All the systems made by all the manufactures also will need debugging as well. So its a fun puzzle for them. Its fun to solve a problem like that under time pressures.

Danny252 .... wow you exceeded me in careful monitoring ! You must come post on the google groups email reflector. When you see something you should post it right away there.

We need more people like you who are watching to post on the email reflector. We need a little team of people watching. I have to sleep, work and do other stuff so its impossible for me to watch all the time.

The group is at the top of the list on the portal links. If anyone wants to stay anonymous just create a gmail account just for this use.

User avatar
Xymox
Site Admin
Posts: 1022
Joined: Wed Nov 11, 2009 12:55 pm
Location: Phoenix, Az USA
Contact:

Re: ** Current problem **

Post by Xymox » Fri Nov 27, 2009 4:34 pm

Looks like the problem is solved. Faster then expected..

On we go...

Im going to drop this topic back to the forum from a global annoucement and re title it.

Lexa
Posts: 30
Joined: Fri Nov 27, 2009 2:25 pm
Location: Thoiry/Geneva (CERN)

Re: ** Current problem **

Post by Lexa » Fri Nov 27, 2009 4:37 pm

It is likely to happen again ^^.
We could start betting on which sector will be affected next time!

User avatar
Xymox
Site Admin
Posts: 1022
Joined: Wed Nov 11, 2009 12:55 pm
Location: Phoenix, Az USA
Contact:

Re: ** Current problem **

Post by Xymox » Fri Nov 27, 2009 6:55 pm

Hahahaha. a Automat betting pool... Well im sure they will get to the bottom of it. Im sure it has been quite the annoyance..

SO now we have a QPS problem.... Again,,, what is that ?

User avatar
Allan
Posts: 109
Joined: Fri Nov 27, 2009 7:10 pm
Location: Denver, Colorado

Re: ** Current problem **

Post by Allan » Fri Nov 27, 2009 7:15 pm

Hi

IMHO QPS stands for Quench Protection System.

Lexa
Posts: 30
Joined: Fri Nov 27, 2009 2:25 pm
Location: Thoiry/Geneva (CERN)

Re: ** Current problem **

Post by Lexa » Fri Nov 27, 2009 7:16 pm

Quench Protection System

Speaking about commisionning very complicated devices, QPS is a good example :).

User avatar
Xymox
Site Admin
Posts: 1022
Joined: Wed Nov 11, 2009 12:55 pm
Location: Phoenix, Az USA
Contact:

Re: ** Current problem **

Post by Xymox » Fri Nov 27, 2009 7:36 pm

I have not done much reading on it. Obviously it has to work quickly and I would imagine it has to be able to shut down things and initiate a beam dump completely on its own with no operator intervention because speed is of the essence ?

Lexa
Posts: 30
Joined: Fri Nov 27, 2009 2:25 pm
Location: Thoiry/Geneva (CERN)

Re: ** Current problem **

Post by Lexa » Fri Nov 27, 2009 8:30 pm

Speed is in millisecond for the QPS.

I don't know for the dump thing. Its point is to protect a magnet by spreading a hot spot (created by a quench) to the whole magnet in order to avoid any local damage (that would impair the whole magnet x_x).
So it has to detect a quench (in most of the configuration possible) and trigger whatever has to be done to avoid damages.

Danny252
LHCPortal Guru
Posts: 169
Joined: Mon Nov 23, 2009 4:45 pm

Re: ** Current problem **

Post by Danny252 » Fri Nov 27, 2009 8:51 pm

CERN's Eplanation of what the QPS is needed for, how it works, etc.:

https://edms.cern.ch/file/338035/1.0/LH ... -10-00.pdf

I'm guessing the QPS is moaning because certain cryo things are reporting as too hot (real or not), and saying that turning it on would be a bad idea.

Post Reply