Page 1 of 2

problems 10/26/09 - 10/28/09

Posted: Fri Nov 27, 2009 5:08 am
by Xymox
They lost Cryo lock on section 3-4

As I write this temps are still rising. Im not sure why.

I have seen this before and it look about 1/2 hrs to cool it back down again. Everything should be fine.. I would just feel better seeing the temps start to go down again rather then up.

Its odd how section 3-4 seems to warm up like this. With one small section of magnets getting hot quickly. This section is just past the ones they replaced after the incident last year..

Overall I dont think this is any big deal.. But worth watching to make sure.

http://hcc.web.cern.ch/hcc/cryo_main/cr ... n=Sector34

Re: ** Current problem **

Posted: Fri Nov 27, 2009 2:11 pm
by Xymox
Things are cooling back down now...

We will never know what caused the problem most likely. The log mentions "Automat" whatever that is.

These kinds of events can be a good thing as well. They allow underground access to the machine directly and workers can do stuff that would normally have to wait until the machine was turned off.

I doubt this will effect the schedule at all.

They expect cooling will be done around 11PM to midnight CERN time.

These types of things are completely normal. These are just part of the normal process. All part of the checkout process of a new machine.

Re: ** Current problem **

Posted: Fri Nov 27, 2009 2:44 pm
by Lexa
The "Automat" term refer to PLC.

The problem already shows up on wednesday in sector 12. And might happen again randomly if not solved.

Re: ** Current problem **

Posted: Fri Nov 27, 2009 3:02 pm
by Xymox
So some form of software problem then ?

Well certainly understandable. I wonder how much custom written software code there is at LHC ? If you include all the firmware, PLC stuff and higher level stuff... Must just be a enormous amount.

Working out bugs would just be normal and expected. Im sure its annoying for the LHC team.. It is also fun however because its like a puzzle that needs to be solved. Debugging software/hardware can be fun..

Re: ** Current problem **

Posted: Fri Nov 27, 2009 3:20 pm
by Lexa
I don't want to spoil/leak anything, but... responsibility might not be on our custom stuffs in that case :].

Re: ** Current problem **

Posted: Fri Nov 27, 2009 3:27 pm
by Danny252
Cryo was acting up 1200/0000 CERN time last night - I saw a couple of the sector 34 cryo displays go red for about 5 minutes, as well as a few other yellows and warning signs. Weren't they having problems with beam loss at position 4 (and RF, situated there, was acting up all day as well) earlier too?

Re: ** Current problem **

Posted: Fri Nov 27, 2009 3:46 pm
by Xymox
Lexa.. Hehehe... So its a manufacturer's glitch... Well thats OK too. There are a tremendous number of systems at the LHC.. All the systems made by all the manufactures also will need debugging as well. So its a fun puzzle for them. Its fun to solve a problem like that under time pressures.

Danny252 .... wow you exceeded me in careful monitoring ! You must come post on the google groups email reflector. When you see something you should post it right away there.

We need more people like you who are watching to post on the email reflector. We need a little team of people watching. I have to sleep, work and do other stuff so its impossible for me to watch all the time.

The group is at the top of the list on the portal links. If anyone wants to stay anonymous just create a gmail account just for this use.

Re: ** Current problem **

Posted: Fri Nov 27, 2009 4:34 pm
by Xymox
Looks like the problem is solved. Faster then expected..

On we go...

Im going to drop this topic back to the forum from a global annoucement and re title it.

Re: ** Current problem **

Posted: Fri Nov 27, 2009 4:37 pm
by Lexa
It is likely to happen again ^^.
We could start betting on which sector will be affected next time!

Re: ** Current problem **

Posted: Fri Nov 27, 2009 6:55 pm
by Xymox
Hahahaha. a Automat betting pool... Well im sure they will get to the bottom of it. Im sure it has been quite the annoyance..

SO now we have a QPS problem.... Again,,, what is that ?

Re: ** Current problem **

Posted: Fri Nov 27, 2009 7:15 pm
by Allan
Hi

IMHO QPS stands for Quench Protection System.

Re: ** Current problem **

Posted: Fri Nov 27, 2009 7:16 pm
by Lexa
Quench Protection System

Speaking about commisionning very complicated devices, QPS is a good example :).

Re: ** Current problem **

Posted: Fri Nov 27, 2009 7:36 pm
by Xymox
I have not done much reading on it. Obviously it has to work quickly and I would imagine it has to be able to shut down things and initiate a beam dump completely on its own with no operator intervention because speed is of the essence ?

Re: ** Current problem **

Posted: Fri Nov 27, 2009 8:30 pm
by Lexa
Speed is in millisecond for the QPS.

I don't know for the dump thing. Its point is to protect a magnet by spreading a hot spot (created by a quench) to the whole magnet in order to avoid any local damage (that would impair the whole magnet x_x).
So it has to detect a quench (in most of the configuration possible) and trigger whatever has to be done to avoid damages.

Re: ** Current problem **

Posted: Fri Nov 27, 2009 8:51 pm
by Danny252
CERN's Eplanation of what the QPS is needed for, how it works, etc.:

https://edms.cern.ch/file/338035/1.0/LH ... -10-00.pdf

I'm guessing the QPS is moaning because certain cryo things are reporting as too hot (real or not), and saying that turning it on would be a bad idea.