Clara Gaspar, July 2005
RTTC Control System
Status and Plans
Clara Gaspar, July 2005
2
Architecture & Components
❙Farm Infrastructure
❙Gaudi Jobs
❙Event Builder
❙MEP Producer
❙Run Control
Event Builder Switch
.
.
.
.
.
.
.
.
.
. . .
Control PCPVSS
Control PCPVSS
SFC
CPU
CPU
CPU
SFC
CPU
CPU
CPU
SFC
CPU
CPU
CPU
❚Monitoring & Control of:
Clara Gaspar, July 2005
3
Farm Infrastructure
❚Very Complete Monitoring of:
❙Each CPU:
❘Processes running, CPU usage, Memory usage, etc.
❘Network traffic
❘Temperature & Fan speeds, etc.
❚Control:
❙Task Manager: Start/Stop Jobs on nodes
❙IPMI Manager: Switch on/off, reboot nodes
❙FSM Automated Monitoring (& Control):
❘Set CPU in "ERROR" when monit. quantities bad
〡Could/will take automatic actions
Clara Gaspar, July 2005
4
Farm Monitoring
Clara Gaspar, July 2005
5
Gaudi Job Monitoring
❚Gaucho Tasks:
❙Start/Stop Gaudi Jobs on each node:
❘2 L1 (Euler) Processes
❘2 HLT (Moore) Processes
❙Receive Monitoring Info from jobs:
❘Counters (seen/rejected/accepted events)
❘histograms, etc
❙Accumulate and Save Statistics
❘per job type/per node/per sub-farm/full farm
❙Automate operations (start/stop run)
❘According to configuration
Clara Gaspar, July 2005
6
Gaucho Example
❚Accumulated Statistics from 2 nodes (4 L1/Euler Jobs)
Clara Gaspar, July 2005
7
RTTC Run Control
❚Event Builder(s) & MEP Producer:
❙Receive monitoring information
❙FSM control:
❘Deduce a state and start/stop RUN
❚RUN Control
❙Coordinate all components
❘Distribute Configuration and RUN commands to:Evt_builder/ MEP Producer / Gaucho
〡Depending on the state of the FARM infrastructure
❘Show overall status and summary information
Clara Gaspar, July 2005
8
Control System Status
❚Farm Monitoring
❙First version ready
❙Permanently running, acquiring:
❘~1000 quantities per node every 20 seconds
❚Farm Control
❙Power Control working
❙Task Manager heavily used
❙FSM rules currently disabled
❘But still around 200 quantities per node beingchecked
❚But: PVSS perf. degrades after some days...
Clara Gaspar, July 2005
9
Control System Status
❚Gaucho
❙Prototype Ready but:
❘Can not run it for more than a few nodes currently
❚Problem: FSM
❙Current FSM implementation:
❘A Control Unit (a sub-system) spawns one FSMprocess and one PVSS process
❘A Device Unit (a device) is light weight
❙The way most users designed their system is:
❘One control unit per device (or few devices)(in Gaucho: 7 CU per node * 44 nodes)
❙Uses too much memory
Clara Gaspar, July 2005
10
Solution:
❚New FSM version
❙A Control Unit (a sub-system) spawns an FSMprocess
❙A Logical Unit (grouping devices) light weight
❙A Device Unit (a device) light weight
❙Only one PVSS process for all CUs
❚Almost ready
❙Will be released this week or next week
Clara Gaspar, July 2005
11
Plan
❚Farm monitoring & Control
❙Run alone on PC until PVSS "degradation" isunderstood
❙Optimize project in general
❚Gaucho
❙Install new version of FSM
❙Test Gaucho on large scale
❚Run Control & others
❙Integrate when above ready
Clara Gaspar, July 2005
12
Conclusions
❚All control developments can be done inparallel with DAQ studies andmeasurements
❚The Farm test bed will be used until allproblems are solved. Our aim is still tohave an integrated Run Control with allcomponents in one Control PC.
❚We will report regularly on progress