Errors occurring sporadically:

• the Liberas loose their connection: they appear red in the detailed status panel, giving the status “Software Error” or “BPM Communication Error”. The Libera can not be controlled from TOPOS any more. Most of the time, a reboot of the Libera via ssh helps.

In at least one case, I also had to restart the FESA classes on the CCCPs to make the system working again. This happens sometimes during beamtime. Or after the system was not used for a while and is started again (= the GUI was closed for a while and is started again).

Other errors which could be related:

• the saving of raw data sometimes leads to a timeout
• aux BPM confuses the whole system (Start and Stop is triggered “out of itself”)
• low system performance (switching of mode lasts several seconds)

Bunch detection / Measurment data errors (KL):

• Under certain beam/signal conditions, Liberas deliver errernous measurment data. This causes TOPOS to display hard or not usable results.

Suggestions (Every idea to track down the problems are appreciated):

• add debug output into the Libera generic servers and the FESA Classes (see below)
• make testcases like: set a defined set of calibration values and check if they are set in the generic server/FPGA registers

Log in the FESA classes:

• version number on startup
• connection to BPM established/lost/reconnected
• debug output at every status change of the system (Initializing, Start, Stop,…)
• logging should use the Log4j framework (from within the FESA class possible with SDLog (HBr))
• the GUI should not encapsulate exceptions thrown by cmw / rda into its own Exception class (HBr)
• a detailed documentation of the meaning and reasns for each error message, exception etc. should be made (HBr)

Log on the generic servers (with timestamps!):

• version number (or similar) at startup
• internal register values (when changed, on start trigger, on stop trigger)
• when a start or stop trigger arrives
• when the ring buffer is full
• operating mode (raw, bunch to bunch, calibrations, log every change from - to)
• log any other useful events
• log buffer overflows
• there seem to be logs on the liberas under /var/log. But without timestamps. When a separate network for the Liberas is used, the time can't be queried from a global NTP server. → Setup a “proxy” on the concentrators?

Network considerations:

• put the Liberas into a own network (not the GSI/ACC network)
• bootfile on CCCPs, static IPs, nfs mount to store debug logs
• is this a lot of work? does it require changes in the gen servers / FPGA code (Change of FPGA code won't be necessary for this issue (KL))?

Connection to the PTIF:

• Display the connection status and if a command which has been sent, was “acknowledge” by the PTIF. Can give a hint, when the PTIF seems to be reachable by TCP/IP but the FESA class is not sending commands. Test case: Pull network cable and reconnect. System should be able to detect this, if the PTIF cannot be controlled afterwards.

Have a standalone tool to see ALL system components directly: FESA server classes, CCCPs, Liberas (pingable), the gen servers (are up and running?), show error flags. It might be a good idea to have a button for each component to perform a check. E.g. perform a ping for the Liberas and CCCPs. Or perform a data query from the FESA classes. Some of this information is provided by the detailed status panel, but you have to be an expert to interpret some errors.

GUI improvements for wrong measurement data (KL):

• A strong indicator for wrong measurment are the window lengths. Two indicators, which can be easily identified, are negative window lengths and window lengths bigger than 1.17 us (150 samples). If one of these conditions is met, the relating position can be marked as errorneous. Therefore a concept can be considered, which shows these errors somehow in the GUI.

Add logging output to all system components to know what is going on in each component for each status change. There should be a flag to en-/disable logging at startup. Provide tools to observe the health status of the system components.

Some more considerations (MSchw):

• I strongly support to have as much logging information as possible, e.g. to a textfile
• From my point of view it is very important that we clearly understand what SHOULD happen, e.g. when the user presses a button, BEFORE we try to understand why something we intend to do DOES NOT happen.
• I would recommend to have one or several (very basic, synoptic, NOT on code basis) diagramS of the internal process flow. This/these should be created by SD (Rainer?) together with Cosylab. The diagram/diagrams should include the same status names/exceptions used in the logfiles proposed above, thus helping to understand the history of error states.
• I also support the idea of a stand-alone diagnostic tool as described above, just make sure the displayed information is clearly defined and leaves few space for misinterpretations.