MTCA Crate Configuration

This document describes the setup and initial commissioning of the Powerbridge mTCA systems.
This manual covers the standard 2U - 6 slot and the 9U - 12 slot systems according to the 2019 tender.

A completely packed mTCA system consists of a 19-inch MTCA.4 chassis (2U or 9U) with NAT-PM600D (600W) or Wiener PS-1000W-M.4-AC (1000W) power supply, a Concurrent Technologies (CCT) AM-G64/471-51 CPU and a NAT-PHYS-80 MCH.

Following tools are required for full configuration and commissioning of the Systems:

• Checklist over all required steps for documentation (see Chapter 5)
• Linux PC with LAN access to ACC network, resp. BEA-DAT hardware database, as well as keyboard, monitor and mouse
• microUSB cable to connect MCH from the LinuxPC
• USB-C to USB-A Adapter x2, for keyboard access to CPU and the AM G64 BIOS USB-Stick
• an USB-Keyboard and the AM G64 BIOS USB-Stick
• an extra monitor
• a DP2VGA Adapter for the monitor
• Two LAN patch cables (MCH and CPU)
• Access to BEA-DAT barcode printers (small sdaprn04 and large labels sdaprn03)
• NATview (GSI license available)
• Minicom or similar for serial access to MCH
• Web browser (Internet Explorer for the SD-Data base and Mozzilla for the MCH-UI)
• a power cable

For the moment there is sdlx040 prepared for this type of work.

Take a printed version of the checklist (Chapter 5) and fill out all possible items.
At the very end, after capter 4, prepare two further copies of completly filled capter 5 document.

1. version: collect in a folder
2. version: stick it on the crate
3. version: stick it on the box

Powerbridge provides test protocols for each individual system with all relevant data before or at delivery time.
Besides all serial numbers, which are helpful to identify the crates with their installed modules, the MAC addresses of CPU and MCH are provided.
As a first step, it is recommended to request valid IP addresses and network names for the new products.
NB: All 35 crates from the 2019 tender are already provided with ACC IP addresses.
If other and new systems are not yet registered in the ACC DHCP system, following steps are required:
A mail has to be sent to ACO (Pfister, Vincelli) with the MAC addresses (ETH1 for CPU), a proposed network name and a valid boot server or a valid exemplary FEC name to be copied from.
Example DHCP parameter: server=“fsl00c” filename=“/lobi/pxe/pxelinux.0”
The network names for MCH are always: sdmchXXX
The network names for CPUs (FEC) are always: sddscXXX
Remark: dsc means device stub controller

After delivery, you should check the packaging for damage and make a complaint if necessary.
The usual scope of delivery for a system is the pre-assembled chassis (with PS, MCH and CPU along with slot fillers for better cooling), power cable and a bag with microUSB cable, USB-C adapter, driver CD (sometimes) and NAT Quick Start Guide.

As soon as the IP addresses of MCH and CPU are available, the systems should be registered in the BEA-DAT hardware database by assigning a CID and a barcode.
The database is located here: http://sdlx046.acc.gsi.de:8080/bihw/login
To access the database you need an account (→ H. Bräuning).
First, a DB entry of an existing crate is opened and copied. The name and network name must be adapted and the MAC address, serial number and order number need to be filled.
It is helpful to enter a purpose or the sponsor for the system in the note field.
The Powerbridge test protocol (pdf) should be uploaded and attached to the crate's DB entry.
A large barcode (sdaprn001.acc.gsi.de or sdaprn003.acc.gsi.de) with crate name and serial number shall be printed and sticked onto the right front panel of the crate.

Figure 1: Barcode for MTCA Chassis with Crate-No. and S/N
Figure 2: Barcodes and labels on MTCA chassis and modules
Figure 3: Barcodes and labels on MCH
Figure 4: Barcodes and labels on MTCA chassis and modules
Figure 5: Large barcode on power supply

Same procedure is valid for the CPU, MCH and the power supply module.
Select an existing DB entry, copy it and modify the required fields like module name, MAC, S/N, order number and add some info in the notes field.
With 6mm tape label the CPU with FEC name and IP address (skip the GSI standard part 140.181). Same with MCH, here with name, IP and MAC address.
CPU and MCH require a small barcode (sdaprn002.acc.gsi.de and sdaprn004.acc.gsi.de), which can be placed like in Fig. 2.
If all modules and the crate are entered in the database, then all modules should be virtually added to the crate (as a container).
To do this, press the icon in the component overview. If you do not enter a slot number, it will simply be incremented.

NAT MCH comes without a MAC address sticker. The MAC can be found in the test documents or via command line interface (cli) using the serial port.

1. Connect the MCH to the network (port GbE 1 on the front panel, the middle RJ45 input).
2. Power up the system
3. Connect a microUSB cable to the microUSB port of the MCH (next to the tiny handle)
4. Connect the USB cable to a Linux PC.
   Remark: If older MCH are connected during powering up of the MCH it might enter in bootloader mode instead or will not start at all.
5. A serial device (ttyACM0) should appear in your system. Open a serial terminal session (Putty, Realterm, minicom (as root, n0k1….), etc), parameters: 19200 bps, 8N1.
   

With the USB Connection you can open a terminal on the Linux PC and Login by typing su as “super user”.
You will Need the Password n0k1…. and then you can type “minicom -s”
Example: minicom setup (serial port setup)

After pressing enter a few times, you should see a command line prompt:
nat>
Type “mchcfg”
Select [3]
Press enter until “enable ssh access” appears. Type “y” and enter.
Press enter until reaching: “MCH IP address source:”
Select “2” for DHCP access.
Press “enter” until the end of the cfg menu. Leave menu with “q”.
Type “reboot”
Type “ni” to get the info about network setup:

For ssh access a valid password must be created using “root_pw” on the cli.
Please create the actual BEA password for root access.
The password is required for ssh and web browser access.
The standard login/pw from NAT is root/nat

With correct IP settings and a new password access via web browser is granted.
Just enter the IP or DNS name into the location bar and enter the password.
Check to use http and not https.
The following menu is shown:

Figure 6: MCH website menu

There are most of the time two firmware types to be installed on the MCH.
So a first check, which version is installed and which is available at NATs website is important before doing anything else with the MCH.
The latest firmware should be always checked and stored at sdlx040:/usr/local/NAT

Via web access select “Board Information” and check the MCH firmware version:

Figure 7: MCH Information, here V2.21.7

How to upgrade:

Download latest firmware here: https://www.nateurope.com/services/downloads/software.html

Or find it here: /usr/local/NAT/Firmware/MCH_firmware_2_23_5

Or via Filezilla: ftp.nateurope.com User: natmch PW: natmch Port 21

Change directory to NAT-MCH and download the latest firmware zip, when higher than the actual one.
Unpack the password protected zip (MarsH_mallow).
Older password was LilyOfTheValley. You will need a tar file for the upgrade.
A valid update file in “bin” subdirectory should look like this: mch_fw_2_23_5_webupdate.tar (as example)

1. Select Update MCH from the web access menu Search for the file “mch_fw_2_23_5_webupdate.tar (as example)” with “DURCHSUCHEN”
2. Upload the file and press Update, when the message appears.
3. Wait
4. Press reboot when asked for.
5. Check the updated firmware version after reboot under “Board Information”
   

Remark: firmware update has some impact on DHCP service, which does not work anymore. A power cycle is recommended.

Another important firmware update concerns the HUB firmware version. An important bug can be fixed by upgrading from Microcontroller Version V1.7 to V1.10.

1. Select in the web access main menu the chapter Board Information and check the micro controller firmware version presented in “Hub module information”
   
   Figure 8: MCH Information
2. Select “Update MCH” from the web access menu.
3. Upload the HUB Firmware (e.g. mch_pcie_webupdate.tar)
4. Wait
5. Reboot, when asked for
6. Check the updated firmware version after reboot under “Board Information”
   

Remark: firmware update has some impact on DHCP settings, which does not work anymore. A power cycle is recommended.

Switch to menu item Base Configuration, check all relevant settings and change as follows:

Figure 9: MCH base configuration

Scroll down, press Save and reboot the System.

To prevent illegal CPU MAC addresses (rear ports 1 & 2) in our ACC network the LAN ports of the CPU slot have to be deactivated:
Remark: in 12 slot crates there are no #/1 ports. Leave all on #/0.

Figure 10: Disable LAN ports from CPU on backplane

NB: The virtual LAN options are only required for CUPID configuration.
This prevents incompatible network access and creates a local subnet for the GBE video cameras.
Special attention is required to protect the ACC network from loops etc.
You may also follow the instructions here: https://www-bd.gsi.de/dokuwiki/doku.php?id=projects:cupid:overview&s[]=amc217

First configure the AMC219 switch.
Do not boot or better remove the CPU while doing this.
Activate under “Port on/off” the ports as shown in the figure below:

Figure 11: MCH Port On/Off configuration

Then activate and configure the Port based VLAN under “Port VLAN” as shown in the figure below:

Figure 12: MCH Port VLAN configuration

Only now you may enable the CPU.

=== 4.5.8 PCIe virtual switch settings ===

Figure 13: MCH Port On/Off configuration

Make sure, that all AMC slots have the lanes 8…11 set to NONE.
So far none of our modules supports X8 PCIe links.
The correct settings overcome the issue of disconnected PCIe links after soft reboot.
To guarantee that changes applied to the PCIe settings get recognized by the CPU a cold reboot is recommend.
Remark: in 12 slot crates there are no 8…11 ports. Leave all on 4…7.
Be aware: wrong settings here have impact on the Linux systemd boot device name (here: enp15s0)

1. Extract the supplied ZIP archive to the root folder of a USB Flash drive.
2. Power on the board and when prompted press “F2” to enter the BIOS Setup menus.
   NOTE: it may be necessary to press “F2” more than once for it to be recognised and there may be a short delay before the Setup menus are displayed.
3. Go to “Main”→“Boot Features” and change the “Restricted Setup” option to “Disabled”
4. Press “F4” to save these changes and exit the Setup menus. Allow the board to restart and press “F2” to enter the BIOS Setup menus again.
5. Go to “Chipset” → “PCH-IO Configuration” → “Security Configuration” and change the “BIOS Lock” option to “Disabled”
6. Press “F4” to save these changes and exit the Setup menus.
7. Power cycle the board and boot to the EFI shell. If the EFI shell is not the default boot device then it can be selected by pressing “F12” and selecting EFI Shell.
8. At the EFI shell prompt, select the USB Flash Drive, e.g.
   > fs0: (alternative “fs1:” if the USB stick is plugged in in the other port)
   Remark: the Keyboard maybe default english, the german keyboard uses identical keys, except for the “:” that corresponds with upper case Ö.
9. Start the update by executing the BIOS.nsh shell script. e.g.
   > BIOS
   
10. The BIOS update will start and progress will be displayed on the console. There is no need to press any keys.
    

When the update has completed, power cycle the board.
The settings changed in steps 3) and 4) are automatically reset to their default state after the update.

The CPU is equipped with an pre-installed CentOS for first tests etc., but is not required for future use.
It can be entered after boot with gsi/centos.
This boot device should be disabled.

Before the CPU can be used with PXE based network boot some settings have to be set within the CPUs BIOS.

1. Enter BIOS at boot time by pressing F2 or via F12 and then Enter Setup.
2. First check date and time.
3. In the submenu (Boot features?) of the first BIOS page you must activate PXE Boot, Auto retry PXE Boot and the Front Panel ETH 1 port as shown in Fig. 15. The other ETH ports should be disabled.
   
   Figure 14: BIOS settings for PXE Access
   
4. Press Save & Exit, [F4] leave the BIOS and reboot again into the BIOS.
5. Switch to the Boot page and select the now available LAN port as first boot device. All other devices can be disabled.
6. Save and Exit [F4]

To enable PXE boot the new CPU must be correctly set-up on the tftp-boot server.
The default tftp-boot server can be accessed via login on asl740 or after update to Rocky 8 on asl751.
This requires an ACC network account. Without this preparation, the CPU will not boot from the network.
Execute the following steps:
ssh username@asl740.acc.gsi.de (or asl751)
cd /common/tftp/lobi/pxe/pxelinux.cfg
Remark: There is a “space” between “cd” and “/common”

Look for the latest config file. For the time being the filename starts with R8 and ends with the form factor like UTCA.

Use the script: ./createPxeLink.sh <fec-name> <config-file>
e.g. ./createPxeLink.sh sddsc194 R9_R5_93_UTCA_NET10.enp15s0
If the data already exists, it can be removed by typing “rm” followed by the fec-name. e.g. “rm sddsc194”
Also the is a HEX-code file correspondig to that, that has to be removed. e.g. “rm 8CB584F0”

or manually:
gethostip <fec-name> e.g. sddsc010
Output of gethostip looks like this:
sddsc010.acc.gsi.de 140.181.132.240 8CB584F0
Create a symbolic link from BootOS-Config File e.g. C7_R14_79_UTCA_NET10.enp15s0 to fec-name
ln -s C7_R14_79_UTCA.enp15s0 sddsc010
Create a symbolic link from fec-name to the HEX IP Code
ln -s sddsc010 8CB584F0

Finally boot the CPU and try a login as root / new…. (R0ckyB….)

Remark: With correct settings of the PCIe switch (4…7) the LAN device is always named by Linux with enp15s0.
Selecting a PXE config file supporting this dedicated LAN port will speed up boot time.
Any mistake in the switch settings or with a 12 slot crate may lead to improper boot and a hang in the Dracut environment.
If unsure select a PXE config file without dedicated LAN port (e.g R9_R5_93_UTCA_NET10).