ds:projects:unimon:gateware
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
ds:projects:unimon:gateware [2021/10/12 14:48] rgeissler |
ds:projects:unimon:gateware [2023/02/09 10:10] (current) rgeissler |
||
|---|---|---|---|
| Line 1: | Line 1: | ||
| ====== UniMon Gateware Documentation ====== | ====== UniMon Gateware Documentation ====== | ||
| - | This documentation was automatically generated from Latex. | + | This documentation was automatically generated from the Markdown file '' |
| - | Any changes should be applied to the Latex sources | + | Any changes should be applied to the Markdown file, which can be converted to DokuWiki format using the script '' |
| - | The Latex sources | + | |
| The documentation is also available as a PDF: {{: | The documentation is also available as a PDF: {{: | ||
| + | |||
| + | An up to date PDF version can be downloaded from the CI/CD section: https:// | ||
| + | |||
| + | Documentation that is common to multiple FPGA based projects can be found here: [[: | ||
| + | It is also available as a PDF: {{: | ||
| ===== Resources ===== | ===== Resources ===== | ||
| - | All of the code of this project, the helper scripts | + | The code of this project and also the source of this documentation are under version control in a Git repository whose upstream is:\\ |
| + | https:// | ||
| + | The relevant branch is //master//. | ||
| + | |||
| + | Common code is included as a Git submodule of the main Git repository. The upstream of the submodule is:\\ | ||
| + | https:// | ||
| + | The relevant branch is // | ||
| + | |||
| + | Installation scripts to set up a Gitlab runner for continuous integration including all necessary software to build the gateware can be found in a Git repository whose upstream is:\\ | ||
| + | https:// | ||
| + | The relevant branch is // | ||
| + | |||
| + | ===== 1 Introduction ===== | ||
| + | |||
| + | This document describes the gateware (= FPGA firmware) implementation of the UniMon Tank Signal Monitor. | ||
| + | |||
| + | The system uses 4 IOxOS ADC_3117 FMC cards which are mounted on two OpenHardware AFC v4.0 FMC carriers. Each of the FMCs provides 20 ADCs with a resolution of 16 bits. The sampling rate of the ADCs is adjustable between 1 kHz and 5 MHz. Each ADC input is individually configurable as a differential or single ended and different gain settings are available. | ||
| + | |||
| + | {{: | ||
| + | |||
| + | **Figure 1.1:** UniMon crate in the HKR | ||
| + | |||
| + | The gateware of each AFC stores the data stream of respectively 40 ADCs to a ring buffer, from which it can be read via PCIe. | ||
| + | |||
| + | ===== 2 Common FPGA based projects documentation ===== | ||
| + | |||
| + | This project incorporates the code from the // | ||
| + | |||
| + | ==== 2.1 Common monitoring and control features ==== | ||
| + | |||
| + | Documentation about the register bank, the architecture information storage and the observer can be found here:\\ | ||
| + | https:// | ||
| + | |||
| + | ==== 2.2 FPGA Observer ==== | ||
| + | |||
| + | There is a expert GUI that can be used together with multiple projects: | ||
| + | https:// | ||
| - | Installation scripts to set up a Gitlab runner for continuous integration including all necessary software to build the gateware can be found in a Git repository whose upstream is: https:// | + | ==== 2.3 Build flow and simulation ==== |
| - | ====== 1 Introduction ====== | + | You can find instructions on how to build and simulate the gateware here:\\ |
| + | https:// | ||
| - | This document describes the gateware (= FPGA firmware) implementation of the UNIMON Main Control Room Voltage Monitor. | + | ==== 2.4 Helper scripts ==== |
| - | The gateware is intended to be used together with a hardware upgrade of the existing system. The current design is implemented for the OpenHardware AFC v3.1 FMC carrier. | + | You can find usefull scripts here:\\ |
| + | https://git.gsi.de/ | ||
| - | The system uses a total of 4 IOxOS ADC_3117 FMC cards. Each of them provides 20 ADCs with a maximum sampling rate of 5 MHz and a resolution of 16 bits. | + | ==== 2.5 Continuous integration environment ==== |
| - | There are two AMC FMC carriers which carry respectively two FMC cards. The gateware of each AMC card stores | + | Information about the continuous integration setup can be found here:\\ |
| + | https:// | ||
| - | The sampling rate of the ADCs is adjustable between 1 kHz and 5 MHz. Due to the parallel storage of the data streams in a ring buffer, the sampling rates of the all the ADCs mounted on a single AMC card must be equal. | + | ==== 2.6 Programming |
| - | Each ADC input is individually configurable as differential (two wires) or as single ended relative | + | You can find instructions on how to program |
| + | https://git.gsi.de/ | ||
| - | ====== 2 Peripheral devices | + | ===== 3 Peripheral devices ===== |
| The gateware communicates with the following devices on each of the IOxOS ADC_3117 FMC cards: | The gateware communicates with the following devices on each of the IOxOS ADC_3117 FMC cards: | ||
| Line 34: | Line 78: | ||
| * 10 * EXAR XRA1405 GPIO expander | * 10 * EXAR XRA1405 GPIO expander | ||
| - | ===== 2.1 Analog Devices LTC2323-16 ADC ===== | + | ==== 3.1 Analog Devices LTC2323-16 ADC ==== |
| Each LTC2323-16 consists of two ADCs that are sampled in parallel. | Each LTC2323-16 consists of two ADCs that are sampled in parallel. | ||
| Line 42: | Line 86: | ||
| Each ADC provides a differential input pair and is directly controlled by the FPGA without any external clock. | Each ADC provides a differential input pair and is directly controlled by the FPGA without any external clock. | ||
| - | The FPGA has to read the samples at a clock speed of 105 MHz. The sampling rate of the ADCs is controlled by the distance between the so called conversion pulses that have to be generated by the FPGA [[#12_references|[1]]]. | + | The FPGA has to read the samples at a clock speed of 105 MHz. The sampling rate of the ADCs is controlled by the distance between the so called conversion pulses that have to be generated by the FPGA [[#6_references|[1]]]. |
| - | Due to this driving | + | Due to this clock generation |
| - | ===== 2.2 EXAR XRA1405 GPIO expander | + | ==== 3.2 EXAR XRA1405 GPIO expander ==== |
| - | The GPIO expanders are used for multiple purposes. Among these are the control of the inputs to each ADC. | + | The GPIO expanders are used for multiple purposes. Among these is the control of the inputs to each ADC. |
| - | ==== 2.2.1 ADC input selection | + | === 3.2.1 ADC input selection === |
| There are dedicated switch devices in front of each of the differential ADC inputs, which can be controlled independently. | There are dedicated switch devices in front of each of the differential ADC inputs, which can be controlled independently. | ||
| Line 61: | Line 105: | ||
| * fixed calibration voltage of + 4.128 V | * fixed calibration voltage of + 4.128 V | ||
| - | ==== 2.2.2 Gain selection | + | === 3.2.2 Gain selection === |
| The gain of each differential ADC input pair can be controlled by a dedicated fixed gain amplifier device which can be programmed to an amplifications of 1, 2, 5 or 10. | The gain of each differential ADC input pair can be controlled by a dedicated fixed gain amplifier device which can be programmed to an amplifications of 1, 2, 5 or 10. | ||
| Line 67: | Line 111: | ||
| In front of each ADC input there is an attenuator with a fixed gain of 0.4. | In front of each ADC input there is an attenuator with a fixed gain of 0.4. | ||
| - | In total the following gains are available: | + | In total the following gains are available |
| ^ amplifier gain ^ input range ^ resolution per LSB ^ | ^ amplifier gain ^ input range ^ resolution per LSB ^ | ||
| Line 75: | Line 119: | ||
| | 10 | +/- 1.024 V | 31.25 {{: | | 10 | +/- 1.024 V | 31.25 {{: | ||
| - | ====== 3 Gateware implementation | + | ===== 4 Gateware implementation ===== |
| - | ===== 3.1 Clocking | + | ==== 4.1 Clocking ==== |
| The gateware uses only one primary clock. | The gateware uses only one primary clock. | ||
| - | ==== 3.1.1 PCIe reference clock ==== | + | === 4.1.1 PCIe reference clock === |
| The PCIe reference clock has a nominal frequency of 100 MHz and feeds the reference clock input of the PCIe IP core by Xilinx, which contains a PLL producing a 125 MHz output clock for the AXI interface named // | The PCIe reference clock has a nominal frequency of 100 MHz and feeds the reference clock input of the PCIe IP core by Xilinx, which contains a PLL producing a 125 MHz output clock for the AXI interface named // | ||
| Line 93: | Line 137: | ||
| The SDRAM interface IP core contains a MMCM which generates a 100 MHz clock named // | The SDRAM interface IP core contains a MMCM which generates a 100 MHz clock named // | ||
| - | ===== 3.2 Resets | + | ==== 4.2 Resets ==== |
| - | ==== 3.2.1 PLL not in lock ==== | + | === 4.2.1 PLL not in lock === |
| As long as the PLL in the MMCM producing the main processing clock //clk_100// is not yet in lock, the design is held in reset. After this the lock should be stable until the next power cycle. | As long as the PLL in the MMCM producing the main processing clock //clk_100// is not yet in lock, the design is held in reset. After this the lock should be stable until the next power cycle. | ||
| - | ==== 3.2.2 Reset button | + | === 4.2.2 PCIe reset === |
| + | |||
| + | The design will be reset whenever the PCIe connection is re-initialized. This happens e.g. when the FEC is rebooted. | ||
| + | |||
| + | === 4.2.3 Reset button === | ||
| There is a push button labeled //RST// at the center of the AFC front panel which is connected to the microcontroller for the MMC firmware. The OpenMMC firmware forwards a button press with a duration of at least two seconds to the FPGA pin AG26 as an active low signal to initiate a reset of the gateware. | There is a push button labeled //RST// at the center of the AFC front panel which is connected to the microcontroller for the MMC firmware. The OpenMMC firmware forwards a button press with a duration of at least two seconds to the FPGA pin AG26 as an active low signal to initiate a reset of the gateware. | ||
| - | Resetting the FPGA leads to the loss of the PCIe connection. To re-enable the connection, the CPU unit has to be power cycled. | + | Resetting the FPGA leads to the loss of the PCIe connection. To re-enable the connection, the FEC has to be rebooted. |
| - | ===== 3.3 Data flow diagram | + | ==== 4.3 Data flow diagram ==== |
| - | {{: | + | {{: |
| - | **Figure | + | **Figure |
| - | Figure | + | Figure |
| * processing clocks and clock domain crossings | * processing clocks and clock domain crossings | ||
| Line 119: | Line 167: | ||
| * data width conversions of AXI connections | * data width conversions of AXI connections | ||
| - | ==== 3.3.1 SPI Interface ==== | + | ==== 4.4 PCIe Interface ==== |
| - | + | ||
| - | ===== 3.4 PCIe Interface | + | |
| This gateware uses the Xilinx IP core // | This gateware uses the Xilinx IP core // | ||
| Line 131: | Line 177: | ||
| The PCIe reference clock is routed to the FPGA via the MMC firmware and is configured to be driven by the 100 MHz //FCLKA// clock coming from the AMC connector. | The PCIe reference clock is routed to the FPGA via the MMC firmware and is configured to be driven by the 100 MHz //FCLKA// clock coming from the AMC connector. | ||
| - | ===== 3.5 SDRAM interface | + | ==== 4.5 SDRAM interface ==== |
| For the communication with the SDRAM, an IP core by Xilinx is used. The clock frequency of the SDRAM interface’s AXI bus is 100 MHz, but has a different source so that an AXI clock converter is used to connect it to the rest of the AXI infrastructure which is clocked at the main processing clock of 100 MHz. | For the communication with the SDRAM, an IP core by Xilinx is used. The clock frequency of the SDRAM interface’s AXI bus is 100 MHz, but has a different source so that an AXI clock converter is used to connect it to the rest of the AXI infrastructure which is clocked at the main processing clock of 100 MHz. | ||
| - | ====== 4 Build flow and simulation ====== | + | ==== 4.6 Observer |
| - | The build flow is designed and tested to be run on a Linux operation system. The bitstream generation should also work on a Windows installation, | + | This project incorporates the observer interface from the //FPGA_Common// Git submodule (see https://git.gsi.de/ |
| - | ===== 4.1 Prerequisites ===== | + | The following signals are connected to the observer inputs: |
| - | An installation | + | ^ |
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| - | You can set your preferred Vivado version in the script '' | + | The rest of the multiplexer inputs are connected to zero. |
| - | ===== 4.2 Build flow ===== | + | ===== 5 Gateware software interface |
| - | ==== 4.2.1 Scripted build flow ==== | + | The communication between the gateware and the software takes place via a PCIe driver by Xilinx called XDMA. There is only one PCIe Bar in use in the gateware which maps the memory space to different physical memories on the AMC board. |
| - | For a completely automatic script based build flow without using the Vivado GUI proceed as follows: | + | The following mapping is applied: |
| - | | + | ^ |
| - | | + | | '' |
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| - | A project will be generated in the folder '' | ||
| - | ==== 4.2.2 Vivado GUI based build flow ==== | + | **Table 5.1:** Memory mapping |
| - | === Via a Bash script === | + | The // |
| + | https:// | ||
| - | * navigate to the root folder of the repository in a terminal | + | ==== 5.1 PCIe Driver ==== |
| - | * type '' | + | |
| - | This will open the Vivado GUI and set up a project, which can take some minutes. The project will be generated in the folder '' | + | Read and write accesses are mapped to virtual file accesses: |
| - | === Via the Vivado GUI === | + | * '' |
| + | * '' | ||
| - | If you intend to use the Vivado GUI itself to set up the project proceed as follows: | + | === 5.1.1 Reading from a register === |
| - | * open Vivado | + | Example |
| - | * use the TCL console | + | |
| - | * type '' | + | |
| - | ===== 4.3 Simulation ===== | + | < |
| + | uint32_t address | ||
| + | int fd = open("/ | ||
| + | lseek(fd, address, SEEK_SET); | ||
| + | uint64_t value; | ||
| + | read(fd, &value, sizeof(uint64_t)); | ||
| + | </ | ||
| + | === 5.1.2 Writing to a register | ||
| - | ==== 4.3.1 Vivado GUI based simulation ==== | + | It is important to write the whole register width of 64 bits. If a register has less than 64 bits, the unused MSBs have to be written to any value. 32 bit write accesses will not have any effect. |
| - | Prerequisite: an existing Vivado project see [[# | + | Example in C: |
| - | Click on //Run Simulation// in the Vivado GUI. | + | < |
| + | uint64_t value = 42; | ||
| + | uint32_t address = 0x00000400; | ||
| + | int fd = open("/dev/xdma0_h2c_0", | ||
| + | lseek(fd, address, SEEK_SET); | ||
| + | write(fd, &value, sizeof(uint64_t)); | ||
| + | </code> | ||
| + | === 5.1.3 Reading of scope data === | ||
| - | ==== 4.3.2 Scripted simulation ==== | + | Example in C: |
| - | The scripted simulation checks that the simulation results match a predefined reference pattern. | + | < |
| + | uint32_t address = 0x80000000; | ||
| + | int fd = open("/ | ||
| + | lseek(fd, address, SEEK_SET); | ||
| + | char data[1024]; | ||
| + | read(fd, data, 1024); | ||
| + | </ | ||
| + | This example reads 1024 bytes of data to an array. For bigger data blocks, instead of using an array, you will probably prefer | ||
| - | * navigate to the root folder of the repository in a terminal | + | ==== 5.2 Scope memory ==== |
| - | * type '' | + | |
| - | * you will find the output files of the simulation in the folder '' | + | |
| - | ==== 4.3.3 Peripherals simulation models ==== | + | The complete SDRAM is used for the scope memory. The ADC data is stored in the following format: |
| - | The toplevel simulation includes a Verilog simulation model from Micron, the manufacturer of the AFC’s SDRAM, which allows the simulation of the behavior | + | ^ |
| + | | '' | ||
| + | | '' | ||
| + | | | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | | | ||
| - | The SDRAM interface IP needs an initial calibration process which finishes after about 120 us. If the communication to the SDRAM is of interest the simulation time should be chosen to be longer than that. | ||
| - | ====== | + | **Table |
| - | The communication between the gateware inside the FPGA and the software | + | The scope is a free running |
| - | The following mapping is applied: | + | The scope memory can hold up to {{:ds: |
| - | ^ **address**^ | + | ==== 5.3 Register map ==== |
| - | | | + | |
| - | | | + | |
| - | | | + | |
| - | **Table | + | === 5.3.1 Status registers === |
| - | ===== 5.1 Scope memory ===== | + | The following status registers can be read by software: |
| - | The complete SDRAM is used for the scope memory. The ADC data is stored in the following format: | + | ^ index^ |
| + | | '' | ||
| + | | '' | ||
| + | | …| | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| - | ^ | ||
| - | | '' | ||
| - | | '' | ||
| - | | | ||
| - | | '' | ||
| - | | '' | ||
| - | | '' | ||
| - | | '' | ||
| - | | '' | ||
| - | | | ||
| - | **Table 5.2:** ADC data storage format | + | **Table 5.3:** List of status registers |
| - | The scope is a free running ring buffer. The write pointer is incremented with each incoming | + | == 0 - 39: latest |
| - | The scope memory | + | This register holds the latest ADC data sample. |
| + | |||
| + | == 48: Scope next write address == | ||
| + | |||
| + | Address where the next data sample will be stored during the scope’s capturing process. The recently stored ADC data can be found below this address, or - in case of a ringbuffer wraparound - also below the end of the memory region. | ||
| + | |||
| + | == 111: SDRAM initial calibration complete == | ||
| + | |||
| + | The communication | ||
| + | |||
| + | == 124: build timestamp == | ||
| + | |||
| + | Time when the bitstream was created. This information can be used to identify the gateware | ||
| + | |||
| + | Format: | ||
| + | |||
| + | ^ bits 31 - 27 ^ bits 26 - 23 ^ bits 22 - 17 | ||
| + | | day | ||
| + | |||
| + | == 125: FPGA serial number == | ||
| + | |||
| + | The XDMA PCIe driver by Xilinx numbers | ||
| + | |||
| + | == 126: module ID == | ||
| + | |||
| + | The module ID can be used to identify the type of the current bitstream.\\ | ||
| + | The module ID of the UniMon gateware is '' | ||
| + | |||
| + | The fields are defined as follows: | ||
| + | |||
| + | ^ bits 63 - 56 | ||
| + | | minor gateware version | ||
| + | |||
| + | Here is an incomplete list of project IDs: | ||
| + | |||
| + | ^ project ID^project name ^ | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | |||
| + | == 127: magic number == | ||
| - | ===== 5.2 Register map ===== | + | The magic number can be used to determine if the gateware uses the expected register format.\\ |
| + | The value of this register is the same for all module IDs: '' | ||
| - | ==== 5.2.1 Configuration registers | + | === 5.3.2 Configuration registers === |
| The following registers can be written by software: | The following registers can be written by software: | ||
| - | ^ **index**^ **address**^ **bits**^**radix** | + | ^ index^ |
| - | | '' | + | | '' |
| - | | '' | + | | '' |
| - | | '' | + | | '' |
| - | | '' | + | | '' |
| - | | '' | + | | '' |
| - | | '' | + | | '' |
| - | | '' | + | | '' |
| - | | '' | + | | '' |
| - | | '' | + | | '' |
| - | | '' | + | | …| |
| - | | …| | + | | '' |
| - | | '' | + | | '' |
| - | | '' | + | | …| |
| - | | …| | + | | '' |
| - | | '' | + | | '' |
| - | **Table 5.3:** List of configuration registers | ||
| - | === 0: ADC conversion pulse length | + | **Table 5.4:** List of configuration registers |
| + | |||
| + | == 0: ADC conversion pulse length == | ||
| This parameter {{: | This parameter {{: | ||
| Line 268: | Line 395: | ||
| The allowed range for {{: | The allowed range for {{: | ||
| - | === 1: MLVDS calibration delay === | + | == 1: MLVDS calibration delay == |
| The signals from the timing receiver enter the FPGA via the MLVDS lines with virtually no delay, while the corresponding ADC samples will be delayed due to the sampling process. | The signals from the timing receiver enter the FPGA via the MLVDS lines with virtually no delay, while the corresponding ADC samples will be delayed due to the sampling process. | ||
| Line 276: | Line 403: | ||
| The delay can be chosen from the range {{: | The delay can be chosen from the range {{: | ||
| - | === 15: Partial reset === | + | == 16, 24: FMC {0, 1} ADC gain == |
| - | + | ||
| - | Writing a 1 to this register triggers a partial reset on the gateware, which affects the scope and the FMC interfaces. The PCIe interface, the SDRAM interface, the AXI infrastructure and the register bank will not be resetted. | + | |
| - | + | ||
| - | The reset will be automatically lifted so that the register does not have to be written to 0 after initiating a reset. | + | |
| - | + | ||
| - | === 16, 24: FMC {0, 1} ADC gain === | + | |
| There are programmable amplifiers in front of each ADC input, which provide four discrete gains. | There are programmable amplifiers in front of each ADC input, which provide four discrete gains. | ||
| Line 288: | Line 409: | ||
| The following gains are available: | The following gains are available: | ||
| - | ^ value^ | + | ^ value^ |
| - | | 0| | + | | '' |
| - | | 1| | + | | '' |
| - | | 2| | + | | '' |
| - | | 3| 10| +/- 1.024 V| 31.25 {{: | + | | '' |
| This register holds the concatenation of the gain settings of the FMC’s 20 ADCs: | This register holds the concatenation of the gain settings of the FMC’s 20 ADCs: | ||
| Line 302: | Line 423: | ||
| | 39 .. 38| 19| | | 39 .. 38| 19| | ||
| - | === 17, 18, 25, 26: FMC {0, 1} ADC input mux select p / n === | + | == 17, 18, 25, 26: FMC {0, 1} ADC input mux select p / n == |
| The ADCs have differential inputs. The two lanes p and n can be configured separately and individually for each ADC. | The ADCs have differential inputs. The two lanes p and n can be configured separately and individually for each ADC. | ||
| Line 308: | Line 429: | ||
| The following input settings are available: | The following input settings are available: | ||
| - | ^ value^input | + | ^ value^input |
| - | | 0|differential p / n input pin | | + | | '' |
| - | | 1|ground | + | | '' |
| - | | 2|not implemented | + | | '' |
| - | | 3|calibration voltage of +4.128 V | | + | | '' |
| This register holds the concatenation of the input settings of the FMC’s 20 ADCs: | This register holds the concatenation of the input settings of the FMC’s 20 ADCs: | ||
| Line 322: | Line 443: | ||
| | 39 .. 38|19 | | | 39 .. 38|19 | | ||
| - | === 32 - 71: ADC {0 - 39} offset correction summand | + | == 32 - 71: ADC {0 - 39} offset correction summand == |
| Correction summand for a possible offset deviation of the ADC. The offset correction precedes the gain correction. | Correction summand for a possible offset deviation of the ADC. The offset correction precedes the gain correction. | ||
| - | === 72 - 111: ADC {0 - 39} gain correction factor | + | == 72 - 111: ADC {0 - 39} gain correction factor == |
| Correction factor for a possible gain deviation of the ADC. The default value 0x8000 corresponds to a multiplication by 1. The possible correction range is {{: | Correction factor for a possible gain deviation of the ADC. The default value 0x8000 corresponds to a multiplication by 1. The possible correction range is {{: | ||
| - | ==== 5.2.2 Status registers ==== | + | == 127: Reset == |
| - | The following status registers can be read by software: | + | Writing |
| - | + | The reset will be automatically | |
| - | ^ **index**^ | + | |
| - | | '' | + | |
| - | | '' | + | |
| - | | …| | + | |
| - | | '' | + | |
| - | | '' | + | |
| - | | '' | + | |
| - | | '' | + | |
| - | | '' | + | |
| - | | '' | + | |
| - | + | ||
| - | **Table 5.4:** List of status registers | + | |
| - | + | ||
| - | === 0 - 39: latest ADC {0 - 39} data === | + | |
| - | + | ||
| - | This register holds the latest ADC data sample. | + | |
| - | + | ||
| - | === 48: Scope next write address === | + | |
| - | + | ||
| - | Address where the next data sample will be stored during the scope’s capturing process. The recently stored ADC data can be found below this address, or - in case of a ringbuffer wraparound - also below the end of the memory region. | + | |
| - | + | ||
| - | === 124: magic number === | + | |
| - | + | ||
| - | The magic number can be used to identify the position of the following three registers in the register bank. This register contains the value '' | + | |
| - | + | ||
| - | === 125: module ID === | + | |
| - | + | ||
| - | The module ID can be used to identify the type of the current bitstream. The module ID of the UNIMON 5 MHz ADC gateware is '' | + | |
| - | + | ||
| - | === 126: build timestamp === | + | |
| - | + | ||
| - | Time when the bitstream was created. This information can be used to identify the gateware version (together with the Git commit information documented in [[# | + | |
| - | + | ||
| - | Format: | + | |
| - | + | ||
| - | ^ | + | |
| - | | 0 - 5|seconds | + | |
| - | | 6 - 11|minutes | + | |
| - | | 12 - 16|hours | + | |
| - | | 17 - 22|last two decimal digits of the year | | + | |
| - | | 23 - 26|month | + | |
| - | | 27 - 31|day | + | |
| - | + | ||
| - | === 127: FPGA serial number === | + | |
| - | + | ||
| - | The XDMA PCIe driver by Xilinx numbers the devices randomly and is not able to identify the slot number of an AMC board. This register holds the FPGA’s unique serial number and can be used to identify an AMC board. | + | |
| - | + | ||
| - | ====== 6 Extended gateware software interface ====== | + | |
| - | + | ||
| - | Besides the interface documented in [[# | + | |
| - | + | ||
| - | While the productive interface is intended to be kept as downward compatible as possible, the extended interface may be subject to major changes during the development process. | + | |
| - | + | ||
| - | ===== 6.1 Extended register map ===== | + | |
| - | + | ||
| - | ==== 6.1.1 Additional status registers ==== | + | |
| - | + | ||
| - | The following additional status registers can be read by software: | + | |
| - | + | ||
| - | ^ **index**^ | + | |
| - | | '' | + | |
| - | + | ||
| - | **Table 6.1:** List of additional status registers | + | |
| - | + | ||
| - | === 111: SDRAM initial calibration complete === | + | |
| - | + | ||
| - | The communication | + | |
| - | + | ||
| - | ===== 6.2 Architecture information storage ===== | + | |
| - | + | ||
| - | The first seven eights of the Block RAM (see memory mapping, table 5.1) are used to store information about the registers and the gateware version. | + | |
| - | + | ||
| - | ==== 6.2.1 Register information ==== | + | |
| - | + | ||
| - | Information about the 128 configuration registers and the 128 status registers is stored in the third quarter of the Block RAM. Following information is stored for every register: | + | |
| - | + | ||
| - | * name of register (31 bytes) | + | |
| - | * number of bits (1 byte) | + | |
| - | + | ||
| - | ^ | + | |
| - | | '' | + | |
| - | | '' | + | |
| - | | | + | |
| - | | '' | + | |
| - | | '' | + | |
| - | | '' | + | |
| - | | | + | |
| - | | '' | + | |
| - | + | ||
| - | **Table 6.2:** Register information storage format | + | |
| - | + | ||
| - | Table 6.2 shows the storage format of the 256 entries, each of which has a width of 32 bytes. The names are stored as ASCII strings. If a name is shorter than 31 bytes, the remaining bytes are filled with Null characters. If not all registers are in use, a width of 0 bits indicates that a register is not present. | + | |
| - | + | ||
| - | The register information is used by the FPGA Observer software to display the registers in the Register Access tab (see [[# | + | |
| - | + | ||
| - | ==== 6.2.2 Gateware information ==== | + | |
| - | + | ||
| - | The address range from 0x80006000 to 0x80006FFF is used to store information about the gateware | + | |
| - | + | ||
| - | The gateware information is used by the FPGA Observer software to display the information in the Gateware Information tab (see [[# | + | |
| - | + | ||
| - | ====== 7 Test software ====== | + | |
| - | + | ||
| - | ===== 7.1 FPGA Observer ===== | + | |
| - | + | ||
| - | There is a graphical test software intended to be run on the CPU unit. It is implemented in Python using the GTK 3 GUI toolkit. | + | |
| - | + | ||
| - | ==== 7.1.1 Installation and usage ==== | + | |
| - | + | ||
| - | The sources and an installation script can be found under '' | + | |
| - | + | ||
| - | === Installation === | + | |
| - | + | ||
| - | Connect to the CPU unit e.g. via ssh. Clone the // | + | |
| - | + | ||
| - | '' | + | |
| - | + | ||
| - | Install the PCIe driver: | + | |
| - | + | ||
| - | '' | + | |
| - | '' | + | |
| - | + | ||
| - | Install the FPGA Observer software: | + | |
| - | + | ||
| - | '' | + | |
| - | '' | + | |
| - | + | ||
| - | === Usage === | + | |
| - | + | ||
| - | For the PCIe driver to work, the bitstreams of the FPGAs have to be loaded before powering the CPU unit. If that is not the case, power cycle the CPU unit by pulling out the Hot Swap Handle and pushing it in again. A software reboot does not work. | + | |
| - | + | ||
| - | Connect to the CPU unit e.g. via '' | + | |
| - | + | ||
| - | '' | + | |
| - | + | ||
| - | A GUI should open and a choice of FPGA serial numbers should be displayed on the upper left corner. If the list is empty, either the loading of the FPGAs finished after powering the CPU unit or the PCIe driver did not install correctly. The FPGA serial numbers can be used to identify the AMC board you like to access. Choose a serial number and click // | + | |
| - | + | ||
| - | ==== 7.1.2 Register Access tab ==== | + | |
| - | + | ||
| - | {{: | + | |
| - | + | ||
| - | **Figure 7.1:** FPGA Observer - Register Access tab | + | |
| - | + | ||
| - | The names and widths of the registers are read from an information memory region in the FPGA (see [[# | + | |
| - | + | ||
| - | The //read// button reads all the status registers either once or continuously if the // | + | |
| - | + | ||
| - | ==== 7.1.3 Scope tab ==== | + | |
| - | + | ||
| - | {{: | + | |
| - | + | ||
| - | **Figure 7.2:** FPGA Observer - Scope tab | + | |
| - | + | ||
| - | The scope tab displays the signals of all the ADCs that are mounted on a single AFC carrier. | + | |
| - | + | ||
| - | For this purpose the write pointer of the ring buffer is polled and the required number of samples are read from positions before this pointer. | + | |
| - | + | ||
| - | ==== 7.1.4 Gateware Information tab ==== | + | |
| - | + | ||
| - | {{: | + | |
| - | + | ||
| - | **Figure 7.3:** FPGA Observer - Gateware Information tab | + | |
| - | + | ||
| - | The URL of the remote server of the Git repository, the latest commit hash and the latest commit date are read from an information memory region in the FPGA (see [[# | + | |
| - | + | ||
| - | The bitstream generation date is read from status register 126 ’build timestamp’. | + | |
| - | + | ||
| - | ===== 7.2 Test scripts ===== | + | |
| - | + | ||
| - | ==== 7.2.1 PCIe access test script ==== | + | |
| - | + | ||
| - | There is a PCIe access test script under '' | + | |
| - | + | ||
| - | ====== 8 Helper scripts ====== | + | |
| - | + | ||
| - | ===== 8.1 VHDL beautification ===== | + | |
| - | + | ||
| - | There is a script '' | + | |
| - | + | ||
| - | The script expects one parameter: '' | + | |
| - | + | ||
| - | The script applies several corrections and changes to the //Emacs// formatting result: | + | |
| - | + | ||
| - | * correction of the handling of the comparison operator '' | + | |
| - | * correction of the handling of initializations like '' | + | |
| - | * enforcing of spaces around the operators '' | + | |
| - | * no indentation for closing brackets | + | |
| - | * aligning of full comment lines to the indentation level of the following VHDL command | + | |
| - | * indentation with tabs instead of spaces | + | |
| - | + | ||
| - | ===== 8.2 Remote power cycling of the CPU unit ===== | + | |
| - | + | ||
| - | Whenever the bitstream of an FPGA is reloaded, the CPU unit has to be rebooted via its Hot Swap Handle in order to establish a PCIe connection. A software reboot does not work. | + | |
| - | + | ||
| - | An alternative possibility of remote power cycling the CPU unit is via the MCH. | + | |
| - | + | ||
| - | The script '' | + | |
| - | + | ||
| - | The script takes about 60 seconds to complete. | + | |
| - | + | ||
| - | ===== 8.3 Generation of a VHDL file for monitoring and control ===== | + | |
| - | + | ||
| - | The monitoring and control configuration of the gateware is defined by the configuration files '' | + | |
| - | + | ||
| - | The script '' | + | |
| - | + | ||
| - | The script is also executed by the gateware build flow documented in [[# | + | |
| - | + | ||
| - | ===== 8.4 Generation of documentation ===== | + | |
| - | + | ||
| - | ==== 8.4.1 PDF ==== | + | |
| - | + | ||
| - | There is a script '' | + | |
| - | + | ||
| - | ==== 8.4.2 Markdown ==== | + | |
| - | + | ||
| - | There is a script '' | + | |
| - | + | ||
| - | The result of //Pandoc// is postprocessed for multiple reasons: | + | |
| - | + | ||
| - | * conversion of the math syntax to Gitlab’s .md format | + | |
| - | * corrections of the bibliography, | + | |
| - | * corrections of the references to figures, tables and equations | + | |
| - | * enumeration of chapters, sections and subsections | + | |
| - | * adding of captions for figures and equations | + | |
| - | * enumeration of figures, tables and equations | + | |
| - | * generation of a table of contents | + | |
| - | * implementation of citations | + | |
| - | + | ||
| - | Additional documentation which is not included in the Latex sources is appended from the file '' | + | |
| - | + | ||
| - | ==== 8.4.3 DokuWiki ==== | + | |
| - | + | ||
| - | There is a script '' | + | |
| - | + | ||
| - | The script converts the Latex documentation sources to the DokuWiki format in four steps: | + | |
| - | + | ||
| - | - conversion of Latex sources to Markdown | + | |
| - | - preprocessing of Markdown before the conversion to DokuWiki | + | |
| - | - calling //Pandoc// to convert Markdown to DokuWiki | + | |
| - | - postprocessing for correction, extension of functionality and a different style | + | |
| - | + | ||
| - | The preprocessing actions are: | + | |
| - | + | ||
| - | * removing table of contents since it is automatically | + | |
| - | + | ||
| - | The postprocessing actions are: | + | |
| - | + | ||
| - | * conversion or equations to images since DokuWiki can not render equations | + | |
| - | * replacement of HTML tags, Latex color tags, etc. since DokuWiki can not handle them | + | |
| - | * conversion of the reference format | + | |
| - | + | ||
| - | ====== 9 Continuous integration environment ====== | + | |
| - | + | ||
| - | There is a continuous integration environment setup for the // | + | |
| - | + | ||
| - | At the moment the Gitlab Runner is running on the Linux server //sdlx035// located in a server room in the basement. | + | |
| - | + | ||
| - | The benefits of continuous integration are: | + | |
| - | + | ||
| - | * every change will be tested automatically | + | |
| - | * it is ensured that no files are missing in the repository | + | |
| - | * the master branch can be kept functional at any time | + | |
| - | * build results like e.g. bitstreams are automatically generated and can be archived | + | |
| - | + | ||
| - | ===== 9.1 Installation ===== | + | |
| - | + | ||
| - | There is an installation script '' | + | |
| - | + | ||
| - | After the installation, | + | |
| - | + | ||
| - | On the newly installed Gitlab Runner server, open a terminal and type '' | + | |
| - | + | ||
| - | Enter the following information: | + | |
| - | + | ||
| - | * gitlab-ci coordinator URL: e.g. '' | + | |
| - | * gitlab-ci token: enter the registration token copied before | + | |
| - | * gitlab-ci description: | + | |
| - | * gitlab-ci tags: leave empty | + | |
| - | * executor: '' | + | |
| - | + | ||
| - | You can add multiple repositories with different tokens by running '' | + | |
| - | + | ||
| - | ===== 9.2 Pipeline Stages ===== | + | |
| - | + | ||
| - | {{: | + | |
| - | + | ||
| - | **Figure 9.1:** Gitlab: continuous integration pipelines | + | |
| - | + | ||
| - | Each '' | + | |
| - | + | ||
| - | The following pipeline stages are defined: | + | |
| - | + | ||
| - | * documentation | + | |
| - | * simulation | + | |
| - | * FPGA build | + | |
| - | + | ||
| - | {{: | + | |
| - | + | ||
| - | **Figure 9.2:** Gitlab: Pipeline stages | + | |
| - | + | ||
| - | ==== 9.2.1 Documentation ==== | + | |
| - | + | ||
| - | The script '' | + | |
| - | + | ||
| - | The log file of Pdflatex and - if successful - the PDF of the documentation are archived. | + | |
| - | + | ||
| - | ==== 9.2.2 Simulation ==== | + | |
| - | + | ||
| - | The script '' | + | |
| - | + | ||
| - | The log file of the simulation and - if successful - a file with the output from the simulation are archived. | + | |
| - | + | ||
| - | ==== 9.2.3 FPGA build ==== | + | |
| - | + | ||
| - | The script '' | + | |
| - | + | ||
| - | Different log files from synthesis and implementation, | + | |
| - | + | ||
| - | {{: | + | |
| - | + | ||
| - | **Figure 9.3:** Gitlab: Pipeline progress console | + | |
| - | + | ||
| - | ==== 9.2.4 Timing check ==== | + | |
| - | + | ||
| - | The script '' | + | |
| - | + | ||
| - | Since timing failures do not necessarily result in system malfunctions, | + | |
| - | + | ||
| - | ===== 9.3 Build results ===== | + | |
| - | + | ||
| - | For each of the pipeline stages the archiving of build results can be configured for an adjustable time period, which is set to one week. If the period has passed and the build results have been deleted, they can be generated again by restarting the pipeline. | + | |
| - | + | ||
| - | The build results can be downloaded from the Gitlab web front end where they are called //job artifacts// (see figure 9.3). | + | |
| - | + | ||
| - | <color green> | + | |
| - | having to set up a build environment.**</ | + | |
| - | + | ||
| - | ===== 9.4 Settings ===== | + | |
| - | + | ||
| - | You can define individual settings for the CI/CD section of each Git repository in the Gitlab web front end. The following settings should fit for most cases: | + | |
| - | + | ||
| - | * Use git clone to get the recent application code, otherwise the pipelines might fail during git fetch:\\ | + | |
| - | //Settings CI/CD General pipelines Git strategy for pipelines: git clone// | + | |
| - | * Increase the timeout to allow FPGA build to finish in any case:\\ | + | |
| - | //Settings CI/CD General pipelines Timeout: 6h// | + | |
| - | + | ||
| - | ====== 10 Programming and hardware configuration ====== | + | |
| - | + | ||
| - | ===== 10.1 Programming the gateware ===== | + | |
| - | + | ||
| - | ==== 10.1.1 Using a JTAG Switch Module ==== | + | |
| - | + | ||
| - | ===== 10.2 Configuration of the MCH ===== | + | |
| - | + | ||
| - | ==== 10.2.1 Via the MCH’s web interface ==== | + | |
| - | + | ||
| - | === Base configuration === | + | |
| - | + | ||
| - | //MCH global parameter SSH access: enabled// | + | |
| - | This will trigger SSH key generation which takes some minutes to complete. | + | |
| - | + | ||
| - | //PCIe parameter Upstream slot power up delay: 5 sec//\\ | + | |
| - | Delay before the CPU unit will power up on start up. For making sure that the bitstreams are loaded to the AMC’s FPGAs from Flash memory before the CPU unit boots you might have to increase this value. | + | |
| - | + | ||
| - | //PCIe parameter PCIe hot plug delay for AMCs: 0 sec//\\ | + | |
| - | Delay before the AMC boards will power up on start up. | + | |
| - | + | ||
| - | === Switch PCIe x80 === | + | |
| - | + | ||
| - | Set the CPU-Unit as upstream AMC source in ’Virtual Switch 0’: | + | |
| - | + | ||
| - | //PCIe Virtual Switches Upstream AMC: AMC1/ | + | |
| - | (for CPU unit in AMC slot 1) | + | |
| - | + | ||
| - | ==== 10.2.2 Via USB ==== | + | |
| - | + | ||
| - | The most comfortable way of configuring the MCH is via its web interface. If you have accidentally disabled the webserver, set an invalid IP or DHCP configuration or reset the MCH settings to default, you can access the MCH via an USB connection to the micro USB port on the left side of the front panel. | + | |
| - | + | ||
| - | On a Linux PC, connect a micro USB cable and check via '' | + | |
| - | + | ||
| - | Now typing '' | + | |
| - | + | ||
| - | ===== 10.3 Enabling network boot on the CPU unit ===== | + | |
| - | + | ||
| - | Shortly | + | |
| - | + | ||
| - | In the //Main// tab, go to //Boot Features// and select the following (using F6 for enabling and F5 for disabling): | + | |
| - | + | ||
| - | * '' | + | |
| - | * '' | + | |
| - | * '' | + | |
| - | + | ||
| - | In the // | + | |
| - | + | ||
| - | In the //Boot// tab, go to //Legacy// and //Boot Type Order//. There should be an //Others// entry that has to be shifted to the top of the list using F6. In newer BIOS versions, there is a //Boot Option #1// entry in the //Boot// tab, which has to be chosen to //IBA GE Slot 1600 v1513//. | + | |
| - | + | ||
| - | Save the settings by pressing F4. | + | |
| - | + | ||
| - | The CPU unit should boot from network after the next reboot. For the loading of the correct network image, the MAC address of the desired Ethernet port of the CPU unit has to registered in the DHCP server responsible for distributing the locations from which to load the network images | + | |
| - | + | ||
| - | ===== 10.4 Configuration of the timing receiver ===== | + | |
| - | + | ||
| - | The GPIOs of the timing receiver can be controlled via command line commands available in the network boot image of the CPU unit: | + | |
| - | + | ||
| - | * display all available GPIOs: '' | + | |
| - | * enable the outputs to the backplane: '' | + | |
| - | * display the properties of a single GPIO: e.g. '' | + | |
| - | * enable the output of a single GPIO: e.g. '' | + | |
| - | * drive the output of a single GPIO high: e.g. '' | + | |
| - | + | ||
| - | ====== 11 System limitations ====== | + | |
| - | + | ||
| - | ===== 11.1 CPU unit reboot ===== | + | |
| - | + | ||
| - | The FPGA bitstreams have to be loaded before the CPU unit boots, otherwise the PCIe driver will not detect the FPGAs. This can be ensured by setting suitable power up delays in the MCH (see [[# | + | |
| - | + | ||
| - | Whenever a new bitstream is loaded to a FPGA, the CPU unit has to be rebooted either via its Hot Swap Handle or remotely via the MCH (see [[# | + | |
| - | + | ||
| - | ====== 12 References ====== | + | |
| - | + | ||
| - | [1] IOxOS ADC_3117 datasheet, https:// | + | |
| - | [2] Analog Devices LTC2323-16 datasheet, https:// | + | ===== 6 References ===== |
| + | [1] Analog Devices LTC2323-16 datasheet, https:// | ||
| + | [2] IOxOS ADC_3117 datasheet, https:// | ||
ds/projects/unimon/gateware.1634042910.txt.gz · Last modified: 2021/10/12 14:48 by rgeissler
Page Tools
