Shared Libraries with FESA

This documents provides a straightforward recipe how to use shared libs with FESA and FECs

CSCO encourages to link statically as this significantly eases deployment.
However, there are cases where vendors neither supply source code nor static libraries.

CAEN's HVWrapper, for instance, was only optionally available static for a short period of time on request.
They don't ship static libs anymore - for technical reasons, whatever that actually means.

When developing FESA classes on the ASL73x cluster, the directory

/common/fesa/fec/arch/L866_{32,64}/lib

is available to place the shared libraries.
More specifically, /common/fesa is the mount point of fsl00c:/fesa.

On the FEC

fsl00c:/common/fesa/fec/arch/L866_{32,64}

is mounted at

/fec/arch

i.e. the libraries are available at

/fec/arch/lib

It is encouraged to create a subdirectory for each lib, e.g. HVWrapper resides in

/common/fesa/fec/arch/L866_{32,64}/lib/caen

with

libcaenhvwrapper.so -> libcaenhvwrapper.so.5.77
libcaenhvwrapper.so.5.22
libcaenhvwrapper.so.5.56
libcaenhvwrapper.so.5.61
libcaenhvwrapper.so.5.62
libcaenhvwrapper.so.5.77

to make testing against different library versions simple.

However, since this affects all FECs globally,
make sure, that no operational system relying on that
lib boots or reloads the FESA binary while pointing
to an unwanted version and hooks it up - as this might
lead to unexpected behaviour!

The required linker flag

-L<path-to-lib>

only provides resolution of symbols and, thus, the specified
path is not the look-up path for the binary runtime.

This is still provided by the system libs
and/or the LD_LIBRARY_PATH env variable.

However, a third option exists which does not require adminstration of the diskless system.
The linker flag

-Wl,-rpath=<path1>[,-rpath=<path2>]

includes hard-coded search paths for the shared lib into the binary.
This RPATH segment in the binary looks like the PATH variable string.

Here is an example for the 64-bit CaenHvSYx527 FESA binary :

Dynamic section at offset 0xf38d68 contains 31 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libetherbone.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libxml2.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [librt.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libcaenhvwrapper.so]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]
 0x000000000000000f (RPATH)              Library rpath: [/common/fesa/fec/arch/L866_64/lib/caen:/fec/arch/L866_64/lib/caen]
 ...

The search order for shared libraries is as follows:

LD_LIBRARY_PATH (env var) -> RPATH (via linker flag) -> system libs (e.g. via ldconfig for custom libs)

When building FESA for a target architecture, the variable CPU is set to either i686 or x86_64
and accessible from the Makefile. Depending on where linking to the custom vendor shared library happens,
this can be leveraged to define the directory name regarding bitness in the appropriate Makefile.specific:

ifeq "$(CPU)" "i686"
  ARCH_DIR=L866_32
else
  ARCH_DIR=L866_64
endif

(Currently, that's actually more connected to the distribution instead of the architecture - see RFC)
Next create to directory variables that reference the mount point on the ASL73x and the path as seen on the FEC.
Mind again that 'caen' is specific to this example.

LIB_PREFIX_LOCAL=/common/fesa
LIB_FEC_PATH_LOCAL=/fec/arch/$(ARCH_DIR)/lib/caen
LIB_FEC_PATH=/fec/arch/lib/caen

Two LIB_FEC* paths are required to make the binary run on the FEC and locally on the ASL73* cluster.
We can now append all appropriate paths to linker and rpath flags.

LINKER_FLAGS += -L$(LIB_PREFIX_LOCAL)$(LIB_FEC_PATH_LOCAL) -lcaenhvwrapper
LINKER_FLAGS += -Wl,-rpath=$(LIB_PREFIX_LOCAL)$(LIB_FEC_PATH_LOCAL),-rpath=$(LIB_FEC_PATH)

There must be no whitespace around the commas!
Otherwise, the linker will complain.

This allows to test the binary on the ASL73x cluster and run it on the FEC.

I suggest changing the directory structure for shared libs from

fec/arch/L866_{32,64}/lib/<libdir>/

to

fec/lib/<libdir>/{i686,x86_64}/

or

fec/lib/{i686,x86_64}/<libdir>/

for two reasons. (Where <libdir> is your directory, e.g. caen, in this example)

First, we will have other distributions soon, i.e. instead of differentiating only
between L866_32 and L866_64, there will be more (CentOS7, Systems with X, etc…)
However, only the architecture really matters.

Second, we can directly leverage the CPU variable set during build time to
generate the appropriate target directories.