====== Shared Libraries with FESA ======

This documents provides a straightforward recipe how to use shared libs with FESA and FECs\\
\\
CSCO encourages to link statically as this significantly eases deployment.\\
However, there are cases where vendors neither supply source code nor static libraries.\\
\\
CAEN's HVWrapper, for instance, was only optionally available static for a short
period of time on request.\\
They don't ship static libs anymore - for technical reasons, whatever that actually means.\\
==== Shared Lib Directories on ASL73x and FECs ====
When developing FESA classes on the ASL73x cluster, the directory\\
<code>
/common/fesa/fec/arch/L866_{32,64}/lib
</code>
is available to place the shared libraries.\\
More specifically, ''/common/fesa'' is the mount point of ''fsl00c:/fesa''.\\

On the **FEC**
<code>
fsl00c:/common/fesa/fec/arch/L866_{32,64}</code>
is mounted at
<code>
/fec/arch
</code>
i.e. the libraries are available at\\
<code>
/fec/arch/lib
</code>
It is encouraged to create a subdirectory for each lib, e.g. HVWrapper resides in
<code>
/common/fesa/fec/arch/L866_{32,64}/lib/caen
</code>
with 
<code>
libcaenhvwrapper.so -> libcaenhvwrapper.so.5.77
libcaenhvwrapper.so.5.22
libcaenhvwrapper.so.5.56
libcaenhvwrapper.so.5.61
libcaenhvwrapper.so.5.62
libcaenhvwrapper.so.5.77
</code>
to make testing against different library versions simple.\\

However, since this affects all FECs globally,\\
make sure, that no operational system relying on that \\
lib boots or reloads the FESA binary while pointing\\
to an unwanted version and hooks it up - as this might \\
lead to unexpected behaviour!
==== Search Paths ====
The //required// linker flag
<code>
-L<path-to-lib>
</code>
only provides resolution of symbols and, thus, the specified\\
path **is not** the look-up path for the binary runtime.\\
\\
This is still provided by the system libs\\
and/or the LD_LIBRARY_PATH env variable.\\

However, a third option exists which does not require adminstration of the diskless system.\\
The linker flag 
<code>
-Wl,-rpath=<path1>[,-rpath=<path2>]
</code>
includes hard-coded search paths for the shared lib into the binary.\\
This RPATH segment in the binary looks like the PATH variable string.\\
\\
Here is an example for the 64-bit CaenHvSYx527 FESA binary :
<code>
Dynamic section at offset 0xf38d68 contains 31 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libetherbone.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libxml2.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [librt.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libcaenhvwrapper.so]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]
 0x000000000000000f (RPATH)              Library rpath: [/common/fesa/fec/arch/L866_64/lib/caen:/fec/arch/L866_64/lib/caen]
 ...
</code>
The search order for shared libraries is as follows:
<code>
LD_LIBRARY_PATH (env var) -> RPATH (via linker flag) -> system libs (e.g. via ldconfig for custom libs)
</code>

==== Boilerplate Code ====

When building FESA for a target architecture, the variable ''CPU'' is set to either //i686// or //x86_64//\\
and accessible from the Makefile.
Depending on where linking to the custom vendor shared library happens,\\
this can be leveraged to define the directory name regarding bitness in the appropriate Makefile.specific:
<code>
ifeq "$(CPU)" "i686"
  ARCH_DIR=L866_32
else
  ARCH_DIR=L866_64
endif
</code>
(Currently, that's actually more connected to the distribution instead of the architecture - see RFC)\\
Next create to directory variables that reference the mount point on the ASL73x and the path as seen on the FEC.\\
Mind again that 'caen' is specific to this example.
<code>
LIB_PREFIX_LOCAL=/common/fesa
LIB_FEC_PATH_LOCAL=/fec/arch/$(ARCH_DIR)/lib/caen
LIB_FEC_PATH=/fec/arch/lib/caen
</code>
Two LIB_FEC* paths are required to make the binary run on the FEC and locally on the ASL73* cluster.\\
We can now append all appropriate paths to linker and rpath flags.
<code>
LINKER_FLAGS += -L$(LIB_PREFIX_LOCAL)$(LIB_FEC_PATH_LOCAL) -lcaenhvwrapper
LINKER_FLAGS += -Wl,-rpath=$(LIB_PREFIX_LOCAL)$(LIB_FEC_PATH_LOCAL),-rpath=$(LIB_FEC_PATH)
</code>
There must be no whitespace around the commas!\\
Otherwise, the linker will complain.\\
\\
This allows to test the binary on the ASL73x cluster and run it on the FEC.
==== Proposal ====
I suggest changing the directory structure for shared libs from
<code>
fec/arch/L866_{32,64}/lib/<libdir>/
</code>
to
<code>
fec/lib/<libdir>/{i686,x86_64}/
</code>
or
<code>
fec/lib/{i686,x86_64}/<libdir>/
</code>
for two reasons. (Where <libdir> is your directory, e.g. //caen//, in this example)\\
\\
First, we will have other distributions soon, i.e. instead of differentiating only\\
between L866_32 and L866_64, there will be more (CentOS7, Systems with X, etc...)\\
However, only the architecture really matters.

Second, we can directly leverage the ''CPU'' variable set during build time to \\
generate the appropriate target directories.