How to setup ejfat nodes

1. Install BOOST with:

   sudo apt-get install lib boost-all-dev

2. Install ZMQ with:

   sudo apt install libzmq3-dev

3. For ERSAP install protobuf:

   sudo apt install libprotobuf-dev protobuf-compiler

4. For pip (python installer for python) install:

   sudo apt install pip

5. For prometheus:
   1. installation:

      sudo apt install prometheus

      sudo apt install prometheus-node-exporter

   2. python client libraries:

      pip install prometheus-client

      pip install prometheus-api-client

   3. exporter start up:

      sudo systemctl start prometheus-node-exporter

      sudo systemctl enable prometheus-node-exporter

6. For increased UDP buffer & queue sizes:
   1. For the long term write the following lines into the sysctl.conf file:

      net.core.rmem_max=25000000

      net.core.wmem_max=25000000

      net.core.netdev_max_backlog=65536

   2. To make it immediately effective, execute:

      sudo sysctl -w net.core.rmem_max=25000000

      sudo sysctl -w net.core.wmem_max=25000000

      sudo sysctl -w net.core.netdev_max_backlog=65536

      ifconfig <ifname> txqueuelen 10000

7. To mount /daqfs:
   1. sudo mkdir /daqfs
   2. edit /etc/fstab to include the line:

      dubhe:/raid/daqfs          nfs     rw,bg           0       0

   3. sudo apt-get install nfs-common
   4. sudo mount /daqfs
8. To change one's home directory to /daqfs/home/myUserName:
   1. Edit the /etc/passwd file to include a line something like:

      myUserName:x:2101:1208:John Doe,,,:/daqfs/home/myUserName:/bin/bash

      where 2101 is user id and 1208 is epsci group. Hint: set your user id and group ids on the ejfat nodes to be the same as it is on the computer center machines or there will be an issue when you access files served by computer center like /daqfs/home or /group/...
9. When sending UDP packets:

• • See UDP and IP Fragmentation and Wikipedia IP Fragmentation. The basic low-down is that IP fragmentation of UDP packets should be avoided at all costs since it will slow things down, increase the amount of data sent, but most importantly in our case, fragmented packets will most likely be dropped. Look at the following section from the second link:

    In IPv4, hosts must make a best-effort attempt to reassemble fragmented IP packets with a total reassembled size of up to 576 bytes. They may also attempt to reassemble fragmented IP packets larger than 576 bytes, but they are also permitted to silently discard such larger packets. Applications are recommended to refrain from sending packets larger than 576 bytes unless they have prior knowledge that the remote host is capable of accepting or reassembling them.

    In IPv6, hosts must make a best-effort attempt to reassemble fragmented packets with a total reassembled size of up to 1500 bytes, larger than IPv6's minimum MTU of 1280 bytes. Fragmented packets with a total reassembled size larger than 1500 bytes may optionally be silently discarded. Applications relying upon IPv6 fragmentation to overcome a path MTU limitation must explicitly fragment the packet at the point of origin; however, they should not attempt to send fragmented packets with a total size larger than 1500 bytes unless they know in advance that the remote host is capable of reassembly.

• • Yet another reason to avoid IP fragmentation: unless you know your network can handle jumbo frames, break up your data into chunks of 1400 bytes. This is because you don't want to fragment your packets on high speed networks. On fast networks the 16-bit packet-id can overflow within the reassembly timeframe and - if the checksum matches or is not set - fragments of different packets could be reassembled.

• • As far as jumbo frames go, most network devices that support it do so for a size of 9,216 bytes. This isn’t standardized like Ethernet’s 1,500 byte MTU, though, so you want to check with your particular manufacturer on the largest frame size their devices support and how to configure the changes. Even within a single manufacturer’s line of network products, the MTU capabilities may vary greatly, so it is important to do a thorough investigation of all your devices in the communication paths and validate their settings.