UDP Packet Header Formats

Load Balancer Header

Following is the C routine to write the header that the LB reads, and eventually strips off, in order to direct the packet to the proper destination.

Currently using version = 2 and protocol = 1.

        /**
         * Set the Load Balancer header data.
         * The first four bytes go as ordered.
         * The entropy goes as a single, network byte ordered, 16-bit int.
         * The tick goes as a single, network byte ordered, 64-bit int.
         *
         * Bit#
         *  0                   1                   2                   3
         *  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
         *  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
         *  |       L       |       B       |    Version    |    Protocol   |
         *  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
         *  3               4                   5                   6
         *  2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3
         *  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
         *  |              Rsvd             |            Entropy            |
         *  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
         *  6                                               12
         *  4 5       ...           ...         ...         0 1 2 3 4 5 6 7
         *  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
         *  |                                                               |
         *  +                              Tick                             +
         *  |                                                               |
         *  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
         *
         * @param buffer   buffer in which to write the header.
         * @param tick     unsigned 64 bit tick number used to tell the load balancer
         *                 which backend host to direct the packet to.
         * @param version  version of this software.
         * @param protocol protocol this software uses.
         * @param entropy  entropy field used to determine destination port.
         */
        static void setLbMetadata(char* buffer, uint64_t tick, int version, int protocol, int entropy) {
            *buffer     = 'L';
            *(buffer+1) = 'B';
            *(buffer+2) = version;
            *(buffer+3) = protocol;
            // Put the data in network byte order (big endian)
            *((uint16_t *)(buffer + 6)) = htons(entropy);
            *((uint64_t *)(buffer + 8)) = htonll(tick);
        }

Reassembly Header

Following is the C routine to write the header that passes through the LB and is used by the receiving software to reassemble UDP packets into their original buffer.

Currently using version = 2.

      /**
        * Set the Reassembly Header data.
        * The first 16 bits go as ordered. The dataId is put in network byte order.
        * The offset, length and tick are also put into network byte order.</p>
        *
        *  Bit#
        *  0                   1                   2                   3
        *  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
        *  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
        *  |Version|        Rsvd           |            Data-ID            |
        *  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
        *  |                         Buffer Offset                         |
        *  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
        *  |                         Buffer Length                         |
        *  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
        *  |                                                               |
        *  +                             Tick                              +
        *  |                                                               |
        *  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
        *
        * @param buffer  buffer in which to write the header.
        * @param offset  byte offset into full buffer payload.
        * @param length  total length in bytes of full buffer payload.
        * @param tick    64 bit tick number used to tell the load balancer
        *                which backend host to direct the packet to. Necessary to
        *                disentangle packets from different ticks at one destination
        *                as there may be overlap in time.
        * @param version the version of this software.
        * @param dataId  the data source id number.
        */
        static void setReMetadata(char* buffer, uint32_t offset, uint32_t length,
                                  uint64_t tick, int version, uint16_t dataId) {

            buffer[0] = version << 4;

            *((uint16_t *)(buffer + 2))  = htons(dataId);
            *((uint32_t *)(buffer + 4))  = htonl(offset);
            *((uint32_t *)(buffer + 8))  = htonl(length);
            *((uint64_t *)(buffer + 12)) = htonll(tick);
        }

Format of Sync Packet to Control Plane

Following is the C routine to write the packet send to the LB's control plane to inform it of the latest event number that was sent to the data plane.

Currently using version = 2.

        /**
         * Set the data for a synchronization message sent directly to the load balancer's control plane.
         * The first 3 fields go as ordered. The src id, event number, event rate and time are all
         * put into network byte order.</p>
         *
         *   Bit#
         *    0                   1                   2                   3
         *    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
         *    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
         *    |       L       |       C       |    Version    |      Rsvd     |
         *    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
         *    |                           EventSrcId                          |
         *    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
         *    |                                                               |
         *    +                          EventNumber                          +
         *    |                                                               |
         *    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
         *    |                         AvgEventRateHz                        |
         *    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
         *    |                                                               |
         *    +                          UnixTimeNano                         +
         *    |                                                               |
         *    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
         *
         * @param buffer   buffer in which to write the data.
         * @param version  version of this software.
         * @param srcId    id number of this data source.
         * @param evtNum   unsigned 64 bit event number used to tell the load balancer
         *                 which backend host to direct the packet to. This message
         *                 is telling the load balancer that this application has
         *                 already sent this, latest, event.
         * @param evtRate  in Hz, the rate this application is sending events
         *                 to the load balancer (0 if unknown).
         * @param nanos    at what unix time in nanoseconds was this message sent (0 if unknown).
         */
        static void setSyncData(char* buffer, int version, uint32_t srcId,
                                uint64_t evtNum, uint32_t evtRate, uint64_t nanos) {
            buffer[0] = 'L';
            buffer[1] = 'C';
            buffer[2] = version;

            // Put the data in network byte order (big endian)
            *((uint32_t *)(buffer + 4))  = htonl(srcId);
            *((uint64_t *)(buffer + 8))  = htonll(evtNum);
            *((uint32_t *)(buffer + 16)) = htonl(evtRate);
            *((uint64_t *)(buffer + 20)) = htonll(nanos);
         }