Make and use tensorflow-lite

This gives a recipe for using a TensorFlow/Keras model in a C++ program via tensorflow-lite.

AL/ML models generated in Keras/Python can be used for inference in C++ via tensorflow-lite. The tensorflow-lite package is meant for embedded systems and does not support all Keras model types. However, it covers a large enough range of them to make it a useful option. The basic formula is:

1. Build, train, and save a model using Keras/Python into a .h5 file
2. Convert the model from .h5 into a Keras save_model format which is a directory with multiple files. Then use tensorflow-lite/Python to convert the save_model into a .tflite file
3. Build the tensorflow-lite source into a shared library for use with C++
4. Use the shared library to make a C++ program than can read the .tflite file and do inference with it

Technically, you can save the model directly into a save_model format in step 1. However, if you already have the model from a previous session saved in .h5 format then this lets you use that. The following assumes you have already done step one and have saved a model into a file mymodel.h5

Convert from .h5 into save_model format

Create a script like the following, replacing mymodel.h5 with the name of your model file.

 #!/usr/bin/env python3
 import tensorflow as tf
 from tensorflow import keras
 
 # load model
 saved_model_file = 'mymodel.h5'
 model = keras.models.load_model( saved_model_file )
 
 saved_model_dir = 'mymodel_save_model'
 model.save( saved_model_dir )
 converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
 tflite_model = converter.convert()
 
 with open('mymodel.tflite', 'wb') as f:
     f.write(tflite_model)

Build Tensorflow Lite dynamic library from source

The Tensorflow source comes with Tensorflow Lite, but the pre-built libraries for Linux do not include this. Presumably because the intended use case is embedded systems which would require you to build it yourself anyway for the target system. The source comes with a nicely set up cmake system and instructions on how to use it to build your project. The issue is that if you want to include tensorflow-lite in a project with a different build system (e.g. scons) then it does not give clear instructions for this. What is misleading is that you can easily build a libtensorflow-lite.a, but there are also many other static libraries that are built and buried in subdirectories of the build tree (e.g. tensorflow/tflite_build/_deps/flatbuffers-build/libflatbuffers.a). These are also required, but trying to link those and the top-level libtensorflow-lite.a library with your source manually results in many undefined reference errors. It seems the linker is sensitive to the order the static libraries are added.

This solution uses the included cmake system to create a dynamic library that can then be used with an external build system. Start by getting the tensorflow source. Note, you will need to keep this source directory around so that the header files are available to your project later so give consideration to where you put it.

 git clone https://github.com/tensorflow/tensorflow.git tensorflow_src_v2.8.0
 cd tensorflow_src_v2.8.0
 git checkout v2.8.0
 export TENSORFLOW_LITE=${PWD}
 cd ../

At this point we have to deviate from the tensorflow documentation. Grab the minimal.cc example from the source, but replace "main" with "dummy_main". This is done so that all of the routines in the example are linked into the shared library, while preventing a symbol for "main()".

 mkdir dummy_src
 cd dummy_src
 cat ${TENSORFLOW_LITE}/tensorflow/lite/examples/minimal/minimal.cc | sed -e 's/main/dummy_main/g' > dummy.cc

Now, create a file CMakeLists.txt with this content:

cmake_minimum_required(VERSION 3.16)
project(tflite C CXX)

add_subdirectory(
  "$ENV{TENSORFLOW_LITE}/tensorflow/lite"
  "${CMAKE_CURRENT_BINARY_DIR}/tensorflow-lite" EXCLUDE_FROM_ALL)

add_library(tflite SHARED dummy.cc)
target_link_libraries(tflite tensorflow-lite)

Go back up one directory and make a build directory (this should be parallel to both the ${TENSORFLOW_LITE} and dummy_src directories).

 cd ..
 mkdir build
 cd build
 cmake3 ../dummy_src
 make -j8

At this point you can copy the libtflite.so file to wherever you want to install it. It is probably the most useful to put it next to the header files.

nb. You will also need the flatbuffers header files from the build directory so copy them as well!

mkdir ${TENSORFLOW_LITE}/lib
cp libtflite.so ${TENSORFLOW_LITE}/lib
cp -rp flatbuffers/include/flatbuffers ${TENSORFLOW_LITE}

You can now delete the build directory if you wish.

Linking libtflite.so to you C++ program

At this point, the TENSORFLOW_LITE environment variable should be set. To illustrate it's use do the following:

export LD_LIBRARY_PATH=${TENSORFLOW_LITE}/lib:${LD_LIBRARY_PATH}
cp ${TENSORFLOW_LITE}/tensorflow/lite/examples/minimal/minimal.cc .
g++ -std=c++14 -I${TENSORFLOW_LITE} -L${TENSORFLOW_LITE}/lib -ltflite -o minimal minimal.cc
minimal mymodel.tflite

This should print some information about the model.