MNN: A Lightweight and Efficient Deep Learning Inference Framework

MNN is Alibaba's open source lightweight deep learning inference framework, focusing on model deployment for mobile and embedded devices. It supports efficient model inference and training, and is widely used in a variety of Alibaba's business scenarios, such as Taobao, Nail and Youku, etc. MNN provides cross-platform compatibility, supports iOS, Android, and embedded devices, and optimizes the computational performance of CPU and GPU. Its core advantages are lightweight, high performance and ease of use, which can significantly reduce the resource consumption of model deployment.MNN-LLM is a language model running solution developed based on MNN, which supports multiple large-scale language models running locally and guarantees data privacy. The framework is open-sourced on GitHub, with an active community and rich documentation and examples, which is suitable for developers to get started quickly.

Function List

• Support for multiple deep learning model formats, including TensorFlow, Caffe, ONNX, and TorchScript.
• Provides efficient model inference with support for neural networks such as CNN, RNN, GAN and Transformer.
• Support for multi-device hybrid computing across CPU and GPU (Metal, OpenCL, Vulkan).
• Provides MNN-LLM to enable running large language models on mobile and PC.
• Support for model quantization (e.g., FP16 and Int8) to reduce model size 50%-70%.
• Provides MNN-Express to support control flow modeling and general purpose computing.
• Built-in MNN-CV, a lightweight OpenCV-like image processing library.
• Supports MNN-Train for training MNN models.
• Provides multimodal applications such as MNN-LLM-Android with support for text, image and audio processing.

Using Help

Installation process

To use MNN locally, you first need to clone the project code from your GitHub repository. Here are the detailed installation steps:

1. clone warehouse
   Open a terminal and run the following command to clone the MNN repository:

   git clone https://github.com/alibaba/MNN.git
   

2. Installation of dependencies
   MNN supports iOS 8.0+, Android 4.3+ and embedded devices with POSIX interface. Make sure the necessary compilation tools are installed on your system:
   • Linux/macOS: Install CMake (3.10 or higher), Git, and GCC/Clang.

     sudo apt-get install cmake git build-essential
     

   • Android (operating system): Install the Android NDK (r17 or above).
   • iOS: Ensure that Xcode and Metal support are installed.
3. Compilation MNN
   Go to the MNN directory, create a build folder and run CMake:

   cd MNN
   mkdir build && cd build
   cmake .. -DMNN_BUILD_MINI=ON
   make -j4
   

   utilization MNN_BUILD_MINI Reduces package size by approximately 25% for resource-constrained devices. Once compiled, the resulting executable and library files are located in the build Catalog.

4. Install MNN-LLM Android app
   For the MNN-LLM Android app, download the pre-compiled APK file from the GitHub Releases page or build it yourself:
   • Download the source code:project/android/apps/MnnLlmAppThe
   • Open the project using Android Studio, configure the NDK path and click Build > Build APK.
   • Install the APK on your Android device, which should support Android 4.3 or above.

Model inference using MNN

The core function of MNN is model inference. The following are the specific steps:

1. model transformation
   MNN supports TensorFlow, Caffe, ONNX and other model formats. Models need to be converted to MNN format:
   • Suppose there is a TensorFlow model model.pb, run the following command:

     ./MNNConvert -f TF --modelFile model.pb --MNNModel model.mnn --bizCode MNN
     

   • converted model.mnn documents can be used for reasoning.
2. Loading and running the model
   MNN provides C++ and Python APIs for inference. Here is a simple C++ example:

   #include <MNN/Interpreter.hpp>
   int main() {
   auto mnnNet = MNN::Interpreter::createFromFile("model.mnn");
   auto session = mnnNet->createSession();
   // 输入数据
   auto input = mnnNet->getSessionInput(session, nullptr);
   // 运行推理
   mnnNet->runSession(session);
   // 获取输出
   auto output = mnnNet->getSessionOutput(session, nullptr);
   return 0;
   }
   

   Developers can refer to demo/ sample code in the directory.

3. Using MNN-LLM
   MNN-LLM supports running large language models (e.g. Qwen, Llama) on mobile. After installing the MNN-LLM Android app:
   • Open the application and browse the list of supported models.
   • Click to download the model (e.g. Qwen-7B) and the model will be stored locally on the device.
   • Enter a text prompt and the app will return the generated result.
   • Support multimodal tasks, such as image description or audio to text, the operation is similar, just upload the corresponding file.
     The application runs completely offline and data is not uploaded to the server to safeguard privacy.
4. optimize performance
   • utilization FP16 maybe Int8 Quantitative modeling to reduce memory footprint.
   • Enable GPU acceleration (requires device support for Metal or Vulkan).
   • For complex models, the MNN-Express Optimizing control flow computation.

Image Processing with MNN-CV

MNN-CV provides lightweight image processing functions similar to OpenCV. sample code:

#include <MNN/ImageProcess.hpp>
int main() {
MNN::CV::ImageProcess::Config config;
config.destFormat = MNN::CV::RGB;
auto process = MNN::CV::ImageProcess::create(config);
// 加载图像
process->convert(imageData, width, height, 0, output);
return 0;
}

Developers can refer to include/MNN/ImageProcess.hpp Get more features.

application scenario

1. AI Applications for Mobile
   MNN supports image search, recommendation systems and live effects in applications such as Taobao and Nail. For example, Taobao uses MNN for product image recognition so that users can quickly find similar products by uploading images.
2. Embedded device deployment
   MNN supports IoT devices for smart home or industrial scenarios. For example, running a face detection model in a smart camera takes only 5 milliseconds to process a single frame.
3. localized language model
   MNN-LLM allows users to run large language models on their phones, suitable for privacy-sensitive scenarios such as local document summarization or voice assistants.
4. multimodal task
   The MNN-LLM Android app supports image description, audio to text, and other features for home maintenance, voice notes, and other scenarios.

QA

1. What model formats does MNN support?
   MNN supports TensorFlow, Caffe, ONNX and TorchScript formats, covering 178 TensorFlow operations, 158 ONNX operations, and more.
2. How to run MNN on low-end devices?
   utilization MNN_BUILD_MINI Compile option to reduce package size and enable Int8 Quantitatively reduce memory footprint.
3. Does MNN-LLM support external models?
   Currently the app needs to be recompiled to load external models, and there are official plans to improve model import in the future.
4. How does MNN perform?
   MNN is over 20% faster than TensorFlow Lite on models like MobileNet, and MNN-LLM is 8.6x faster than llama.cpp for inference on Android.