Deploying quantized Ultralytics YOLOv8 models on edge devices with DeGirum

Welcome to the recap of another insightful talk from our YOLO VISION 2023 (YV23) event, held at the Google for Startups Campus in Madrid. This talk was delivered by Shashi Chilappagar, Chief Architect and Co-Founder at DeGirum. It covered quantization and deploying quantized models, exploring key challenges, solutions, and future possibilities.

Introduction to quantization and deploying quantized models#

Shashi provided a comprehensive overview of quantization, highlighting its importance in optimizing Ultralytics YOLO models for deployment on edge devices. From discussing the basics to exploring approaches for improving quantization, attendees gained valuable insights into the intricacies of model porting and deployment.

Challenges in quantizing YOLO models#

Quantization often poses challenges, particularly with YOLO models in TFLite. Our audience learned about the significant drop in accuracy observed when all outputs are quantized with the same scale/zero point, shedding light on the complexities of maintaining model accuracy during the quantization process.

Improving quantization of YOLO models#

Fortunately, solutions exist to address these challenges. The introduction of the DeGirum fork offers a quantization-friendly approach by separating outputs and optimizing bounding box decoding. With these enhancements, quantized model accuracy sees a significant improvement from baseline levels.

More quantization-friendly model architectures#

Exploring new model architectures is key to minimizing quantization loss. Attendees discovered how replacing SiLU with bounded ReLU6 activation leads to minimal quantization loss, offering promising results for maintaining accuracy in quantized models.

Deploying quantized models#

Deploying quantized models is straightforward, with just five lines of code needed to run any model on the DeGirum cloud platform. A live code demo showcased the simplicity of detecting objects with a quantized Ultralytics YOLOv5 model, highlighting how easily quantized models can be integrated into real-world applications.

To this effect, Ultralytics provides a variety of model deployment options, enabling end-users to effectively deploy their applications on embedded and edge devices. Different export formats include OpenVINO, TorchScript, TensorRT, CoreML, TFLite, and TFLite Edge TPU, offering versatility and compatibility.

This integration with third-party applications for deployment allows users to assess the performance of our models in real-world scenarios.

Using different models on different hardware#

Attendees also gained insights into the versatility of deploying different models on various hardware platforms, showcasing how a single codebase can support multiple models across different accelerators. Examples of running different detection tasks on diverse hardware platforms demonstrated the flexibility and scalability of our approach.

Resources and documentation#

To empower attendees further, we introduced a comprehensive resources section, providing access to our cloud platform, examples, documentation, and more. Our goal is to ensure that everyone has the tools and support they need to succeed in deploying quantized models effectively.

Wrapping up#

As the field of quantization evolves, it's essential to stay informed and engaged. We're committed to providing ongoing support and resources to help you navigate this evolving field. Watch the full talk on YouTube.

Join us as we continue to explore the latest trends and innovations in machine learning and artificial intelligence.