Powering open-source computer vision with Hugging Face's transformers

As we keep exploring highlights from the YOLO VISION 2023 (YV23) event, let’s meet Merve Noyan, Developer Advocacy Engineer at Hugging Face, the leading NLP platform with pre-trained models for efficient development of language applications. In her talk, Merve shared insights into the world of open-source computer vision.

Here's a recap of her talk on transfer learning, transformers, and the open-source computer vision ecosystem.

Transfer learning unveiled: A quick recap#

Merve kicked things off with a quick primer on transfer learning, a technique that transfers knowledge from one neural network to another. Imagine training a model on the universal features in the early layers, like edges and corners, and then fine-tuning it for specific tasks. This is the essence of transfer learning, reducing data dependencies and boosting accuracy.

Merve highlighted classical convolutional backbones like ResNet and Inception, setting the stage for the transformer models that followed.

Enter the transformers: A riddle unveiled#

What makes Transformers special? Merve likened it to a riddle, showcasing how they differ from traditional convolution-based models. A key strength lies in their ability to perform self-supervised learning, capturing features without the need for labeled data. Vision Transformer, Data-efficient Image Transformer (DeiT), CLIP, and Swin Transformer were among the transformer-based models she introduced.

Ultralytics also supports a transformer model designed for object detection, RT-DETR. This model features an effective hybrid encoder, IoU-aware query selection, and adjustable inference speed. Notably, it adheres to the familiar pattern of other Ultralytics YOLOv8 models, presenting options for prediction, training, validation, and export.

Your one-stop-shop#

Merve then walked through Hugging Face's offerings, with over 8,000 models for classical computer vision tasks and 10,000 models for multimodal applications. The Hugging Face Hub hosts more than 3,000 datasets, making it a useful resource for developers and enthusiasts alike. Merve emphasized the seamless experience, thanks to Hugging Face's consistent API, offering ready-to-use models for various use cases.

Hands-on magic with Hugging Face#

The talk transitioned into practical demonstrations, showing how easily one can work with models. From instantiating models and processors to fine-tuning with the Trainer API, Merve made it clear that the Hugging Face Transformers library is designed with developers in mind. She even introduced the Pipeline API, a personal favorite, simplifying the workflow for users.

Fig 1. Merve Noyan presenting at YV23 at the Google for Startups Campus in Madrid.

A glimpse into applications#

Merve wrapped up the talk with a glimpse into some applications, including a model for visual question answering, BLIP for image captioning, and the Segment Anything Model for image segmentation. The Hugging Face Ecosystem's Pipeline API took the spotlight, making it a breeze to use models without diving deep into the technicalities.

The cherry on top was Merve's showcase of creating optical illusions with a diffusion model, a captivating experience that adds a fun twist to the world of AI.

In a nutshell!#

In conclusion, Merve's talk left us inspired to explore what open-source computer vision can do. Hugging Face has made AI more accessible and fun, helping developers put their ideas into practice. Here's to the future of the open-source community and the innovations it will bring!

Watch the full Hugging Face computer vision talk!