Powering open-source computer vision with Hugging Face's transformers

As we keep exploring highlights from the YOLO VISION 2023 (YV23) event, let’s meet Merve Noyan, Developer Advocacy Engineer at Hugging Face, the leading NLP platform with pre-trained models for efficient development of language applications. In her talk, Merve shared some incredible insights into the world of open-source computer vision.

Join us as we take you on a journey through the fascinating universe of transfer learning, transformers, and the open-source computer vision ecosystem.

Link to this sectionTransfer learning unveiled: A quick recap#

Merve kicked things off with a quick primer on transfer learning, the magic wand that allows us to transfer knowledge from one neural network to another. Imagine training a model on the universal features in the early layers, like edges and corners, and then fine-tuning it for specific tasks. This is the essence of transfer learning, reducing data dependencies and boosting accuracy.

Merve highlighted classical convolutional backbones like ResNet and Inception, setting the stage for the transformational journey ahead.

Link to this sectionEnter the transformers: A riddle unveiled#

What makes Transformers special? Merve likened it to a riddle, showcasing how they differ from traditional convolution-based models. The secret sauce lies in their ability to perform self-supervised learning, capturing features without the need for labeled data. Vision Transformer, Data Efficient Transformer, CLIP, and Swin Transformer were among the star-studded cast of transformer-based models she introduced.

Laying some common ground with Ultralytics who provides support for a transformer model designed for object detection. This model features an effective hybrid encoder, IOU-aware query selection, and adjustable inference speed. Notably, it adheres to the familiar pattern of other Ultralytics YOLOv8 models, presenting options for prediction, training, validation, and export.

Link to this sectionYour one-stop-shop#

Merve then delved into the treasure trove of Hugging Face's offerings, with over 8,000 models for classical computer vision tasks and 10,000 models for multimodal applications. The Hugging Face Hub boasts a whopping 3,000+ datasets, making it a playground for developers and enthusiasts alike. Merve emphasized the seamless experience, thanks to Hugging Face's consistent API, offering ready-to-use models for various use cases.

Link to this sectionHands-on magic with Hugging Face#

The talk transitioned into practical demonstrations, showcasing how effortlessly one can work with models. From instantiating models and processors to fine-tuning with the Trainer API, Merve made it clear that the Hugging Face Transformers library is a developer's best friend. She even introduced the Pipeline API, a personal favorite, simplifying the workflow for users.

Fig 1. Merve Noyan presenting at YV23 at the Google for Startups Campus in Madrid.

Link to this sectionA glimpse into applications#

Merve wrapped up the talk with a glimpse into some fantastic applications, including the Plot model for visual question answering, Blip for image captioning, and the powerful Segment Anything model for image segmentation. The Hugging Face Ecosystem's Pipeline API took the spotlight, making it a breeze to use models without diving deep into the technicalities.

The cherry on top was Merve's showcase of creating optical illusions with Elysian Diffusion, a captivating experience that adds a fun twist to the world of AI.

Link to this sectionIn a nutshell!#

In conclusion, Merve's talk left us inspired and itching to explore the endless possibilities of open-source computer vision. Hugging Face has truly made AI accessible, fun, and exciting, empowering developers to unleash their creativity. Here's to the future of the open-source community and the incredible innovations it holds!

Watch the full Hugging Face computer vision talk!