Panoptic Segmentation

As you might know, Computer Vision (CV) is a scientific field that researches software systems trained to extract information from visual data, analyze it, and draw conclusions based on the analysis. The area consists of so-called CV or vision AI tasks. Each task is unique and incorporates techniques and heuristics for acquiring, processing, analyzing, understanding the data, and extracting various details from it. One of these tasks in Panoptic Segmentation. On this page, we will:

• Understand the basics of the Image Segmentation field in Machine Learning;
• Cover in-depth the Panoptic Segmentation vision AI task;
• See how Panoptic Segmentation compares with Instance and Semantic Segmentation;
• Research the real-life applications of Panoptic Segmentation;
• Cover some popular Panoptic Segmentation datasets and SOTA results on them;
• See features that CloudFactory offers for streamlining a Panoptic Segmentation task.

Let’s jump in.

What is Image Segmentation in Machine Learning?

It is essential to start with a bigger picture. The logical question is, what is Image Segmentation in Machine Learning?

Well, Segmentation is a well-known term in business and marketing. In short, it defines the process of splitting customers (or a whole market) into separate groups based on specific patterns in their behavior. Fortunately, such a definition is close to what we refer to when saying Image Segmentation in ML.

Image Segmentation in Machine Learning is a part of the vision AI field that incorporates different methods of dividing visual data (for example, an image) into segments featuring specific, similar, and significant information of the same class label.

As of today, corporate Data Science regularly solves Image Segmentation challenges in various spheres. In CloudFactory, we see that the demand for high-quality Segmentation solutions has rapidly grown over the past couple of years. It also applies to Data Scientists who specialize in the Image Segmentation field. As a result, the industry is developing and growing, bringing new SOTAs, solution techniques, and challenges.

Nowadays, researchers say that the Image Segmentation field consists of three vision AI tasks. These are:

• Semantic Segmentation;
• Instance Segmentation;
• Panoptic Segmentation.

Let’s take a closer look at the Panoptic Segmentation vision AI task.

What is Panoptic Segmentation in Machine Learning?

Moving from the bigger picture to the details, let’s see what Panoptic Segmentation is.

Panoptic Segmentation is a hybrid Computer Vision task that combines the ideas behind the Semantic (SS) and Instance Segmentation (IS) tasks bringing a more holistic understanding of a scene.

As SS, Panoptic Segmentation focuses on classifying every pixel in an image to create a pixel-perfect segmentation map. From IS, Panoptic Segmentation borrowed the concept of segmenting each object of the same target class as distinct objects.

Here is a simple example of how a Panoptic Segmentation output might look like:

As a result, the output in the Panoptic Segmentation case consists of the following:

• Pixel-perfect segmentation map of the whole image;

• A bounding box and a segmentation mask for each object of instance target classes.

The data annotation process for a Panoptic Segmentation task is as follows:

1. You predefine some semantic (for example, ‘sky’, ‘road’, and ‘grass’) and instance target classes (for example, ‘person’ and ‘umbrella’). Please note that as in any Semantic Segmentation task, in Panoptic Segmentation, there is a default ‘background’ class that is used to mark all the space that does not correspond to any other category;

2. You assign each pixel on a picture to one of the semantic classes while treating objects of instance classes as distinct instances of the same category.

To summarize, Panoptic Segmentation receives an image and some semantic and instance target classes as input. As an output, you get a pixel-perfect segmentation map of the whole picture, with each pixel assigned to a specific category. Additionally, you get a bounding box and a segmentation mask for each object of instance target classes.

Panoptic Segmentation Vs. Instance Segmentation

Although Panoptic and Instance Segmentation live in the same Image Segmentation field and share many similarities, they are still different vision AI tasks. You must remember it when identifying which task you aim to solve.

Instance Segmentation ensures that all the objects of the same class are viewed as distinct instances. So, from an IS model, you get a bounding box and a segmentation mask for each object.

On the other hand, Panoptic Segmentation is a combination of Semantic and Instance Segmentation. In other words, with Panoptic Segmentation, you can obtain information such as the number of objects for every target class (countable objects), bounding boxes, segmentation masks, and a segmentation map of the whole image. These come from Instance Segmentation. However, you also get a segmentation map from Semantic Segmentation and know a target class for each pixel. So, Panoptic Segmentation provides a way more holistic understanding of a scene.

Panoptic Segmentation Vs. Semantic Segmentation

Although Panoptic and Semantic Segmentation live in the same Image Segmentation field and share many similarities, they are still different vision AI tasks. You must remember it when identifying which task you aim to solve.

Semantic Segmentation classifies every pixel on an image, so all instances of the same category share the same class label. As a result, you get a segmentation map.

On the other hand, Panoptic Segmentation is a combination of Semantic and Instance Segmentation. In other words, with Panoptic Segmentation, you can obtain information such as the number of objects for every target class (countable objects), bounding boxes, segmentation masks, and a segmentation map of the whole image. These come from Instance Segmentation. However, you also get a segmentation map from Semantic Segmentation and know a target class for each pixel. So, Panoptic Segmentation provides a way more holistic understanding of a scene.

Panoptic Segmentation Vs. Instance Segmentation Vs. Semantic Segmentation

Panoptic Segmentation real-life applications

Nowadays, Image Segmentation is widely used across various industries. From our experience, it seems like Instance Segmentation is more popular than Panoptic and Semantic Segmentation. Still, it does not mean that Instance Segmentation holds the market, as there is always room for other tasks and approaches.

Panoptic Segmentation is a combination of Instance and Semantic approaches, so logically, it might be a good fit for both SS and IS real-life applications. To support this, please consider the reasoning behind picking an Instance or Semantic Segmentation case. From our experience, it lies in the task definition, which might vary depending on the end goals. This is why the same Image Segmentation cases might be viewed simultaneously as Instance, Semantic, or Panoptic Segmentation problems.

So, noteworthy Panoptic Segmentation applications include:

• Autonomous driving systems (for example, segmentation of visual input data from the camera - classification of a pedestrian on the road, a vehicle, signs, etc.);

• Medical data processing (for example, segmentation of MRI images searching for tissues, tumors, anomalies, and their characteristics such as area, dynamics, etc.);

• Aerial, satellite, and UAV image processing (for example, segmentation of a landscape);

• Video surveillance systems;

• Digital image processing (for example, advanced camera options in smartphones);

• And many other use cases.

Panoptic Segmentation datasets

When it comes to the Panoptic Segmentation datasets, it is pretty easy to orient. As you might know, Image Segmentation datasets usually have annotations for various vision AI tasks, such as Instance Segmentation, Semantic Segmentation, and even Object Detection.

So if a dataset has annotations for Semantic and Instance cases for the same pictures, you can consider it a Panoptic Segmentation dataset and use it for your needs. The trustworthy benchmark datasets regularly used to evaluate recent Panoptic Segmentation model architectures and approaches are as follows:

• COCO;

• Cityscapes;

• ADE20k;

• KITTI Panoptic Segmentation;

• Mapillary.

How do we solve a Panoptic Segmentation task in CloudFactory?

Throughout years in the industry, CloudFactory's IT team has developed many internal instruments that our cloudworkers and Data Scientists use when working on client cases.

Let’s go through the available options step-by-step. To streamline the Panoptic Segmentation annotation experience, CloudFactory's internal data labeling tool supports:

• Manual annotation tools such as Polygon and Brush. Also, there is an option to convert polygon to mask and vice versa in a single click;
• Semi-automated annotation tools such as ATOM powered by SAM and Box to instance;
• AI-powered Semantic Segmentation and Instance Segmentation assistants.

As of today, these are the key technical options CloudFactory has for Panoptic Segmentation cases. If you want a more detailed overview, please check out the further resources or book some time with us to get deeper into CloudFactory with our help.

Further Resources

• Guide on how to choose between Object Detection, Instance, and Semantic Segmentation for your use case.