Improving the AI Consensus Scoring results

Sometimes even AI solutions can underperform, and it is a pain Data Science and R&D teams try to evade or overcome daily. Nobody is safe in terms of a neural network occasionally showing a bad result, and AI teams should not be afraid of that fact and know what to do if such a situation occurs.

In Hasty, we highly rely on our AI-powered features such as AI Assistants and AI Consensus Scoring to deliver real value to you. Still, we are also in the same zone as every other AI team - our solutions might not out-of-the-box perform as intended. It can happen for various reasons, but using our vast experience in the field, we identified the possible causes. On this page, we want to share them with you, talk about how they can affect the AI Consensus Scoring performance, and let you know the best steps to overcome the problem if it occurs.

Surprise! It is all about the data…

Let’s cut the crap and keep the conversation straight here. The performance of any AI solution is as good as the data it was trained on. When talking about AI Consensus Scoring, the data it uses as a training set is your data. We are sorry to address the elephant in the room, but you have probably come to this page after getting a notification that the AI CS run did not complete because the underlying Machine Learning metrics were not good enough. Right now, you are likely thinking of how you can improve the metrics to get a run done. Let’s check out the possible steps.

Well, our AI features are strongly tied with one another, so it is reasonable to check the performance of the AI Assistants first. If they work fine, AI Consensus Scoring should work as well. If you experience some difficulties with AI CS and the AI Assistants work fine - please reach out to us.

However, if AI Assistants produce weird results, AI CS will likely underperform. There are two potential reasons for the second case:

• You might simply not have enough data in your run for AI algorithms to adequately train;
• Your data might be too noisy.

The fixes are relatively simple in both cases - add more data or use the Manual Review feature to clean up bad data. Let’s break both of these cases down.

A small amount of data → add more data

As you might know, in many tasks, AI models need a high amount of high-quality data as a training set to have a chance for a good performance. In vision AI, such a demand is more than relevant. You must have heard of the massive CV benchmark datasets such as ImageNet and COCO consisting of hundreds of thousands of images. Now imagine a complex AI solution such as AI Consensus Scoring that focuses on finding the potential mistakes in your data. Both the task and the algorithms underneath AI CS are very difficult even compared to other AI tasks and neural network architectures.

So, our primary goal as users should be to give AI CS all the possible resources for the algorithms to perform. Suppose you give it a small amount of high-quality data. In this case, the feature will likely underperform as it will not have enough pictures and/or annotations to train on (for example, if you have 100 images, 20 different classes, and only 10 annotations per class). Therefore, if you see a notification telling you that the AI CS run did not complete - please check out the quantity of data you passed to the run. We can not tell you the exact number of images or annotations AI CS needs to perform well as it strongly depends on a vision AI task and the case you are trying to solve. Still, we can ask you to add as much data as possible because it is the most likely blocker for the feature.

Noisy data → clean it up

In some cases, you might pass many images to the run but still receive a notification. In such a situation, we recommend you think if your data is clean enough for any AI algorithm to be trained on. In Data Science, another vital aspect besides the data quantity is its quality. If the quality of the annotations is poor, AI models will not be able to perform well. Sure, they will output something, but their generalization and prediction power will be questionable.

Therefore, if you pass noisy data to AI CS and hope that the solution will somehow sort everything out, you are mistaken. AI CS, just as any other AI solution, needs data of a good quality to train adequately. So, if you can describe your data as a noisy one, please use the Hasty Manual Review feature to clean it up before scheduling AI CS to run.

We are always trying to improve our features, so please do not wait and get in touch if you are sure that both data quality and quantity are on the board. We are happy to help you with troubleshooting.