How to Choose an Image Annotation Tool?

Image annotation is the process of describing an image. It can be categorizing the whole image, localizing certain objects and drawing boxes around them, segmenting every pixel, and many other use cases. The image annotation process is utilized to label images and prepare them for tuning or training a conventional mathematical or intelligent model to detect, recognize and analyze specific image objects. When such a model is fully prepared and deployed in production, it is expected to recognize similar concepts in new images that have not been annotated previously. Image annotation tools are widely used for image classification and segmentation tasks, detecting and recognizing objects, and identifying boundaries and shapes within an image. Today, they present needed additions to various Artificial Intelligence (AI) and Machine Learning (ML) models whose purpose is to cover computer vision tasks. This popularity caused an almost uncountable number of annotation solutions on the market, with a continuously increasing number of new tools and publicly available annotation data and image datasets for machine learning practitioners and experts in the field.

In order to select high-quality image annotation solutions for a single task, it is required to find the right tool for a specific use case and in accordance with available image data. Whether to choose the tool for the project purposes or just to outsource image annotation services, the selection phase can dramatically affect the quality of generated results and the time costs for completing the overall annotation process. In general, when choosing the best tool, the workflow should be analyzed from the perspectives of the annotation problem, the technical features of tools, available budget, user’s requirements, and required actions during the annotation process (Fig. 1). To achieve time-efficient manual or automated annotations, all user actions should be insured by an easy-to-use interface and a convenient support system of the tool.

How to Choose an Image Annotation Tool?

Figure 1. The issues and factors influencing the choice of the right image annotation tool

From the perspective of technical features, it is desirable to analyze the main functionalities needed to solve a specific annotation use case. The overall functionality of the tool should be evaluated through the technical factors a user requires, available productivity statistics and analytics, task management features, and formatting capacity that should guarantee output annotations in a target format. Functionalities that are necessary for one problem may not be required for solving another one. However, some of them can be accepted as universal features essential for most of the tools and will be covered next.

For the beginning, it is desirable to select an easy-to-use solution with intuitive shortcuts and easy on-premise installation. Depending on the problem and technical environment, the choice between a stand-alone or web image annotation tool may be an important criterion for selection. If the task is to use a secure and fast application, the standalone tool should be the choice. It executes with the full capabilities of a user machine, provides more functionalities than a web application, and the overall usability is higher. On the other hand, the web-based tool is the right choice if the main goal is to use easily deployed and maintained applications available from any remote computer with a random operating system.

The core factors are the annotation methods themselves, which are nowadays based primarily on using AI techniques and auto-labeling features that help human operators improve their annotations or even perform annotation tasks entirely instead of them. Some advanced annotation tools support the option to integrate a user’s ML algorithm and plug it as an additional feature into an existing image annotation tool. If the existence of AI features within the tool is vital for a user, it should be beneficial to check if the tool also supports optional features such as TensorFlow Object Detection, Deep Learning Deployment Toolkit, etc.

Another critical factor of an annotation workflow is to ensure that the tool supports storing and processing required volumes of data and all needed image types and formats. This dataset management feature should always include filtering options, searching, merging, and cloning mechanisms necessary for image file treatment. When talking about data treatments, it is also important to consider data quality control features crucial for providing reliable and verified results. Some of the core processes used for controlling data quality are data profiling and reporting, data repair workflows, and exploitation of appropriate data metrics in order to measure the data quality.  One more important aspect of data processing is securing sensitive protected information, limiting an annotator’s rights to see the data, and preventing the download of images if required. If sensitive data is in focus, privacy issues of using a 3rd-party application could arise. In that case, the recommendation is to use a local application not based on cloud services but local drives.

Finally, price always matters, and small companies or individual users will also focus on this criteria when selecting the annotation tool. Like other software products, three possible pricing solutions exist for available image annotation tools: commercial, open-source, and freeware. Open source and freeware tools are free and ideal solutions for one-time or smaller annotation projects based on pre-prepared image classification data. However, they are built for individual users and may require additional development resources to adjust their features for specific needs. They may have a poor workforce management environment and common problems with scaling a project into production. On the other hand, commercial solutions could accelerate project timeline, provide more efficient work, helpful annotation shortcuts and enable best practices into tools. In general, various pricing models can be distinguished. For example, a Flat rate pricing model provides a single product with a set of features for a fixed one-time price.

The Usage-based pricing model offers the option for paying the service depending on the usage level:

The more the application is used, the more it costs. Further, the per-user pricing model is based on the price of offered features that changes according to the number of users. One more popular pricing model is a Freemium model that is free of charge for a specified trial period, after which it must be paid to be used in the feature. Finally, it is also essential to pay attention to those who use the tool and handle exceptions. An expert in the annotation field may require a completely different and more complex technical environment than an ordinary user who needs comprehensive training, a user-friendly environment, and a quality support system.