Since most modern AI systems are based on learning patterns and meaning from data, it is critical to understand and manage that data – as with knowing the problem domain, it is important that the AI team understands their data landscape intimately. 

This includes understanding the quantity and quality of the data available to them, the labelling and annotation of that data and any tradeoffs as regards cost of obtaining and managing the data.

One aspect of building AI systems is the amount of data used to train and validate the systems. There are a number of extremely large open source data sets available in many of the standard AI domains, e.g. ImageNet,  Open Images Dataset, and if the problem being addressed maps to those domains, then these can be used. However, often the problem domain is sufficiently different that data sets need to be identified or built from scratch. 

It is not just a matter of indiscriminately acquiring as much data as can be found – the cost and effort of acquiring data, and the need to understand its quality and preprocess/label/annotate the data (for supervised learning approaches) may mean that this approach is not viable. 

This is where an agile, iterative approach can de-risk an AI project. Data sets can be built iteratively and incrementally as the solution develops from the proof of concept stage through to full maturity. Using approaches such as transfer learning from similar domains, a PoC can be built with a surprisingly small amount of data, which in consultation with stakeholders can be used to scope the data required to build and validate the final system. 

We have found this approach is preferable to, and less expensive than, gathering all data up-front and then using this data to build the system, as it allows early identification of data issues, and subsequent data cleaning, annotation protocol improvement, and enrichment during the project, rather than at the end.

As mentioned above, annotation/labelling of data is a key part of building AI systems, particularly when applying supervised learning, although even when using unsupervised approaches, validation will typically be done against an annotated or labelled ground truth. 

The majority of AI systems obtain labels and annotations from human experts, and as such may be prone to human errors and/or biases, and if multiple sources are used, may be inconsistently applied. These introduce risk in the form of error or bias in the final validated system, and so care must be taken in obtaining the ground truth data and controlling the quality thereof. 

We have found benefit in working with stakeholders and domain experts to develop annotation protocols, which can then be used to train annotators and attempt to ensure consistency. 

One subtle point is that the annotation protocol should be specifically directed towards building AI models and the solution scope. For example, when working with pathologists, we have found that their understandable tendency is to annotate classes which have clinical importance and identify morphologies on this basis. Whilst these can be used to develop computational pathology algorithms, we have found benefit in machine learning experts and pathologists working together to define the classes appropriate to the problem at hand.

Putting processes in place for data and annotation quality (and the people and/or systems to manage these processes) is obviously more ‘costly’ from a superficial perspective, but having the appropriate ones de-risks the delivery from a technical and business point of view. However, it will be a business decision as to whether this benefit is worth the added costs and complexity