We've been talking about AI models learning from data, but what is this data exactly?
Just imagine the following scenario in your head before you read further. You are a developer who has been assigned a task to create an AI model that is able to tell if the user is holding a green or a red bottle.
There are 2 types of data that are used in the process of building an AI model. These are namely:
The set of data that is given to the model in the beginning for the model to train on, and figure out patterns and relations, is known as training data. This process is basically referred to as ‘training the model’. This is the time when the machine is trying to become ‘intelligent’.
This data is usually the largest in terms of size because, the larger the data, the more the model has information to learn from.
Let’s take the scenario we talked about into consideration. In the given scenario, you’ll first have to train the AI model with some labelled images of green and red bottles so that the model is able to distinguish between the two. Remember that an AI model is a clueless baby till the time it is not trained with data. These images that you give to the model are basically the training data.
#Note: The term “labelled” above means that you somehow tell the model which image belongs to which class (red bottle or green bottle).
The set of data that is used to evaluate how good a model has learned the patterns from the training data is known as the testing data. The model is tested on this data after being trained on the training data.
This is the data that the model has never seen before and that is the way you want it to be. If the model would have seen this data before then you would not get a proper unbiased estimate of your model’s true accuracy.
The testing data is usually smaller in size than the training data. This is because we have limited amounts of data and the training process requires a lot more data in comparison to the testing process.
When implemented on the testing data, if the model is able to predict the correct output majority of the time, then it is said that the model has successfully been able to learn the patterns out of the training data.
Let’s bring the “red bottle green bottle” scenario back into the conversation. In the previous section, we saw that you trained your AI with labelled images of red and green bottles. Now what? Do you (the developer) know if the model has really learned the patterns from the training data that are needed to distinguish between a red and green bottle? How do you find that out? You might have already guessed it. You use the testing data. The testing data, in this case, might be some random unlabeled images of red and green bottles. You will feed these images to the AI model and if it is able to correctly distinguish the red and green bottles between them, then voilà, your model has been able to learn the correct patterns from the training data.
Till now, we've learnt that AI systems/machines are fed data and these machines learn and understand that data, which in turn helps them to make informed decisions.
Let us dive deep into understanding the process that tells us how the machine learns from this data.