One of the most typical tasks in machine learning is classification tasks. It may seem that evaluating the effectiveness of such a model is easy. Let’s assume that we have a model which, based on historical data, calculates if a client will pay back credit obligations. We evaluate 100 bank customers and our model correctly guesses in 93 instances. That may appear to be a good result – but is it really? Should we consider a model with 93% accuracy as adequate?
It depends. Today, we will show you a better way to evaluate prediction models.
Let’s come back to our example. The model was only wrong in 7 instances, however, its true quality depends on which instances. Let’s say that out of 100 clients, only 10 would not pay back. In this case, we can assume two scenarios:
Obviously, the second scenario is more costly to the bank, which suffered significant loss because it gave loans to people who didn’t manage to repay. To better explain this whole process, let’s introduce these terms:
Indeed, there are two categories of error: predicting a positive when the instance is negative and predicting a negative when the instance is positive. There are also two categories of good prediction; successful prediction is called true and unsuccessful false. As you can see, we now have 4 variants which form a tighter confusion matrix.
Let’s get back to our example. We have 100 loan clients.
Out of the 100, 10 will not pay back their loan (positives). However:
The other 90 people paid back their loans (negatives). However:
This information in a table will form the confusion matrix.
When we sum it up (8+2+85+5), we get all of our 100 clients.
This method of evaluation is more meaningful, so let’s look next at a model taking into account spcific goals.