Artificial Intelligence: Are We Effectively Assessing Its Business Value? – Predictive Analytics Times – machine learning & data science news

As most data science practitioners know, artificial intelligence (AI) is not new and has been explored by academia back as far back as the fifties. The real core of AI is the branch of mathematics related to neural nets which have been explored both by academia as well as data science practitioners. A number of practitioners including myself familiarized ourselves with these techniques which became one more item within the data scientist toolkit. For those of us involved in using predictive analytics to predict consumer behaviour related to marketing and risk, logistic regression and decision trees in many cases performed at about the same level as neural nets. In some cases such as fraud where there were typically a much larger volume of records, neural nets did exceed the more traditional type of modelling techniques.

But the appetite for AI deployment was always negated by its lack of its explainability to the business stakeholder and as mentioned above the minimal examples of its superior performance relative to the more traditional techniques.

So what changed and what has led to all the excitement about AI. In order to better understand this evolution, one needs to focus on the research. Research in this area for decades always focussed on how these tools could better classify images. Back in the nineties, I remember reading numerous articles from publications where the ability to classify images was approximately 40%-50%. In the last 5 years, though, this accuracy has now achieved levels of 95%+. This game breaking change was caused by two factors with the first factor being related to technology and how data could be processed.

Data and Big Data could now be processed and consumed using parallel processing as opposed to sequential processing. Meanwhile, this newfound technical capability allowed practitioners to consume exponentially much larger volumes of data for analytics (both advanced and non-advanced) purposes. The consumption of these extremely large volumes then allowed users to explore the notion of more complex type neural nets or deep learning, which is the ability to utilize many hidden layers and many nodes as opposed to a single hidden layer with few nodes that was the common occurrence within a restricted sequential data processing environment. This ability to more fully leverage the power of artificial intelligence was the second factor which now improved the image classification accuracy to 95%+.

With this breakthrough, AI had to be more seriously explored as another option in improving results. But does that mean that we should blindly adopt AI in all our business processes. Certainly, we are seeing the emergence of applications to better detect fraud through improved image recognition while enhanced customer service is the outcome of improved AI-developed chat boxes. Many more applications are being explored and which are expected to provide further disruption to an already changing economy. But let’s discuss the notion of AI within the world of predicting consumer behaviour both from a marketing as well as a risk behaviour.

The use of data science and machine learning to predict consumer behaviour has been an ongoing business discipline for many decades. Success for seasoned data science practitioners in this area was never evaluated by the error rate between predicted behaviour and observed behaviour but rather by how much incremental lift is achieved within a given model. Typically, this has been observed by AUC(Area Under Curve) results which measures the cumulative observed behaviour(Y-axis) within a cumulative % of the records where these records are ranked by predicted model score.(X-axis) Let’s take a look at an example of an insurance risk model below.

In the table above, we have a predictive model which is trying to predict the overall loss amount of a given policy. If the measure of success is lift, then the given % of observed losses should be higher than the observed % of scored records. For example, the straight grey line indicates random performance or no lift as 10% of policies yields 10% of loss amounts, 20% of policies yields 20% of loss amounts, etc. Meanwhile, the red line, representing the current pricing methodology, does yield lift as 10% of the policies yields 22% of the loss amounts for a 220((22/10)*100) lift ratio while 20% of policies yields 32.4% of the loss amounts for a 162((32.4/20)*100) lift ratio.

Now when we build a predictive model which is the green line, what kind of lift do we see? At 10% of the policies, we observe 30.9% of all loss amounts for a 309 ((30.9/10)*100) lift ratio while 20% of policies yields 46% of all the loss amounts for a 230((46/20) lift ratio. With the model, you can see that the lift numbers are vastly superior to the existing current pricing methodology. Visually, as the parabola becomes more pronounced, model performance is stronger which in effect can be measured as the total area between the parabola and the straight line (AUC). This area under the curve (AUC) is often referred to as the KS (Kolmogorov-Smirnov) statistic. From the three charts, it is easy to observe that the AUC is strongest for the predictive model. Yet at the same time, we also observe that the impact of the model is strongest for the top deciles. The overall lift deteriorates as we select larger groups of records until there is virtually no lift at 100% of the records which is the point of convergence between all the targeting options.

But in many cases, results need to be looked at larger groups such as the top 50% and how much lift is achieved within the top half of all scored records. From an absolute standpoint, overall business gains are more pronounced due to larger volume of records being selected by the model along with a lift ratio which is strong but sub-optimal where in most cases these business gains are maximized at approximately 50%.

So how would AI disrupt our perspective here? Simply put, can AI deliver a better AUC or more pronounced parabola. Yet, even with a better AUC, we might find that our business application is to the top 50% where indeed a more traditional type model is achieving more or less the same lift. In this case, it might make more sense to adopt the more traditional modelling techniques than attempt to adopt AI techniques which as stated earlier are not easily explainable to the business users. However the important point is that we have defined a metric and the conditions in which to measure success. In this scenario of consumer behaviour, the metric is lift while the condition is the volume of records being selected within the application.

Looking at simply AUC in evaluating AI as a modelling option may be misleading if the business application is towards a sub-segment within that scored base. As with all business initiatives, the key is how one defines success. Once this is established, we now have a reference point in how we will measure and evaluate results. For experienced data scientists, this is the exciting part of our work as this is in effect our report card where we can not only determine the business value of AI but how it compares to other approaches.

About the Author

Richard Boire, B.Sc. (McGill), MBA (Concordia), is the founding partner at the Boire Filler Group, a nationally recognized expert in the database and data analytical industry and is among the top experts in this field in Canada, with unique expertise and background experience. Boire Filler Group was recently acquired by Environics Analytics where I am currently senior vice-president.

Mr. Boire’s mathematical and technical expertise is complimented by experience working at and with clients who work in the B2C and B2B environments. He previously worked at and with Clients such as: Reader’s Digest, American Express, Loyalty Group, and Petro-Canada among many to establish his top notch credentials.

After 12 years of progressive data mining and analytical experience, Mr. Boire established his own consulting company – Boire Direct Marketing in 1994. He writes numerous articles for industry publications, is a well-sought after speaker on data mining, and works closely with the Canadian Marketing Association on a number of areas including Education and the Database and Technology councils.