Matching Pattern-Discovery Models to Business Use Cases

Our sole purpose as data scientists is to create value from data. More specific to machine learning (ML), we use algorithms to learn from data so that we can recognize patterns and use them to build generalizable models that ultimately benefit the top or bottom lines.

That benefit-or value-is defined by the need that is driving our work. If working for a biotech company, that need might be to discover a new treatment. In marketing, value might come from a model that attributes revenue-generation to specific marketing programs. If the workplace is somewhere where a lot of money changes hands, a learning model that recognizes evolving fraud patterns is a likely priority.

Whatever the goal, most of the data science going on in the business world today is using ML for predictive and prescriptive analytics, or what I like to call “seeing around the corner.” When businesses are better at seeing what’s coming, they are more likely to discover emerging opportunities and risks, and so can better prepare for next-best decision and next-best action.

How Machine Learning Improves Predictability

ML gives data scientists the ability to build more accurate predictive models through pattern recognition. Traditional time-series forecasting uses auto-regressive analysis, which finds trends in historical data and fits them to some kind of function and trend line (or a combination function and line). From that knowledge, the model is used to forecast what will happen in the future.

But auto-regressive models have limitations that can prevent data scientists from reaping all of the value that their data offers. They have wide margins of error and can’t predict previously unknown events, which often is the most valuable type of business insight because it signals immense opportunity and/or risk.

More effective ML-based predictive models can make a huge difference in outcomes. Think about the unpredictable weather events the U.S. has seen lately. Even the most high-fidelity, physics-based forecasting models that meteorologists use have a lot of uncertainty; for example, hurricane-landfall predictions that span multiple states and thousands of coastal miles. Using models with a higher degree of predictability could literally save lives, homes and communities through better and more targeted preparation messaging. That knowledge could also help insurers and the construction industry more effectively price their products and services, as well as bring specialized products to areas that are expected to be most affected by changing weather patterns.

ML provides better predictive models through pattern discovery, recognition, exploration and exploitation of multiple diverse data sets. The last two categories correlate with advanced analytics; that is predictive and prescriptive analytics. It’s here that businesses derive hard-dollar value from data because data exploration can be a gateway to product and process innovation; and pattern exploitation is data “commercialized” for value-creating actions, decisions, and opportunities.

Pattern discovery could be considered “easy” compared with pattern exploitation, because exploitation requires more data science. Generalization is the key to building these models. The most generally useful model captures the fundamental pattern in the data and takes into account the natural variance in the data. Here is an example, using x as a growth variable and discovering what the outcome would be using higher and lower values for x.

Figure 1


  • The model on the left uses two parameters and is a straight line. It doesn’t capture all the structure that’s in the data, so it’s not exploiting all of the information in the trend.

  • The other extreme is the model on the right, which has five parameters. It attempts to fit every data point without taking into account that there is some natural variance in the data. We can’t exploit a pattern from a curve that is alternately increasing and decreasing in such unexplainable ways.

  • The model in the middle, the “just right” solution, captures the trends and patterns in a way that conveys a pretty good idea of what is going to happen with higher values and lower values of x. It captures the fundamental pattern without over- or under-specifying the data, so the data scientist can take advantage of the real pattern of the data to create and to exploit a predictive model.

A tip to remember is that precision is not the same as accuracy. A precise model (like the one on the right of Figure 1) can be precisely wrong. A generally accurate model (like the one in the middle of Figure 1) can be useful in a more general set of conditions and situations. Our goal as data scientists is not to zero in on granular data points but to use data to build useful models that predict outcomes that are sufficiently accurate, not precisely wrong.

4 Flavors of Pattern Detection and Discovery

Data scientists can draw on these four flavors of pattern detection and discovery as they build predictive and prescriptive models:

Figure 2


    Class discovery (clustering): This method can be used to discover categories, events and behaviors within data, as well as the rules that constrain the data classes, i.e., what uniquely distinguishes them (different shapes and sizes, proximity, etc.). These data classes could be customers, treatment options, financial assets, or any number of other pertinent groups. (See data classes dc1, dc2, and dc3 in Figure 2.)

Data scientists can also discover how the classes behave when parameters are changed. For example, P2 might be product pricing, and the clusters reveal how purchase volume (P3) changes as a function of time of year (P1) and pricing (P2) for different products (the different clusters).

    Correlation discovery (predictive and prescriptive power): This method can be used to find trends, patterns and dependencies in data that reveal new governing principles or behavioral patterns. The discovery can be predictive. For example, in Figure 2, depending upon P1, we might be able to predict P3 depending upon which class an entity is residing in.

Prescriptive analytics also are possible, For example in data class 2, we can tell the correlation value for P1 and P3 is weaker for smaller values of P2; but the correlation value for P1 and P3 is stronger for higher values of P2. Sticking with the example of P2 being product pricing, we might be able to optimize (prescribe) customer reaction (purchase volume P3) based on choosing a particular product price (P2) at different times of year (P1).

  • Novelty discovery (surprise!): In this type of discovery, data scientists can find the new, surprising, unexpected one-in-a [million/billion/trillion] object, event or behavior. The business need could be discovering new revenue opportunities (e.g., a new category of customer) or reducing risk (e.g., discovering an anomaly in the performance of a jet engine).

  • Association discovery (finding links): This is discovery of unusual (interesting) data associations/links/connections across entities in your domain, known as graph or network analysis. With this method data scientists can discover not just the direct links among data but links that are transitive, i.e., a is connected to b, and b is connected to c; therefore a is transitively connected to c through b. This capability is important for fraud detection, root cause analysis and marketing attribution because sometimes an intermediary is what ultimately connects a and c.

This capability is important for fraud detection, root cause analysis and marketing attribution because sometimes an intermediary is what ultimately connects a and c.

Getting Ready for the Internet of Context

The best forecasting method for business purposes is not autoregressive and univariate. Instead, it is based on rich contextual metadata (i.e., other data sources that describe or co-occur with your primary data), which provide more accurate predictions for seeing around corners.

The need for better forecasting models that use ML is great today and will increase as more and more data flows into enterprises via the Internet of Things (IoT). It doesn’t matter how much data is flowing in; rather, the variety of data available to businesses will continue to increase (social, operational, behavioral, etc.) and expand predictability.

In fact, from the data scientist point to view, the IoT is really the “internet of context,” and for businesses in turn, it could become the “internet of opportunity.”

To learn more about unlocking the value of artificial intelligence and machine learning, check out Oracle’s AI webpage.


Article by channel:

Read more articles tagged: