Transforming Data into Action
The business value of Data Science depends upon aligning AI projects with corporate strategy, culture and resources
Many companies are investing heavily in artificial intelligence because of the perceived need to “do something will the myriad of data that is being captured each day”. Gartner calculates that these investments represent $383 billion this year, in spite of the fact that the largest majority of AI projects have no real-world applications.[i] If the potential benefits of AI are hard to ignore, successful projects depend upon the ability to weigh AI’s potential with organizational realities. How can management align experiments in Data Science with corporate strategy, encourage their Data Science teams to produce models that are directly applicable to the business, and demonstrate how their investments impact the bottom line?
The case of financial fraud is a very visible example of a case in point. Industry observers estimate that online credit card fraud alone will cost the financial community $32 billion during the current calendar year.[ii] According to Javelin Strategy & Research, it takes more than 40 days today for banks to detect fraudulent transactions using traditional rule-based systems. The real costs of these prejudices are even higher, for 20 percent of customers change their banks after experiencing scams. Data Science can transform this challenge into an opportunity in facilitating real-time claim assessment and improving the accuracy of fraud detection.
Yet, despite multiple AI projects over the last decade, neither the traditional banks nor FinTech has been able to make this challenge go away. In practice, current work in applying Data Science to fraud detection follows one of these three scenarios. Anomaly detection builds upon the current rule-based systems to suggest that inefficient processes are the problem, and the solution is in detecting non-respect of the financial processes. A second approach uses predictive analytics to explore purchasing behavior, suggesting that behavioral profiles are the key to fraud detection. Finally, a third approach attempts brute force in harnessing SVMs and neural networks based on the belief that advances in prescriptive analytics will certainly produce the desired results. Given the real costs and potential benefits, in which scenario should management invest?
The Business Value Matrix
Kees Pronk and I introduced the Business Value Matrix™ (BVM) several years ago to explore the relationships between data, context and business value.[iii] The matrix is based on several premises. Managers and their customer don’t see data in the same light because they look at value from different angles. The value of data depends upon corporate strategy, culture and context. The business value of the algorithm doesn’t come from its precision, but from its use in addressing customer challenges. Finally, a manager’s job doesn’t end with analyzing the data, but in using the data to incite action. Let’s examine the construction of the matrix, and then apply it evaluating the different data-based initiatives in fraud detection.
Add alt text No alt text provided for this image The Business Value Matrix™
The Business Value Matrix (BVM) suggests that data can be aggregated to explore customer challenges along three axes : where do stakeholders feel that values comes from, at what level of aggregation are they looking for proof of value, and what metrics are they using to suggest to qualify success. For a Data Science team, the Matrix can be used to provide three deliverables — to explore the best fit between the scope of the AI project and corporate culture, to suggest where to they need to provide proof of value to their stakeholders, and to provide the metrics that can incite management to transform data into action. Let’s explore each axe in turn.
Where does value come from? The first question underlines both how each stakeholder qualifies the problem, and where he or she believes the solutions lies. For some stakeholders, a successful business is built upon well documented, well organized processes — customer challenges reveal process defects that can be detected, elucidated, and addressed. For other stakeholders, successful projects are built around understanding their internal and external customers: challenges to the business require efforts to better understand the objectives, visions and incentives that push people to want to work with the organization. Finally, in the third case, businesses are built around technology — a deep seated belief that innovations in information and physical technologies alone can positively impact the bottom line. Given their particular views on business value, stakeholders collect and monitor different kinds of data as proof of current problems and future opportunities for the organization.
At what level are stakeholders looking for proof of value? This second question addresses three possibilities of providing proof of concept. For those who believe that individuals can make a difference in how a firm performs; data on individual productivity serves as a proxy for organizational success. For others a business is only as good as its weakest link — here data on how the organization performs as a whole is the best indicator of success. Finally, for certain managers or stockholders, the only real proof is in the market itself — productivity metrics must be analyzed in light of customer satisfaction and market share. Data scientists here need to understand which types of data are important to stakeholders and focus their efforts in providing evidence at the level where stakeholders are looking for proof of concept.
How do stakeholders measure success? For a certain number of decision-makers, success is a question of efficiency — data needs to reveal the costs and benefits of the activity in question. For others, effectiveness is a much better measure of success — data needs to reflect the quality of the relationship between the organization and its customers. For others, innovation is the key metric, data needs to illuminate how well the organization responds to external threats and opportunities. For others still, utilization provides the primary metric highlighting how well organizational resources are employed over time. In each case, understanding how stakeholders measure success is a precondition to providing the data that will help them transform the evidence into individual or collective action.
Anomaly detection seeks to organizational processes
Can BVM help Data Scientists their efforts to improve credit card fraud? Anomaly detection algorithms build upon the traditional methods of rule-based processes. The learning model is trained on a continuous stream of incoming data that is labeled as either fraudulent or legitimate. During the test phase, a human agent is then notified of the deviations form acceptable patterns for review. The model is thus trained to accept a baseline “of normalcy” for the contents of banking transactions. This baseline doesn’t reflect consumer behavior per se, but degrees of adherence to organizational processes. Anomaly detection will appeal to stakeholders focused in improving the processes of fraud detection, rather than those interested in better understanding consumer behavior or gaining a technological edge.
Add alt text No alt text provided for this image Image Credit : Kim Phuc TRAN
Automating rule-based fraud detection
If the value here is in improving the process, where would the stakeholders look for proof of concept? It is unlikely that that they would look for proof on an individual basis which might be captured in understanding the number of true positives in the test data. It is also possible, but also unlikely, that they would being interested in using anomaly detection to gain a competitive advantage in the market, since the data model, no matter how efficient, would do little to impact market share. Proof of concept would most likely be sought for the business unit as a whole: a better process would benefit all employees regardless of their personal knowledge or contribution.
Finally, for sponsors interested in improving the process, how would they measure success? On one hand, efficiency metrics, including a reduction in fraudulent transactions or a reduction in the cost of inspecting each transaction would certainly be welcome. On the other, metrics that measure effectiveness would be of little help here, for the data produced would do little directly to improve the relationships between the organization and its customers. In a similar light, metrics tied to innovation would also mean little here, for the objective is to refine the process rather than design new service offerings. In catering to stakeholders that believe that value comes from the process, anomaly detection most often needs to provide proof of concept around efficiency metrics at the organizational level.
Clustering algorithms seek to improve the understanding of consumer behavior
Which approach to fraud detection would appeal to stakeholders more interested in understanding consumer behavior? Predictive analytics leverage pattern clustering and/or neural networks to identify outliers in flows of transactional data.. Methods including K means, Density Based or Mean Shift clustering can be used to study spending behavior, allowing machine learning models to elucidate the footprints of fraudulent behavior associated with retail shopping or eCommerce transactions. Exploring spending patterns will appeal to sponsors more interested in improving the organization understanding of particularities of consumer behavior than standardizing operational processes.
Add alt text No alt text provided for this image Image credit: Dhruv Sharma
Exploring clusters of consumer spending patterns to understand fraudulent behavior
If the value here focuses on better understanding human nature, where are stakeholders looking for proof of concept? Three scenarios need to be considered carefully. The ability to elucidate individual consumption patterns using demographic and behavioral data is certainly a plus, though concerns with respecting consumers’ data protection rights may provide an obstacle here. Proof of concept for the market as a whole may prove unrealistic, for the premium may well be put on personalizing the organization’s service offer for targeted populations. Exploring behavioral data to identity the characteristics of consumer profiles appears be a tantalizing alternative.
For stakeholders who believe that value comes from understanding human behavior, how would they measure success.? Efficiency metrics will be less important here given that they measure costs associated with existing organization processes, rather than the opportunity of developing new markets and services. Metrics around the effectiveness of the service offer will have more traction here in describing the quality of the relationship between the organization and its consumers. Finally, innovation would also gain traction in designing new services that permit the organization to better serve its target population. Predictive analytics that develops both human and machine intelligence provide an appealing road forward for stakeholders who believe more in people than in processes or technology.
Can technology alone provide a competitive advantage?
Prescriptive analytics aim to offer organizations solutions that identify ‘fraud signatures’ before they can even take hold and make lasting damage. Data Science teams can build estimation models from analyzing hundreds of millions of transactions and submitting them to a predictive analytics engine (integrating random forest, naïve Bayes, support vector machines….) to provide recommendations for what to do once fraud is detected. Prescriptive analytics take the inputs from predictive models, combined with rules and constraint-based optimization, automate decisions about what to do. This type of machine learning effectively takes “human” judgement out of the equation in calculating the Highest Possible True Positive Rate (TPR) and the Lowest Possible False Positive Rate (FPR).
No alt text provided for this image Image credit : Kathrin Melcher, Rosaria Silipo
Prescriptive analytics based on recommendation agents
If the value proposition here is the technology, where are the stakeholders looking for proof of concept? Three scenarios need to be considered carefully. The ability to identity individual consumption patterns would certainly be a plus, though concerns with respecting consumers’ data protection rights may provide an obstacle. The adoption of prescriptive analytics may also meet resistance from both employees and managers who feel that fraud detection is part of the skill set that they get paid to provide. Pitching proof of concept to the market as a whole may none-the-less be appealing to stakeholders looking for a means of creating sustainable competitive advantage.
How would sponsors who believe that value comes from technology measure success? Efficiency metrics will be difficult to pitch given the cost of developing a viable solution for fraud detection. Metrics around the effectiveness of technology won’t make much sense either given the probable resistance from employees, consumers, and the regulatory authorities. Organizational learning is also highly unlikely given the challenges of the transparency and explainability of these machine learning models. Proponents of prescriptive technologies are left with the metrics of innovation to demonstrate the value of these new use scenarios for fraud management.
Conclusion
The end game of Data Science isn’t analyzing the data but addressing business problems. Successful AI projects depend upon managerial abilities to align the potential of machine learning with the organizational realities reflected in corporate strategy, culture and available resources. The Business Value Matrix (BVM) provides a framework for exploring these realities in focusing on stakeholder perceptions of where value comes from (process, people or technology), where they are looking for proof value (at the individual level, for the organization, or in the market as a whole), and how they qualify success (efficiency, effectiveness, utilization, innovation…).
We have explored the potential value of the BVM in the case of fraud management. Three potential approaches, based on anomaly detection, predictive and prescriptive analytics correspond to contrasting views of the value of processes, people and technology. In exploring stakeholder views through the matrix, Data Science teams can provide data their management is looking for to add value to their organizations. Data Science isn’t about doing something with the myriad of data captured each day but doing something that helps management transform data into action.
