Explainable artificial ­intelligence: the key to trusting machines

Subscribe to ABB Review

Jinendra Gugaliya ABB Corporate Research, Process Automation Bangalore, India, jinendra.gugaliya@in.abb.com; K. Eric Harper Former ABB employee

Artificial Intelligence (AI) models permeate our daily lives, often without our realization: For instance, Netflix uses a recommender engine to suggest movies to its users. This seemingly simple process uses machine learning (ML) to help the algorithms automate millions of decisions based on the user’s activities [1]. Facebook uses its patented process “computer vision contact detection system” to recognize objects as it sorts through images on user accounts for logos and brands so advertisers can target users with ads in sponsored story posts [2]. Beyond entertainment and advertising, big data science and ML is infiltrating mission critical applications like disease detection and diagnosis, loan application decision-making and self-driving cars. These applications can have a crucial impact on our lives. Which movies to view might be frivolous but whether a mortgage is to be issued or not is highly pertinent. Here lies the crux of the problem: these AI models are difficult to explain even among data scientists, thereby making acceptance problematic. Without the ability to explain the models, or “explainability”, it is likely that these models will not be accepted or used. Furthermore, the General Data Protection Regulation (GDPR) calls for greater transparency in data processing and clarity in AI processes, thereby making model explanations essential [3].

Whether humans or machines are involved, a decision rationale would foster an understanding of the motivation behind models and create a sense of urgency; this would increase the likelihood that the resultant recommendations could be accepted and implemented. Without this understanding it is impossible to trust the machines we build. So, ABB researchers delved into the history of explainable AI to illuminate how past lessons can enable future AI expansion.

Rationale for early industrial AI
This currently active research area began with the requirement that diagnoses be defensible. In the 1980’s the first operation-focused knowledge-based decision support system was invented and commercialized by Westinghouse with Carnegie Mellon University [4]. A contributor to the accomplishments cited in this paper, Eric Harper, was directly involved in GenAID, ­Turbin­AID and ChemAID (Artificial Intelligence Diagnostics) and contributed to intellectual property and software technology advances based on a key patent that focused on an operation system [5] with evidence and best practice to address abnormal conditions. Subsequent Westinghouse patents describe a method and system to exhaustively exercise a knowledge base in order to confirm that expected outcomes result from known inputs of abnormal data [6]. Harper built the tools and techniques for exploring knowledge and tracing support to justify specific actions based on diagnoses. This critical intellectual property is now in the public domain and these findings are still relevant today [7].

Innovations for explaining AI results
Recently, a prominent software consultancy adopted ideas to generate a model based on knowledge representation. Their US patent application suggests that a combination of k-means clustering, principal component analysis, forward or backward chaining, and fuzzy logic would resolve the general problem of explaining AI results in the real world [8]. Utilizing additional methods →01, a credit rating agency that manages credit risks runs models exhaustively with different inputs to successfully show customers how various inputs lead to different outputs, thereby helping customers understand the credit decisions made [9]. In a US patent application, Intel describes a technique to identify discrepancies between observed results from the machine learning training phase, and results obtained during operation [10]. Another Intel patent application describes the influence of neural networks for explainable AI: this process comprehensively traces dependencies and strength of support, between lower and higher neural network layers, back to their input features in a way that looks surprisingly like a forward chaining expert system [11]. Google combines these ideas with their tools and framework [12]; IBM established a similar platform [13].

center

Explainable AI trends
This resurgent research attention in XAI is classified into distinct categories based on the degree of transparency: graded from complete black box (low transparency) to white box characteristics (high transparency) [14]:
• Opaque systems
• Comprehensible systems
• Interpretable systems

Although there are many benefits to designing and developing XAI models, they can add significant costs. The trade-off is between explanation and accuracy. This balance and degree of transparency will be driven by business needs and how the application is adopted in the real world.

The logic behind the decisions of ML models is complex and not apparent. Trusting the resultant critical decisions without governance is problematic. This concern was daunting from the beginning: neural networks were available in the 1980’s for machine diagnostics but were not implemented. Hence, it is increasingly important to provide XAI in specific domains. In this way, XAI models can be subjected to formal verification, and this capability is especially important for medical applications where recommendations have life and death consequences.

Another challenge needs to be addressed before ML models can be readily accepted: a bias is exhibited if the training data does not span the complete solution space [15]. Such bias defects might be revealed if tests are performed across a wide range of conditions to benchmark the solution’s strengths and weaknesses.

Still, dazzling new insights can be won from XAI models. Today’s ML systems are trained through millions of examples so that data patterns can be recognized that are not obvious to humans. Westinghouse’s earlier dream that engineers would one day pore over the data collected from models like GenAID, apply data science methods and discover new innovations is almost realized. New insights have emerged through use of XAI systems: one can extract distilled knowledge from the machine learning to acquire new perceptions, eg, new strategies for playing Go¹ were developed with ML and are now used by human players.

Nonetheless, opaque decision support is not attractive to businesses. Clearly, a bank must communicate why a credit request was rejected. And, AI models must comply with the law by providing evidence for the decisions generated.

Generalization of these concerns led the European Union to adapt new regulations that implement a “right to explanation”, whereby a user has the right to ask for the rationale of a relevant algorithmic decision. There is hope that XAI will provide the confidence, trust, fairness and safety that is required for ML models in training and operating modes to win over businesses [16].

Explaining AI models: the current state of affairs
Currently, two approaches are used to make AI models more explainable. First, the model structures are inherently selected with an interpretation goal in mind. Alternatively, complex AI models are reverse-engineered to make them comprehensible. However, designing models with easy-to-understand rationale can compromise accuracy and vice versa, eg, complex Deep Neural Networks (DNN) are accurate but lack interpretability. Algorithms like linear regression or decision tree-based models are much easier to explain but less accurate. Striking a balance between accuracy and interpretability of AI models is currently the focus of intense research [17].

Another XAI research area examines the difference between local and global interpretability [18] →03. The local perspective, from sensitivity analysis (SA) principles, identifies how the model output changes with input or tuning parameter perturbations. Although not producing an explanation of the function value itself, SA can determine the factors and configurations that explain the model results. The global perspective uses two techniques. Layer-Wise Relevance Propagation (LRP) redistributes the prediction function backwards, starting from the output layer of the neural network and back propagating to the input layer. LRP explains the classifier’s decisions by decomposition and can be represented by heatmaps [19]. Data-driven Intrusion Detection System (IDS) is an adversarial approach used to find the minimum modifications (of the input features) required to correctly classify a given set of misclassified samples. The modification magnitude visualizes the most relevant features that explain the reason for the misclassification. Both LRP and IDS were combined by researchers to play Atari games driven by Deep Reinforcement Learning (DRL) [20].

center

Research in the field of XAI has been extended to enable dataset comparisons. Linguistic Protoform Summaries in tandem with Fuzzy Rules are used to design a system that can compare various datasets [21] numerically and explain differences using natural language. So-called, Human Machine teaming is a key factor in XAI research in which usability is an important consideration for model design and use. Here, ML models should allow a user to interactively tune the models based on iterative learning [22].

A practical approach for decision support
Engineering teaches us that any complex problem can be solved by dividing the work into multiple components, building and verifying them independently and integrating them back together for a complete solution. In the 1980’s, a Westinghouse system for explainable decision support and asset management invented techniques [23] to prioritize the repair of plant equipment by validating measurements and then fusing current conditions into system diagnosis and recommendation →02. A major electrical equipment manufacturer continues to use this system for site-based monitoring of power plants and turbo generators, and runs the service in their Power Diagnostics Center [24]. Fundamentally, potential issues can be ranked based on three dimensions:
• CF – confidence level in a diagnosis
• SEV – reciprocal of the time to failure
• IMP – costs caused by failure and to repair maximum damage

center

These dimensions are calculated for each of the equipment sensors, components and systems that contribute to a malfunction or outage: this combination is used to determine the action priority for each possible malfunction. The rationale for the ranking was explainable to customers: the three dimensions, and diagnosis details were traced back through components and sensor readings. In this way customers could accept the rankings without necessarily delving into the details.

center

Relevance to ABB
Since computing power was sparse in the 1980’s, automated service solutions were tuned to extract the best performance with limited resources. Nowadays, each of these originally construed characteristics can be calculated for even the finest granulation of components and then assembled based on their relationships and dependencies. ABB excels at employing knowledge and processes gleaned from early work for their automated service solutions. Today, ABB relies on a wealth of condition monitoring solutions that indicate confidence levels whenever problematic equipment issues arise →04 – 06. ABB Ability™ Advanced Digital Services solutions contain features that address the critical issue of time-to-failure. With talented engineers and information scientists available, ABB produces data science solutions for calculating the important dimensions: CF, SEV, and IMP where appropriate. A keen understanding of the overall cost of repair following failure positions ABB to diagnose issues based on data; this has led to new services like ABB Ability™.

center

As a pioneering leader in the industrial automation domain, ABB delivers value to customers by incorporating AI advances to improve predictive maintenance, optimization, and performance. Nonetheless, plant operators and managers must understand the justification and rationale behind decisions generated by AI models, recommended by operations and service applications, before they will implement potentially costly actions →05. By designing explainable AI in applications, ABB stands out in the market: This fosters trust – more crucial now than ever. When models are explainable, experts and end users can be assured that outcomes are bias-free, safe, legal, ethical and appropriate. 

center

Footnote
1) Go is an abstract strategy board game for two players.

References
[1] Code Academy, “Netflix Recommendation Engine”, Available: https://www.codecademy.com/. [Accessed: May 5, 2020]
[2] A. Razaaq, “Facebook’s New Image Recognition Algorithm Can Scan your Picture for Advertising Opportunities” in B2C Business to Community, May 21, 2019, Available: https://www.business2community.com/. [Accessed May 5, 2020]
[3] A. Woodie, “Opening up Black Boxes with Explainable AI” in Datanami, May 30,2018, Available: https://www.datanami.com/. [Accessed May 5, 2020].
[4] E.D. Thompson, et al., “Process Diagnosis System (PDS) – A 30 Year History” in Proc. 27^th Conf. on Innovative Applications of AI, Jan. 2015, Available: https://dl.acm.org/doi/10.5555/2888116. 2888260. [Accessed: May 5, 2020]
[5] Thompson, et al., “Methods and apparatus for system fault diagnosis and control”, US Patent no. 4,649,515, March 10, 1987.
[6] K.E. Harper, et al., “Expert system tester”, US Patent no. 5,164,912, November 17, 1992.
[7] Y. Lizar, et al., “Implementation of Computer Damage Diagnosis by Expert System Based Using Forward Chaining and Certainty Factor Methods” in International Journal of Scientific & Technology Research, vol. 8 issue 6, June 2019, pp. 141 – 144. Available: http://www.ijstr.org/. [Accessed May 5, 2020].
[8] L. Chung-Sheng, et al., “Explainable Artificial Intelligence”, US Patent Application no. 20190244122, August 8, 2019.
[9] FICO, “How to Make Artificial Intelligence Explainable: A new Analytic Workbench”, in FICO/blog, Sept. 13, 2018, Available: https://www.fico.com/blogs/. [Accessed May 5, 2020].
[10] J. Glen, et al., “Misuse Index for Explainable Artificial Intelligence in Computing Environments”, US Patent Application no. 20190197357, June 27, 2019.
[11] K. Doshi, “Mapping and Quantification of Influence of Neural Network Features for Explainable Artificial Intelligence” US Patent Application no. 20190164057, May 30, 2019.
[12] GoogleCloud, “Understand AI Output and Build Trust”, Available: Google Cloud: https://cloud.google.com/. [Accessed May 5, 2020].
[13] A. Mojsilovic, “Introducing AI Explainability 360”, August 8, 2019, Available: IBM Research Blogs: https://www.ibm.com/blogs/. [Accessed May 5, 2020].
[14] D. Doran, et al., “What does explainable AI really mean? A new Conceptualization of Perspectives”, in AriXiv, October 2, 2017, Available: https://arxiv.org/abs/1710.00794. [Accessed May 5, 2020].
[15] N. Mehrabi, et al., “A Survey on Bias and Fairness in Machine Learning” pre-print based on work supported by DARPA, in ariXiv, September 17, 2019, Available: https://arxiv.org/abs/1908.09635. [Accessed May 5, 2020].
[16] M. Miron, “Interpretability in AI and its relation to fairness, transparency, reliability and trust” in European Commission WITH HUMAINT, September 4, 2018, Available: https://ec.europa.eu/jrc/communities/en/community/humaint. [Accessed May 5, 2020].
[17] W.J. Murdoch, et al., “Definitions, methods, and applications in interpretable machine learning”, in Proc. of the National Academy of Sciences of the United States of America, vol. 116, issue 44, 22071 October 29, 2019, Available: https://www.pnas.org/.[Accessed May 5, 2020].
[18] A. Adadi and M. Berrada, “Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)” in IEEE Access, September 17, 2018, Available: https://ieeexplore.ieee.org/document/8466590 . [Accessed May 5, 2020].
[19] W. Samek, et al., “Evaluating the Visualization of What a Deep Neural Network Has Learned” in IEEE Transactions. On Neural Networks and Learning Systems, Nov. 2017. [Abstract]. Available: https://ieeexplore.ieee.org/abstract/document/7552539. [Accessed May 5, 2020].
[20] H. Jo, and K. Kim, “Visualization of Deep Reinforcement Learning using Grad-CAM: How AI Plays Atari Games?” in IEEE Conf. on Games, August 23, 2019, Available: https://ieeexplore.ieee.org/document/8847950
[21] A. Jain, et al., “Explainable AI for Dataset Comparison” in IEEE Int. Conf. on Fuzzy Systems, June 26, 2019, Available: https://ieeexplore.ieee.org/document/8858911. [Accessed May 5, 2020].
[22] A. Kirsch, “Explain to whom? Putting the User in the Center of Explainable AI” in Proc. 1^st Int. Workshop on Comprehensibility and Explanation in AI and ML, October 21, 2018, Available: https://hal.archives-ouvertes.fr/hal-01845135/ [Accessed May 5, 2020].
[23] Bellows, et al., “Automated system to prioritize repair of plant equipment”, US Patent no. 5,132,920, July 21, 1992.
[24] I. Becerra-Fernandez and R. Sabherwa, “Knowledge Application Systems: Systems that Utilize Knowledge” in Knowledge Management: Systems and Processes, 2^nd ed. New York: Routlege, 2015, pp. 3 – 105.