Any sufficiently advanced technology is indistinguishable from magic.
Arthur C. Clarke 
In the world of software development, there are well-defined testing paradigms and use-cases. Many teams will develop a suite of tests where the internals of the software programs are unknown, to assert the data output from a software system is as expected given the data fed in -- i.e., black-box testing. Conversely, in white-box tests the program structure, design, and implementation being tested are known.
In the world of artificial intelligence & machine learning (AI & ML), black- and white-box categorization of models and algorithms refers to their interpretability. That is, given a model trained to map data inputs to outputs (e.g. emails classified as spam or not), if the mechanisms underlying predictions cannot be looked at or understood, that model is black-box. And just as the software testing dichotomy is high-level behavior vs low-level logic, only white-box AI methods can be readily interpreted to see the logic behind models' predictions.
In recent years with machine learning taking over new industries and applications, where the number of users far outnumber experts that grok the models and algorithms, the conversation around interpretability has become an important one. Exponentially so with the rise of deep learning, a class of machine learning methods known for their powerful performance yet elusive, black-box nature. This is a conversation with implications up and down the stack, from AI scientists to CEOs.
What is interpretable AI?
The definition of interpretable AI isn't exactly black and white. To have a productive conversation it's essential to be clear what model interpretability means to different stakeholders:
Both definitions are clearly useful. The low-level notion of interpretability lends itself to the engineer's ability to develop and debug models and algorithms. High-level transparency and explainability is just as necessary, for humans to understand and trust predictions in areas like financial markets and medicine [4,5].
No matter the definition, developing an AI system to be interpretable is typically challenging and ambiguous. It is often the case that a model or algorithm is too complex to understand or describe because its purpose is to model a complex hypothesis or navigate a high-dimensional space, a catch-22. Not to mention what is interpretable in one application may be useless in another.
Peering inside the box
Consider the visualizations below of deep Gaussian Processes, a probabilistic variety of deep neural networks . The intent of this experiment by Duvenaud et al. is to elucidate a subtle architectural flaw in neural networks as more and more layers are added; the specific insights are beyond the scope of this article, see . Even with the paper's mathematical details of the model architecture and description of the subsequent adverse effects, the "pathology" is unintuitive to understand let alone debug and fix. The visualizations are quite effective in communicating modeling insights that would otherwise go unnoticed.
Post-hoc interpretation methods can provide insights into otherwise black-box AI systems. Nonetheless, the underlying models and algorithms may still be unexplainable. What then is a white-box model? The counterpart to post-hoc is model-based interpretability, where the model itself readily provides insights into the relationships and structures it learns from data. For example, a Gaussian Process is a flexible Bayesian model that enables feature engineering and incorporating domain expertise, and the predictions are intuitive to trace back to underlying logic -- all properties of white-box ML. A Gaussian Process model can be powerful for many real-world machine learning tasks, from financial markets to autonomous robotics [8, 10]. Sometimes stacking Gaussian Processes into a complex network results in a significant performance boost, but also shifts the resulting model towards the black-box end of the spectrum. Developing an interpretable machine learning system can be a practice in building models for descriptive accuracy, but at the cost of predictive accuracy.
Do visualization methods make black-box models interpretable? Not quite. Even with advanced interpretation techniques such as Google's "Inceptionism" , deep neural networks remain prohibitively complex to understand: the explanations underlying predictions (or the "why") is unknown. Consider the remarkable sensitivity of deep neural networks to adversarial attacks, where slightly perturbed inputs (typically imperceptible to humans, such as a piece of duct tape on a stop sign) can completely throw off predictions , raising questions around what the models are actually learning inside that black-box.
The real issues with AI interpretability
Even with improved methods and algorithms for explaining AI models and predictions, two core issues must first be addressed in order to make legitimate progress towards interpretable, transparent AI: underspecification and misalignment.
The notion of model or algorithmic interpretability is underspecified -- that is, the AI field is without precise metrics of interpretability. How can one argue a given model or algorithm is more interpretable than another, or benchmark improvements in explainability? One method could provide beautifully detailed visualizations, while the other provides coherent natural language rationale behind each prediction. It can be apples-and-oranges to compare models on account of their interpretability.
For example, a heuristic for model-based interpretability is "simulatability": whether or not a human is able to internally simulate and reason about the model's entire decision-making process . Yet there is no way to consistently quantify this heuristic, or any of the potentially dozens of interpretability measures. The development of interpretable AI is on a random walk without metrics to guide and measure progress.
Yet more confounding is having to choose between model performance and interpretability, particularly in medical applications where transparency and trust are critical . Herein lies the second core issue: misalignment. Perhaps when faced with a performance vs interpretation tradeoff, solutions can be found with interpretation interfaces and modules: use powerful black-box models, and layer-on an interpretation module for post-hoc inspection and explainability. This can work well in medical practice, as shown above and others . Yet the misalignment cuts deeper:
We want good models. We also want interpretable models. Thus the human wants something the performance metric does not.
Zack Lipton, Asst. Professor, Carnegie Mellon 
In general a machine learning system is built and trained to optimize a specified target objective: classification accuracy in a spam filter or tumor diagnostic, efficiency in route planning or Amazon box packing. Unlike these precise performance metrics, the criteria of safety, trust, and nondiscrimination often cannot be completely quantified. How then can models be trained towards these auxiliary objectives? Case in point, Google's embarrassing episode with a racist image classifier .
Underspecified and misaligned notions of interpretation impede progress towards the rigorous development of understandable, transparent, trusted AI systems.
The path forward
There is strong activity in developing interpretable AI models and algorithms [11,18], largely driven by the implicit motivations of AI scientists and engineers to both advance theoretical foundations and build real-world AI systems. Thought-leaders such as Zack Lipton , Suchi Saria , and Mihaela van der Schaar  are instrumental in moving the field forward.
External forces are also pushing for the development of explainable AI, notably in the arenas it matters most. The Defense Advanced Research Projects Agency (DARPA) is putting $70 million towards a new "Explainable AI" program, with a focus on interpreting the deep learning systems in drones and intelligence-mining . Perhaps more influential is a set of guidelines from the EU towards ethical development and deployment of AI by companies and governments. A main requirement put forth is for transparency, where AI systems must include explanations for their models’ internal logic such that data and decisions could be traced and understood by humans .
One hope is that with more rigorous science [18,16], improved methods for model interpretability can lead to a more fundamental understanding and theory of deep learning. For example, inspecting deep neural networks with counterfactual perturbations to discover which features are most important for resulting predictions , a method which could bear fruit in the development of deep causal inference. Nonetheless deep learning, interpretable or not, is useful across industries and applications, and is not slowing down.
Notes & References
 Lesser known are the other two of Clarke’s three laws:
When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.
The only way of discovering the limits of the possible is to venture a little way past them into the impossible.
 Source: hanne-morstad.myportfolio.com
 In machine learning one will often want to know how well a model's posterior distribution is approximating the underlying true distribution; we cannot validate whether a model is true, rather, "all models are wrong" (Box '76). Posterior predictive checks (PPCs) generate data from the model using parameters from draws from the posterior, and thus provide a great way to validate a probabilistic model.
 Consider the example of breast cancer diagnosis. Given mammogram images and a machine learning model trained to identify malignants, the prediction output would need more substance than a likelihood to be useful. Visualizing predictions as heatmaps is a common method, which enables clinical use by explaining the prediction but also in a visualization the integrates in the doctor's workflow. See for example the deep neural net for breast cancer screening by (Wu et al. '19), which just won Best Paper Award at the 2019 ICML Workshop on AI for Social Good: tweet with poster, paper, and code. Similar methods can be deployed in practice to enable trust with doctors and patients, notably the vision systems from MobileODT to detect cancerous skin tissue with mobile colposcopy.
 Despite proven successes of machine learning methods in healthcare research and practice (Yoo et al '12), deployment has been limited: only 15% of hospitals in the U.S. use machine learning, and only for rather limited purposes (Luo et al '17). Why? Healthcare researchers often lack machine learning expertise to utilize advanced methods, let alone trust them in practice.
 A deep Gaussian Process is a deep belief network where Gaussian Processes (GP) govern the mappings between layers (Damianou & Lawrence '13). One can also think of this architecture as stacked multivariate GPs.
 (Duvenaud et al. '16) show through brilliant derivations and visualizations that in standard neural network architectures, the representational capacity of the network tends to capture fewer degrees of freedom as the number of layers increases, retaining only a single degree of freedom in the limit. They then propose an alternative, generalizable architecture do avoid this "pathology".
 A personal favorite ML book, and must-read for anyone developing probabilistic ML methods: Rasmussen & Williams, "Gaussian Processes for Machine Learning", 2006. For an intro into deep GPs, see (Gram-Hansen & Roberts '15) on the white-box nature of GPs and stacking them for more data insights.
 Consider prowler.io, an impressively capable ML intelligence company with logistics and finance solutions built from Gaussian Process foundations.
 "Inceptionism" is a method for inspecting deep neural networks, specifically the "Inception" network used in image classification applications: Google AI blog post. More from Google Brain, see the work of (Olah et al. '18) towards more capable interpretation interfaces resulting from the combination of interpretation modules. Also explaining some advances in visualizing internals of these convolutional neural nets is Jason Yosinski et al. (Uber AI, Cornell U.) in their "DeepVis" blog post.
 Most research in adversarial AI is in the vision domain, but here is an interesting example in natural language processing: (Carlini & Wagner '18)
 See (Murdoch et al. '18) for a nice breakdown of model-based and post-hoc interpretability methods.
 For example, algorithms predicting trauma to help manage resources in the ICU (Liu et al. '14).
 For example (Lahav et al. '18) first train a neural network to provide risk assessments for heart failure patients, and then design an RL-based clinical decision-support system (DSS) around the neural network model. The DSS presents a sequence of interpretability modules and other forms of evidence about the underlying ML model to the user (e.g., information about the dataset, training methodology, and model accuracy).
 Zack Lipton, asst. professor at CMU, is a thought leader in the AI research community, including his article on "The Mythos of Model Interpretability" (Lipton '17). His posts at approximatelycorrect.com are insightful, provoking, and occasionally humorous.
 There have been some developments towards more formal processes and metrics around interpretable machine learning, e.g. (Doshi-Velez & Kim '17). Exciting activity has been in the discussion and research around dedicated conference workshops: e.g. VISxAI, and at top ML conferences ICML and NeurIPS (also).
 Professor Mihaela van der Schaar leads the ML-AIM lab with the Turing Inst. She's co-author on  and many other works developing AI tools for use in clinical practice.
 See this video on DARPA's initiative towards explainable AI. DARPA has a proven track record of spurring AI fields, notably the Driving Grand Challenge of 2004 that massively boosted the development of autonomous vehicles.
 (Burns et al. '19)