What Explainability in Machine Learning Looks Like
Surveying our pursuit of understanding
AI technology, to live up to its full potential, must become somewhat transparent in its reasoning. If you’re interested in the full argument I make for why that is, check out the article below.
Why AI needs to explain itself.
I’ve spent the last few weeks really evaluating the utility of current AI technology. For a while, I’ve felt as if there has been this massive leap in machine learning capability, yet no real significant breakthrough in what we should be using that power for beyond process automation. In thinking about what that breakthrough might be, I kept circling ba…
In short, real, practical integrations of machine learning technology beyond automation require the user to understand why a model made a particular decision. In areas such as disease diagnosis, genomics research, business decision-making, and more, advanced reasoning capabilities and understanding are required to make the large and small-scale decisions that will impact work in the future. If AI apologists want this technology to be authentically integrated into these processes, then it needs to explain itself and its value a bit more.
Large teams of people are working on Explainable AI technology, which can provide deeper insights into the reasoning behind a Deep Learning model’s decisions. This, to me, seems like where the real scientific breakthroughs might lie. That’s why I’ll be exploring a couple of popular forms of Explainable AI technology and the methods these techniques use to accomplish their goal.
Defining Explainability
Explainable AI (XAI) is a class of techniques that provide clarity through enhancing the interpretability of an AI model. Basically, an AI model is ‘Explainable’ if a human being can interpret its reasoning. Sounds deceptively simple, but the techniques required to accomplish this could make any layperson’s head spin.
Explainability can be accomplished in two main ways as of writing. Globally and Locally.
By-Design Explainability is when all a model’s logic is structured to be easily understood.
Post-Hoc Explainability is an additional technology applied to understand a specific individual prediction(s) made by a model.
By-Design Explainability, while useful for solving simpler problems, is limited by its very concept. Human understanding is limited, as is the mind, so any model acting within the constraints of human traceability and understandability won’t explain complex models.
This leaves us with a variety of Local Explainability techniques, most of which seem to focus on applications in Deep Learning, unsurprisingly. Deep Learning models may be the most complex pieces of programming ever invented, and are completely uninterpretable by human beings without the advent of Post-Hoc Explainability Technology.
Deep Learning technology is being used in medical research worldwide, and interpretability is highly coveted. So, what tools do we have now?
Explainability Technologies
*I am going to dive deep into what appear to be the two biggest and most researched model-agnostic explainability technologies as of today. By no means is this a comprehensive summation of all available research, but I believe this is a good place to start.
Local Interpretable Model-agnostic Explanations (LIME)
LIME adds complex model predictions in a relatable manner so experts can interpret the in-depth details of disorders such as dementia.
More specifically, LIME creates a surrogate interpretable model based on a particular ‘Black Box’ model. Then the surrogate model approximates the decisions of the ‘Black Box’ model and provides the user with a rationale for doing so.
“Your goal is to understand why the machine learning model made a certain prediction. LIME tests what happens to the predictions when you give variations of your data into the machine learning model. LIME generates a new dataset consisting of perturbed samples and the corresponding predictions of the black box model. On this new dataset, LIME then trains an interpretable model, which is weighted by the proximity of the sampled instances to the instance of interest. The interpretable model can be anything from Lasso to a decision tree. The learned model should be a good approximation of the machine learning model predictions locally, but it does not have to be a good global approximation”
The above quote is from Christopher Molnar, an Interpretable Machine Learning Researcher whom I suggest checking out to gain deeper insight into the topic. Read More Here.
The beauty in the approach here is that it’s almost entirely based on a simple computer science principle. Decomposition. If a problem is too complex, one of the first things your programming professor will tell you to do is to break it down. Simplifying is one of the most effective tools in the arsenal of understanding, and that’s exactly what the creators of LIME exploited in relation to Deep Learning Models.
If the goal is to understand why a decision was made, you can’t ask the black box. However, simulating the decision itself in a white box might help you understand the individual decision. It feels technically significant to realize that understanding can come from analyzing the rationale of a decision itself, rather than forcing an analysis of an unreliable decision-maker. The simulation doesn’t even have to do a good job of approximating the whole model, so long as it can emulate a single decision.
LIME is also Model-agnostic, meaning that it can be applied to any machine learning model with no work required to modify the model from within. This grants this technique a universality that makes it attractive to many.
The goal of LIME is to understand why a model made a decision, not what a model considers when making decisions. Understanding this helps us apply this technology correctly, as it’s not capable of understanding the entirety of a model’s ‘thought process’. LIME is excellent at isolating decisions and replicating them in more ‘transparent’ models, helping us gain a deeper understanding of the decisions individually.
SHapley Additive exPlanations (SHAP)
SHAP analyzes the model’s key features, which contribute to model performance and decision-making.
“When a machine learning model makes a prediction, one of the most fundamental questions we can ask is also one of the most difficult to answer: How much does each feature contribute to this specific prediction? This question matters profoundly, whether we’re debugging a model that makes unexpected decisions, explaining predictions to stakeholders, or ensuring regulatory compliance in high-stakes applications. Yet the answer is far from straightforward, because a feature’s contribution depends critically on what other features are present, creating a complex attribution challenge that simple difference-based methods cannot fairly resolve”
The above quote is from Michael Brenndoerfer, a Data Scientist and Entrepreneur who has some of the most beginner-friendly and insightful technical writing on explainability that I have seen on the internet. I’d also recommend that everyone read his work. Read More Here
While LIME focuses heavily on reproducing the decision-making process, SHAP takes a more analytical approach. SHAP draws on Game Theory’s Shapley Values to create a mathematical framework for feature attribution. This framework applies seamlessly to every machine learning model, giving SHAP universality in the same vein as LIME.
Shapley Values are, in simple terms, mathematical values that analyze the contribution of particular features to an outcome. In Game theory, they are used to predict the outcome of a game. In machine learning, the outcome is known while the features remain a mystery. Game Theory prescribes a formula to understand the features that contribute to any outcome. In other words, Shapley Values give mathematical validity to the explanations provided by SHAP.
SHAP can also explain local decisions and the features that a model finds significant globally, making it both locally and globally explainable.
The goal of SHAP is to understand exactly what a model deems as important. Using a well-established mathematical framework, SHAP provides a level of rigidity and justification in its explanations unrivaled to this day. SHAP is excellent at giving us concrete ideas about how a model justifies its decisions.
Conclusion
While the Black Box remains black, we have a plethora of technologies that allow us to interrogate its rationale. This, while still not a glass box, is radically more transparent than I would’ve thought before researching this.
It’s important to emphasize that none of these technologies has reached maturity. These techniques were invented in 2016 and 2017, respectively, so we may be in for much more growth in their development. Isn’t that exciting?!
I genuinely believe that the development of technology like this is what will lead our society to the better world we deserve, so long as we don’t allow tech like this to be monopolized, gatekept, and sold to the highest bidder. Open Weight/Source tech should be prioritized, and I hope that research sharing will continue.