Why AI needs to explain itself.
Would you stick your hand in a Black Box?
I’ve spent the last few weeks really evaluating the utility of current AI technology. For a while, I’ve felt as if there has been this massive leap in machine learning capability, yet no real significant breakthrough in what we should be using that power for beyond process automation. In thinking about what that breakthrough might be, I kept circling back to one key industry. Healthcare.
The scientific community at large has been interested for decades in the possibility of detecting diseases before they present themselves. For most of human history, we have only been able to diagnose diseases through external symptoms; this is steadily changing. Since the late 18th century, we’ve been able to use advanced technology to identify cellular and genealogical differences between the sick and the well. This is how we obtained a greater understanding of how to prevent diseases and asses risk of infection, among many other things.
For the last half-century, we have been collecting petabytes of data regarding individual diseases, including MRI scans, blood pressure tracking, and even autopsy reports. It would be impossible for one teamof humans to parse all this data, even thousands of the world’s most qualified scientists have struggled with a modicum of that volume of information. So imagine what we could accomplish if we could effortlessly parse terabytes in minutes?
Dozens of tech billionaires are confident that the time has come. Sam Altman, CEO of OpenAI, would go as far as to say in reference to cancer, “If AI stays on the trajectory that we think it will, then amazing things will be possible. Maybe with 10 gigawatts of compute, AI can figure out how to cure cancer”. The ‘maybe’ is doing a lot of heavy lifting there. His take only considers technical factors, such as compute power, data-center support, financial backing, etc. It’s devoid of the more ‘human’ variables, one of which is the ability to understand and explain the rationale behind certain decisions. Here, AI tech is far behind the curve.
AI and ‘Understanding’
I think it’s important to characterize AI models correctly, as I don’t believe many people accurately understand what a Machine Learning model is capable of. An AI model can ‘think’ in a very rudimentary sense. Model size determines the complexity of the ‘thinking’ much of the time, but definitionally, AI models are able to recognize patterns.
Deep Learning models are larger, more complex webs of nodes that can process more data and recognize more complex patterns than standard models. This is the technology that empowers researchers to create AI models that can identify Alzheimer’s disease, various cancers, etc.
Now, just because these models can identify such patterns does not mean that these models are capable of ‘understanding’ the information that has been fed to them. These models are given rounds of ‘training data’, which show the model a set of information to analyze for patterns. After the model is trained, it cannot expand its ‘knowledge’ until it is trained again. This opens the possibility for the model to respond unintelligibly if asked to process information it hasn’t seen before.
Even without that possibility, AI models can still be incorrect. Just because a computer processes information differently than us does not mean it’s incapable of error. I feel like I see contradictory rhetoric too often online, so I feel the need to stress the following: Unless these models are used exclusively within the context it has been trained for and are heavily refined, preferably by an expert, these models can be frequently wrong.
Accuracy rates of 90% or above are difficult to come by, often only achieved by models with extremely simple or niche applications or by teams of experts tirelessly training models for months. Even if high accuracy is achieved, one of the biggest problems with this technology still prevails.
Deep learning models alone are uninterpretable by human beings. The technology behind these models’ reasoning is often too complex to track, meaning that you’ll have no idea why the model made the decision that it made. To you, you just gave an input, and then an output appears out of thin air.
For better and for worse, Deep Learning Models are a Black Box.
Trust the Black Box
If I asked you to place your hand into a box you can’t see into, would you do it? Would you hesitate? How much trust can you have in something you can’t know the contents of?
Better scenario. What if I told you that I hand-picked a variety of medical documents related to BMI and overall health and placed them into a magic box that would summarize them all and give you medical advice based on your information. How much do you trust that box? You know the box is an inanimate object, that it cannot understand the true meaning of its inputs beyond the syntax and order of the numbers. You’ll never know how the box makes its decisions, you don’t even know what information the box was fed!
To be clear, this is not a new phenomenon. The above scenario is just a tech-infused version of the reality that many of us already face. Black and brown people within the United States and across the world have been subject to misdiagnosis and poor medical treatment due to poor representation of us in research and academic literature. To this day, the Body Mass Index does not accurately account for the average differences in muscle and bone density within people of African descent. I bring this up to say that this issue is not only important now, but that it has always been of importance.
The existential problem here is that you cannot reason with an AI model. You cannot convince it of anything, as it is only capable of ‘understanding’ within the bounds of its training data. I believe that alone makes this technology a threat to everyone if used without caveats. Expert or layman, the Black Box is not to be uncritically trusted.
Experts and Technology
In any industry, but particularly in scientific fields, experts are encouraged to dive headfirst into research to gain a deep understanding of a topic before they are given the authority to form any conclusions about a topic. This means those experts tend to gravitate towards tools that provide a level of explainability to the conclusions formed.
To a breast cancer researcher, a model that can predict breast cancer is nothing more than an automation tool without the ability for that model to say exactly why it made that prediction. I say that not to demean automation tools, as they absolutely have their place. What I’m trying to point out is that the breakthrough discoveries that tech billionaires are touting will come from this underlying technology, won’t come from the underlying technology. It will come from the many teams of people fervently trying to make the black box into a glass one.
The way that researchers are doing so is deceptively simple. We just need acetone.
Stripping the Paint
Even though the nature of Deep Learning models is to be closed, ML Researchers from across the world have helped develop early technologies to make prediction models more opaque in their reasoning.
There is an entire subsection of the tech industry focused on developing Explainable AI (XAI in short), which is a set of processes and methods to enhance the explainability of Machine Learning models. There exist frameworks and tools like Local Interpretable Model-agnostic Explanations (LIME) and SHapley Additive exPlanations (SHAP), both of which have been featured in several research papers and been deemed promising by the scientific community. While this technology is impressive, it’s far from mass adoption and will require significant funding, research, and time until it’s seen in clinical settings.
The truth is, there’s a long way to go until the black box is fully glass, but we’re on the way. In the meantime, it might be good to reconsider how we think about and use AI technology in our daily lives. I think the public should be informed about what will actually get us to the future we want, instead of being led into trusting the men who have every monetary incentive to lie to our faces.
Please, until the box is glass, don’t put your hand in it.