Since early 2025, AI companies have not shut up about Agentic AI. The concept of an omniscient AI model becoming your personal assistant is delightfully sci-fi, and has been brutally difficult to construct. It seems like companies have been trying, and consistently failing, to package this technology within consumer products in ways people see value in.

I’ve spent a non-insignificant amount of time critically evaluating the latest Machine Learning technology and techniques. I’ve been quite clear in my coverage that a not-so-significant portion of the tech industry is leaning on excitement rather than a rational assessment of capability. So naturally, I don’t get excited about new AI technology very often.

Agentic Software Development broke this pattern.

What could this be?

Imagine for a moment that you had a choir of small, local, and highly specialized AI models at your disposal, which you could conduct to your liking. One model could specialize in writing unit tests, another might excel at general code review, and another could optimize data structures for efficiency.

For the first time in a long time, I see adoption of machine learning technology that actually leverages foundational principles of AI. Here’s what I mean:

Agents allow for models to be highly specialized. An orchestra of agents implies many specialized models. I’ve mentioned that AI models work best when they’re individually and selectively specialized. A team of accurate, lightweight models can work far more efficiently than a heavy, energy-inefficient one.

Agents allow for an expanded model context. Having multiple local models allows the user to grant access to far more than can be given through a browser window. Agents can interact with applications on your computer, maybe even do menial tasks for you (reliability may vary). This may suggest more practical implementations than standard GenAI use cases.

Agents can increase efficiency in your workflow. Current software engineering workflows can be slow when you’re working with a smaller team. Agents specialized in specific software development subcategories can help solo developers act efficiently when building complex programs. (I haven’t explored many use cases outside of software engineering, though I’m sure that valid ones do exist.)

Agents can localize your AI experience. AI companies are currently dependent on owning your model context. They capitalize on using our data to generate revenue with no benefit or credit to the consumer. When you run these models locally, all information shared stays on your device, effectively securing your data. This may entice companies into building better products to push users to return to their platforms.

These are just a few of the logical reasons why this technology intrigues me. However, no logical reason could be as exciting as the picture painted by many champions of this technology: the symphonic dream of directing a legion of agents towards completing the world’s most meaningful work. A swarm of agents running on Gigawatts of compute power is directed to analyze mammograms until we find the cure for breast cancer. Without exaggeration, an Orchestral masterpiece of technology.

Why an Orchestra?

The ‘orchestra’ analogy isn’t something I made up. This is the analogy I have most commonly used to communicate why this technology is so transformative.

Ever since the origin of software development, programming has been an intricate, technical practice. To create good software, programmers historically benefited from an in-depth knowledge of computer processes. This has led to a culture of specialization within the industry. The best software developers were the ones who knew a hell of a lot about a particular area of computing.

This obsession with technical mastery is not too dissimilar from classical music.

Classical music as an art form rewards those with in-depth knowledge of music theory. Instrumentalists permitted to play in an orchestra are often those at the top of their field, known for their technical mastery of a particular instrument. The flute is sonically and technically distinct from the cello, as is the piano from the xylophone, as is the tenor saxophone from the alto, and so on. Each is distinct with its own part to play, but they combine in an orchestra to create this incredible concoction of sounds we call an arrangement.

Atop all performers in an orchestra sits the Conductor. Traditionally, an expert in multiple, if not all, areas of instrumentation, the Conductor directs each element to form the ‘correct’ arrangement. The Conductor potentially plays the most pivotal role, as without direction, a symphony would be nothing but a collection of noises. They are also respected as masters of the craft in a way that an instrumentalist can’t attain without composing or conducting music.

Most importantly, the conductor is essential. An arrangement can miss elements and still stand on its own. You can have a beautiful piece without some subcategories of instrumentation, but without the conductor or the arrangement, there will be no music. I think this is the key to understanding the hype behind agentic programming.

I think the software development community is concluding that they need to become essential to keep doing what they do. The old world prioritized specializing to stay valuable, and this new one transforms that specialization into a liability. In a world where so much can be automated away, the only true way to stay secure and grow is to attain a level of ‘mastery’ that AI can’t reach.

The problems start to arise when you conflate mastery with management.

Programming is not an Art

Leading an AI team does not make you a conductor. At best, you’re a manager. Much less romantic, isn’t it?

Agent orchestration doesn’t require you to have an in-depth understanding of every area of software development. One look at Google’s ADK (Agent Development Kit) documentation is enough to prove this technology isn’t aimed at experts. This tool is meant to enable smaller teams to accomplish more.

I could honestly write another 500 words on the false dichotomy between a conductor and an agentic orchestrator, but I don’t think that’s needed. You know software development isn’t art, and you know that it doesn’t require the same dedication to computing that conducting requires to instrumentation and music theory. I think it’s incredibly interesting that this is what the industry has decided to latch onto, and there seem to be interesting implications within. However, none of this changes my excitement about Google’s ADK and getting to use it.

That begs the question, why do this as opposed to programming normally?

Jack of all trades, Masterful headache

Programming is hard. It’s repetitive, mentally taxing, time-consuming, monotonous, syntactically unforgiving, and lots of other words I found in my thesaurus. Building a complete piece of software is a long and arduous process involving dozens of intricate steps all buried within each other.

I asked Google Gemini to ‘generate me an exhaustive list of aspects of Software Development’. Here’s the output:

• Requirement Analysis: Gathering and documenting what the software needs to do from a business and user perspective.

• Feasibility Study: Assessing whether the project is technically and financially viable.

• Project Management: Choosing a methodology (e.g., Agile, Scrum, Kanban, or Waterfall) to manage timelines and resources.

• Product Roadmap: Defining the long-term vision and specific milestones for the software’s evolution.

• System Architecture: Defining the high-level structure of the software, such as Monolithic, Microservices, or Serverless.

• Database Schema Design: Planning how data will be stored, related, and indexed (SQL vs. NoSQL).

• API Design: Creating the interfaces through which different software components or external services communicate (REST, GraphQL, gRPC).

• UI/UX Design: Crafting the User Interface (visuals) and User Experience (flow and usability) to ensure the product is intuitive.

• Frontend Development: Building the client-side interface that users interact with directly.

• Backend Development: Implementing the “under-the-hood” logic, server-side processing, and database management.

• Version Control: Managing code changes and collaboration using tools like Git.

• Environment Configuration: Setting up development, staging, and production environments.

• Unit Testing: Testing individual components or functions in isolation.

• Integration Testing: Ensuring that different modules or services work together correctly.

• User Acceptance Testing (UAT): Verifying with end-users that the software meets their needs.

• Automated vs. Manual Testing: Balancing speed (automation) with the human touch for complex edge cases.

• CI/CD (Continuous Integration/Continuous Deployment): Automating the process of merging code and pushing it to production.

• Infrastructure as Code (IaC): Using scripts to manage and provision server resources.

• Containerization: Packaging software and its dependencies into units like Docker containers for consistency across environments.

• Cloud Orchestration: Managing containers and services at scale using tools like Kubernetes.

• Security Auditing: Identifying vulnerabilities such as SQL injection or Cross-Site Scripting (XSS).

• Data Encryption: Protecting sensitive information both at rest and in transit.

• Compliance: Ensuring the software follows legal standards like GDPR (privacy), HIPAA (healthcare), or PCI-DSS (payments).

• Monitoring & Logging: Tracking system performance and errors in real-time (e.g., using ELK stack or Prometheus).

• Technical Debt Management: Periodically refactoring code to keep the codebase clean and maintainable.

• Patching & Updates: Releasing fixes for bugs and security holes.

• Scaling: Adjusting the system to handle more users or larger datasets.

• Technical Documentation: Explaining the codebase and architecture for other developers.

• User Documentation: Creating manuals, FAQs, or help centers for the end-users.

• API Documentation: Providing clear instructions on how external developers can integrate with your system.

While much of this output isn’t required for every coding project, almost every project requires you to undergo at least half of the processes listed.

To make a fully functioning application just for personal use already requires you to wear a dozen different hats, each of which can take years of study to fit properly. If you wish to publish your application or host multiple apps that interact with each other, prepare to start practicing your balance.

All of this can be overwhelming for one person, which is why software companies have operated with large teams since software engineering became an engineering discipline. While good for the industry at large, I believe this has put off many solo developers and small teams from attempting to make applications at a certain scale. This can function as a form of monopolization, restricting the availability of competitors in the market. I think this technology can even the scales.

Modern Self Sufficiency

This industry has prioritized the ability to move quickly for years. From software development style to company organization structure, tech-focused people have yearned for the ability to make new things quickly. I am no exception, within reason, of course.

Technology like this can enhance the workflow of smaller teams, giving them the bandwidth to compete with much bigger players in this space with far fewer resources. It also gives the big guys the ability to take even bigger shots and stress far less about the ones that miss. I think this is exactly how this technology should be applied.

The bottom line is I’m now able to do more with what I already own. That’s what excites me. I’m able to build more complex ideas, test them without a massive team, and deploy them without being gouged by a monopoly. That’s a world where technology is being used to empower people to create, and that’s exactly where I want to be amongst all the shifting landscapes and monopolistic corruption.

(As with everything, there exists a negative side to the democratization of this technology. But before I research and utilize this technology more, I thought it important to document my excitement and why I felt it.)

The Inciting Incident

On Thursday, February 26th, Anthropic CEO Dario Amodei announced that the AI company declined to cede safeguards to the Pentagon in relation to Anthropic-powered military technology. He said the following in a public statement:

“Anthropic understands that the Department of War, not private companies, makes military decisions. We have never raised objections to particular military operations nor attempted to limit use of our technology in an ad hoc manner.

However, in a narrow set of cases, we believe AI can undermine, rather than defend, democratic values. Some uses are also simply outside the bounds of what today’s technology can safely and reliably do. Two such use cases have never been included in our contracts with the Department of War, and we believe they should not be included now; Mass Survailence [and] Fully Autonomus Weapons.”

“The Department of War has stated they will only contract with AI companies who accede to “any lawful use” and remove safeguards in the cases mentioned above. They have threatened to remove us from their systems if we maintain these safeguards”

“Should the Department choose to offboard Anthropic, we will work to enable a smooth transition to another provider, avoiding any disruption to ongoing military planning, operations, or other critical missions.”

- Dario Amodei, 2026

Because of this misalignment in values, the Pentagon gave Anthropic until 5:01 pm on Friday, February 27th, to cede control or lose its government contract. Anthropic chose the latter.

In a statement responding to Dario’s decision, the Pentagon’s CTO Emil Micheal told CBS News reporters the following:

“At some level, you have to trust your military to do the right thing,”

- Emil Micheals, 2026

Emil Michael claims that the US Military is already prohibited by law from creating mass surveillance technology, let alone using AI to supercharge such inventions. He also claims Dario is actively lying, stating that the Pentagon has made concessions that Anthropic should have accepted.

I want to make my stance clear here; I believe Dario Amodei and Anthropic’s decision to maintain control over their technology is a good one. It takes an undeniable level of courage to stand up to the US Government, especially knowing that doing so will make you a target. This administration seems to be in the business of bullying countries and corporations into compliance, and I’m elated to finally see some resistance from America’s C-Suite.

However, I cannot in good conscience pretend that the arguments against one company enforcing morality don’t have merit. As much as I believe Amodei is ‘doing the right thing’ here, the company’s position could change with a board vote. That volatility worries me.

I want to use this situation as an opportunity to look at the US’s relationship to Machine Learning as a component of our National Defense strategy. This technology, and the people who wield it, have the potential to enforce pre-established international rules, to restructure the world order, or to chase power. That decision is one that we Americans will finance, and it will impact the quality of all of our lives. This is bigger than Anthropic.

Defense, Machine Learning, and Bias

The proposed purpose of organizations like the Department of Homeland Security and the Department of War (fka the Department of Defense) is to ensure the safety and security of America’s land, people, and assets. Technology is critical to achieving this goal, as is any tool used by large groups of people.

Machine Learning technology enables a type of Mass Surveilance and Data Analysis previously unfeasible. Facial recognition technology was science fiction until advancements in Spatial Scanning and Computer Vision made it a reality. Developments in Data Storage, GPU Processing, 3nm Chip Technology, and much more are why worries about mass surveillance are ramping up now.

The US Government has partnered with companies like Palantir, Flock, and Clear Secure to enhance national security technology through Machine Learning technology. Seems well intentioned enough, what’s the problem?

Issues arise when national security interests supersede a personal right to privacy and the protection of user data. Here’s where the philosophy meets technical reality.

Read the article below to gain a deeper understanding of the black box nature of Deep Learning Models.

Basically, AI models recognize patterns. Deep Learning models are large and can process more data and recognize more complex patterns. This is the technology that companies like Palintir leverage for National Security Technology.

Just because these models can identify patterns doesn’t make them capable of ‘understanding’. AI models can still be incorrect. More importantly, objectivity in an AI model is impossible to obtain. Machine learning engineers know this to be true; laypeople don’t. Computers appear unbiased to those who don’t understand that numbers will never convey a full story.

AI models can’t teach themselves yet, so it’s up to us to teach them.

Because AI models cannot understand what they’re ingesting, there are no safeguards to stop models from becoming ‘biased’. What happens when you try to apply mathematical pattern recognition to assess something more sociological?

This crafts a worrying image. What’s to stop a team of people from training a model to support a narrative that may not be true? What stops that group from using said model as proof of their worldview? How many would be convinced, and how far could you go?

White Nationalists have used the guise of ‘objectivity’ and ‘pattern recognition’ to couch their beliefs from criticism for centuries. Thankfully, we’ve had evolutionary biologists, sociologists, and statisticians who could take a quick look at their ‘work’ and immediately identify aspects of bias. With Machine Learning, that kind of assessment isn’t completely possible, let alone accessible. This poses a dangerous new way to implement structural bias.

Learning the Wrong Lessons

What if I told you that an AI model was trained on government data, and that model would have a significant influence on a judge’s decision to sentence you to jail? Knowing everything I detailed about the possibility of injected training bias and the uninterpretability of AI models, would you be comfortable with that?

It’s an unsettling hypothetical, and it should be horrifying to learn that it’s not one.

In 2016, ProPublica published a scathing analysis of the COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) algorithm. This algorithm, adopted amongst several jurisdictions in New York, was used to provide ‘Risk Assessment Scores’ for judges to consider when sentencing a convict. Already seems a bit dystopian, and moreso when you realize the system was not objective by any means. Here’s the report’s findings:

“The score proved remarkably unreliable in forecasting violent crime: Only 20 percent of the people predicted to commit violent crimes actually went on to do so”

“In forecasting who would re-offend, the algorithm made mistakes with black and white defendants at roughly the same rate but in very different ways.

• The formula was particularly likely to falsely flag black defendants as future criminals, wrongly labeling them this way at almost twice the rate as white defendants.

• White defendants were mislabeled as low risk more often than black defendants.

Could this disparity be explained by defendants’ prior crimes or the type of crimes they were arrested for? No. We ran a statistical test that isolated the effect of race from criminal history and recidivism, as well as from defendants’ age and gender. Black defendants were still 77 percent more likely to be pegged as at higher risk of committing a future violent crime and 45 percent more likely to be predicted to commit a future crime of any kind.”

“…judges have cited scores in their sentencing decisions. In August 2013, Judge Scott Horne in La Crosse County, Wisconsin, declared that defendant Eric Loomis had been “identified, through the COMPAS assessment, as an individual who is at high risk to the community.” The judge then imposed a sentence of eight years and six months in prison”.

- Julia Angwin, Jeff Larson, Surya Mattu and Lauren Kirchner, via ProPublica

There are numerous direct examples of the algorithm ranking black people as higher risk than white people for the same crimes, in ways a human would never justify if race had been excluded. Because many of these systems are for-profit and closed source, we’ll never know what these models were trained on to give such skewed, racist results.

My personal nightmare isn’t that things like this happen in the first place; instead, it’s that we constantly repeat these mistakes as a society. It’s that technologies like this are uncritically adopted by bad actors within our government, and bias now gets discussed as ‘statistical analysis’. I need us to acknowledge that technology, algorithms in particular, cannot exist without bias. We can’t offload decisions with lasting impact on people’s lives to a series of unconscious algorithms incapable of understanding the sociological context behind why it ‘thinks’ in certain ways.

Bad actors will forever try to utilize technology to couch their bias behind an objective framework. Racists did so with Eugenics, and nationalists can do so with AI. That being said, it’s important to call out people in power when they try to hide behind technology. There’s certainly no shortage of people in our government trying to do so.

This story isn’t to be scoffed at. It’s a glimpse into a potential future that nationalists like Pete Hegseth, Emil Michael, and others at the DOW would like to make a reality. They want to impose systems that validate their outdated worldview without a second thought. They want the final say in how to use this tech, even though that use is heinous and unethical. They aren’t concerned with societal well-being; They’re worried about power and making sure they have it. All of it.

Imagine a world where the Department of War has access to an advanced version of this algorithm, where that model could use drones, computer vision technology, and databases of public records to generate Risk Assessment profiles of people in foreign nations. Would you trust that model to be accurate in its judgment, knowing what you know now? What about allowing the model to decide the risk profile of a missile striking a particular location, or the risk profile of sending troops to a specific area?

What’s telling is that the above scenario isn’t even what Anthropic is opposed to. So long as their technology isn’t doing the surveillance or the shooting, the rest is fair game. Even that tiny inkling of a moral backbone was enough for the Department of War to spiral. That begs the question: What was the intent behind imposing these rules?

Creating Corporate Distance

While I’m undeniably in support of Anthropic’s choice, I keep finding myself thinking about their only two imposed rules.

“…Some uses are also simply outside the bounds of what today’s technology can safely and reliably do. Two such use cases have never been included in our contracts with the Department of War, and we believe they should not be included now; Mass Survailence [and] Fully Autonomus Weapons.”

- Dario Amodei, 2026

It occurred to me that this company was only disallowing actions that would give their brand a bad look.

Their statement, while admirable, ignores a plethora of other ways government AI infrastructure could be used to harm people domestically and internationally. This looks to me more like a way to create distance between specific instances of harm than anything else. It’s a quiet admission that we’re willing to do your dirt, so long as you pull the trigger. When framed in that way, it comes across less as a moral stand and more like a play to prevent future PR troubles.

If you think about it, it’s genius. Anthropic gets to take a public stand against a largely disliked leader on moral grounds. Is there a better way to win good favor? They can now paint themselves as ‘the ethical AI company’ in a sea of other, less clean alternatives.

The D.O.W. AI Acceleration Plan

In a memo penned to the Department of War leadership, Pete Hegseth outlines the following:

We will re-focus the Chief Digital and AI Office (CDAO) and these enhanced resources to unlock critical foundational enablers needed to accelerate war-winning efforts across the Department, starting with enabling the set of seven PSPs listed below in fiscal year 2026. These PSPs will address key opportunities for enhanced military AI advantage across Warfighting, Intelligence, and Enterprise mission areas:

Warfighting:

1. Swarm Forge: Competitive mechanism to iteratively discover, test, and scale novel ways of fighting with and against AI-enabled capabilities - combining

   America’s elite Warfighting units with elite technology innovators.

2. Agent Network: Unleashing Al agent development and experimentation for Al

   enabled battle management and decision support, from campaign planning to kill chain execution.

3. Ender’s Foundry: Accelerating AI-enabled simulation capabilities - and sim-dev and sim-ops feedback loops - to ensure we stay ahead of AI-enabled

   adversaries.

Intelligence:

4. Open Arsenal: Accelerating the TechINT-to-capability development pipeline, turning intel into weapons in hours not years.

5. Project Grant: Enabling transformation of deterrence from static postures and speculation to dynamic pressure with interpretable results.

Enterprise:

6. GenAI.mil: Democratizing AI experimentation and transformation across the Department by putting America’s world-leading AI models directly in the hands of our three million civilian and military personnel, at all classification levels.

7. Enterprise Agents: Building the playbook for rapid and secure AI agent development and deployment to transform enterprise workflows.

To me, an initiative like Swarm Forge could easily generate another iteration of COMPAS. This time, reporters won’t even have a chance to access the technology.

The Point

I’ve come to the following conclusions:

1. AI Technology can be ideologically weaponized. For-profit companies did the same years ago for the Prison Industrial Complex. Nationalists can do it to justify homeland security decisions.

2. AI Companies are okay with said weaponization, so long as their technology cannot be held liable. They’re still corporations; they only respond to threats against their bottom line.

3. This technology, if used in specific ways, can break our system. Authoritarians have historically used new developments in technology to justify totalitarian rule. I have no doubt Hegseth and Trump will try to play that game sometime in the near future.

Right now, we’re at a critical juncture. There’s still time before we become a technofascist dystopia. So, how do we hold the line?

What Now?

I wish I could say that the road to AI regulation is a simple one. We can’t rely on billionaires and CEO’s to do the right thing consistently, and we can’t even rely on our elected representatives to advocate for us. I think the only things we can do are teach and be vocal.

Knowledge about how AI models work has to become mainstream. People need to understand what this is and how it will affect them. It’s not magic, nor is it sentient. It’s a highly advanced algorithm that can be wrong and biased. Once people understand that, we’ll have a better idea of where we should and shouldn’t be seeing AI.

Vocal discontent is also critical. I don’t mean vocal as in yelling into Twitter, that accomplishes nothing. I mean genuinely expressing dissatisfaction in meaningful ways. Call your senator, engage with local and state leaders near you. Respond to polls. Make sure that elected leaders know you will not respond well to anyone who supports this kind of tech integration. We live in a representative democracy. While far from perfect, there are avenues to make your voice heard.

I wish I could write more in this section, but right now, this is all I’ve got. I’m still trying my best to do my part, and these articles are a part of that. It’s clearer and clearer to me that we’re going to have to rely on each other and our communities to get through strange times like this.

AI technology, to live up to its full potential, must become somewhat transparent in its reasoning. If you’re interested in the full argument I make for why that is, check out the article below.

In short, real, practical integrations of machine learning technology beyond automation require the user to understand why a model made a particular decision. In areas such as disease diagnosis, genomics research, business decision-making, and more, advanced reasoning capabilities and understanding are required to make the large and small-scale decisions that will impact work in the future. If AI apologists want this technology to be authentically integrated into these processes, then it needs to explain itself and its value a bit more.

Large teams of people are working on Explainable AI technology, which can provide deeper insights into the reasoning behind a Deep Learning model’s decisions. This, to me, seems like where the real scientific breakthroughs might lie. That’s why I’ll be exploring a couple of popular forms of Explainable AI technology and the methods these techniques use to accomplish their goal.

Defining Explainability

Explainable AI (XAI) is a class of techniques that provide clarity through enhancing the interpretability of an AI model. Basically, an AI model is ‘Explainable’ if a human being can interpret its reasoning. Sounds deceptively simple, but the techniques required to accomplish this could make any layperson’s head spin.

Explainability can be accomplished in two main ways as of writing. Globally and Locally.

1. By-Design Explainability is when all a model’s logic is structured to be easily understood.

2. Post-Hoc Explainability is an additional technology applied to understand a specific individual prediction(s) made by a model.

By-Design Explainability, while useful for solving simpler problems, is limited by its very concept. Human understanding is limited, as is the mind, so any model acting within the constraints of human traceability and understandability won’t explain complex models.

This leaves us with a variety of Local Explainability techniques, most of which seem to focus on applications in Deep Learning, unsurprisingly. Deep Learning models may be the most complex pieces of programming ever invented, and are completely uninterpretable by human beings without the advent of Post-Hoc Explainability Technology.

Deep Learning technology is being used in medical research worldwide, and interpretability is highly coveted. So, what tools do we have now?

Explainability Technologies

*I am going to dive deep into what appear to be the two biggest and most researched model-agnostic explainability technologies as of today. By no means is this a comprehensive summation of all available research, but I believe this is a good place to start.

Local Interpretable Model-agnostic Explanations (LIME)

LIME adds complex model predictions in a relatable manner so experts can interpret the in-depth details of disorders such as dementia.

More specifically, LIME creates a surrogate interpretable model based on a particular ‘Black Box’ model. Then the surrogate model approximates the decisions of the ‘Black Box’ model and provides the user with a rationale for doing so.

“Your goal is to understand why the machine learning model made a certain prediction. LIME tests what happens to the predictions when you give variations of your data into the machine learning model. LIME generates a new dataset consisting of perturbed samples and the corresponding predictions of the black box model. On this new dataset, LIME then trains an interpretable model, which is weighted by the proximity of the sampled instances to the instance of interest. The interpretable model can be anything from Lasso to a decision tree. The learned model should be a good approximation of the machine learning model predictions locally, but it does not have to be a good global approximation”

The above quote is from Christopher Molnar, an Interpretable Machine Learning Researcher whom I suggest checking out to gain deeper insight into the topic. Read More Here.

The beauty in the approach here is that it’s almost entirely based on a simple computer science principle. Decomposition. If a problem is too complex, one of the first things your programming professor will tell you to do is to break it down. Simplifying is one of the most effective tools in the arsenal of understanding, and that’s exactly what the creators of LIME exploited in relation to Deep Learning Models.

If the goal is to understand why a decision was made, you can’t ask the black box. However, simulating the decision itself in a white box might help you understand the individual decision. It feels technically significant to realize that understanding can come from analyzing the rationale of a decision itself, rather than forcing an analysis of an unreliable decision-maker. The simulation doesn’t even have to do a good job of approximating the whole model, so long as it can emulate a single decision.

LIME is also Model-agnostic, meaning that it can be applied to any machine learning model with no work required to modify the model from within. This grants this technique a universality that makes it attractive to many.

The goal of LIME is to understand why a model made a decision, not what a model considers when making decisions. Understanding this helps us apply this technology correctly, as it’s not capable of understanding the entirety of a model’s ‘thought process’. LIME is excellent at isolating decisions and replicating them in more ‘transparent’ models, helping us gain a deeper understanding of the decisions individually.

SHapley Additive exPlanations (SHAP)

SHAP analyzes the model’s key features, which contribute to model performance and decision-making.

“When a machine learning model makes a prediction, one of the most fundamental questions we can ask is also one of the most difficult to answer: How much does each feature contribute to this specific prediction? This question matters profoundly, whether we’re debugging a model that makes unexpected decisions, explaining predictions to stakeholders, or ensuring regulatory compliance in high-stakes applications. Yet the answer is far from straightforward, because a feature’s contribution depends critically on what other features are present, creating a complex attribution challenge that simple difference-based methods cannot fairly resolve”

The above quote is from Michael Brenndoerfer, a Data Scientist and Entrepreneur who has some of the most beginner-friendly and insightful technical writing on explainability that I have seen on the internet. I’d also recommend that everyone read his work. Read More Here

While LIME focuses heavily on reproducing the decision-making process, SHAP takes a more analytical approach. SHAP draws on Game Theory’s Shapley Values to create a mathematical framework for feature attribution. This framework applies seamlessly to every machine learning model, giving SHAP universality in the same vein as LIME.

Shapley Values are, in simple terms, mathematical values that analyze the contribution of particular features to an outcome. In Game theory, they are used to predict the outcome of a game. In machine learning, the outcome is known while the features remain a mystery. Game Theory prescribes a formula to understand the features that contribute to any outcome. In other words, Shapley Values give mathematical validity to the explanations provided by SHAP.

SHAP can also explain local decisions and the features that a model finds significant globally, making it both locally and globally explainable.

The goal of SHAP is to understand exactly what a model deems as important. Using a well-established mathematical framework, SHAP provides a level of rigidity and justification in its explanations unrivaled to this day. SHAP is excellent at giving us concrete ideas about how a model justifies its decisions.

Conclusion

While the Black Box remains black, we have a plethora of technologies that allow us to interrogate its rationale. This, while still not a glass box, is radically more transparent than I would’ve thought before researching this.

It’s important to emphasize that none of these technologies has reached maturity. These techniques were invented in 2016 and 2017, respectively, so we may be in for much more growth in their development. Isn’t that exciting?!

I genuinely believe that the development of technology like this is what will lead our society to the better world we deserve, so long as we don’t allow tech like this to be monopolized, gatekept, and sold to the highest bidder. Open Weight/Source tech should be prioritized, and I hope that research sharing will continue.

I’m not very into sports, so naturally my affection for football is slim to none. Most of my interest in this year’s Super Bowl came from Bad Bunny’s Halftime Show, which did not disappoint. I’m not too fond of ads, so I didn’t stay for long after halftime. Little did I know I’d miss what could very well be the most dystopian ad for a consumer technology product ever created.

For those unaware, Amazon’s home security company Ring launched an ad for its new ‘Search Party’ feature. This feature, planned to be added to your Ring Cameras later this year, utilizes Artificial Intelligence to pool together footage from other Ring cameras from across your neighborhood. What’s the use case? They claim it’s exclusively for finding your lost pets. If Chip goes missing, you can hopefully use the ‘Search Party’ feature to track exactly where he went. I think you can see where this is going.

I saw my corner of the internet erupt in criticism, and people were quick to call this a dystopian nightmare of an idea. I’d be lying if I said I couldn’t understand why. If you replace ‘dog’ with literally almost anything else, our reality would come uncomfortably close to literal works of horror fiction.

While I’ve seen a lot of conversation around this topic, I haven’t really seen much productive conversation surrounding the nature of privacy, especially in modern society. If my neighbor has a Ring camera, I now exist within Amazon’s database against my will. Their AI can now analyze me just as easily as it can a dog. What am I to make of that? More importantly, what can we do?

Cameras are Tools

Recording ourselves is commonplace now. It has become socially acceptable to record yourself and others on the train, at school, and even at work. The invention and popularization of the ‘Influencer’ shows a societal shift in how we think about the camera. Above all else, a camera is a tool. That tool was once specialized, requiring high amounts of skill to wield. Nowadays, that tool has been democratized and streamlined in production and accessibility. More people have access to this tool than ever before, and the use cases for this tool are ever-expanding.

One particularly great use case for this tool is for documenting change. Visual documentation can convey more detail and information, which solves hundreds of problems for people, businesses, and even governments. Recordings have proved people innocent and guilty of crimes and observed changes in natural habitats we would have never seen otherwise. Visual documentation has allowed humanity to expand past numerous barriers not possible otherwise, and I don’t intend to argue the contrary.

What I do intend to emphasize is the social and economic ramifications surrounding the normalization of cameras in public spaces. I don’t just mean the negative ones, as the ability to record a crime happening on a city street is a great way to have that crime documented and resolved legally. Taking a picture of your child on a swing at a public park can bring years of joy to you and your family. Documenting officers of the law acting in unjust ways is a fantastic way to dispel false narratives crafted by high-ranking manipulators. In both a mundane and revolutionary sense, documenting change has had and will always have positive effects.

However, there are negatives. As a young person who grew up alongside the advancement of smartphones, I’ve seen people’s ideas change in real time about what is and isn’t appropriate to record. Nurses have leaked patients’ health information by being careless on TikTok. Youtubers have doxxed themselves on their own videos. Grok can generate photographs of anyone altered to make them unclothed. Yes, anyone. These and more are a result of a change in what is not acceptable to document and alter, a newfound comfort level around and in front of a camera. Using a tool without intention and respect leads to injury. Not always, but frequently enough.

Tools for Surveillance

Regarding personal property, I think it’s natural to do all you can to protect what you own. Apple Airtags and Tile trackers are both tools for surveillance, but I believe most people understand the difference between these tools and cameras. The prior are incapable of violating our individual privacy. Cameras can, and that can be an uncomfortable feeling to sit with.

When I see people talk about the negatives of surveillance technology, the invasion of privacy is often the main point of contention. The idea that somebody else can give sensitive information regarding our personal lives away is uncomfortable, but it’s a reality that we’re being conditioned to accept. Over the last 20 years, technology companies have started collecting more and more intimate details about our personal lives for the purpose of advertising, which most people recognize as a less noble use of surveillance tech than stopping crime or finding lost pets.

Because of that recent shift in data collection, I believe society has become much more accepting of surveillance that happens ‘behind the curtain’. We can’t see the cookies on a website tracking us across the web, nor can we see the Meta databases that hold our information. I believe this is a major factor in people’s comfortability with surveillance. In general, we as Americans are okay with tracking under the following circumstances: The tracking is done in exchange for a service, and said tracking is hidden from us.

The Machine Learning of it all

Search Party uses Computer Vision technology to accomplish its goal.

To the uninitiated, Computer Vision is a subcategory of machine learning (or AI) that specializes in training a model to recognize aspects of an image. This is a broad definition, but what you need to know is that this tech allows Amazon to identify your dog in an image. This alone is pretty cool.

What isn’t cool is the implementation. Alongside this technology, Search Party requires that Amazon access every Ring camera within an undefined radius and transfer image data from those cameras to a server. It must do this so the model can analyze the data, since cameras alone cannot run AI models. It then has to identify which camera saw the dog, and extract date, time, and video data from that camera to send to the initiator.

To be as clear as possible, the AI technology is not responsible for the breach of privacy, nor does it even require it. Amazon is a near Trillion Dollar Company. They can afford to put processors in every camera to locally run this model, and they could provide the user with the choice to only send information they’re comfortable with to neighbors through non-integrated services like NextDoor. They could’ve made pro-consumer choices, but they chose not to. AI is not responsible for the breach of privacy; Amazon is.

There are compelling arguments to suggest that AI technology wouldn’t even exist if not for previous breaches of privacy by tech and research institutions, and I would be inclined to agree. But in this specific case, Ring chose to create a horrifying product which unintentionally displayed to 135million americans the true intentions of these corporations. Not to protect your pets, but only to do so under the guise of complete and total control over your information.

Big Brother

In favor of speaking on the sociological and technical details, I’ve postponed stating the obvious. The very existence and showcasing of this technology should make everyone on earth uncomfortable, because this won’t be confined to dogs. To be more accurate, it can’t be. Computer Vision models could be trained on any form of visual media, including videos of you. This is the dystopian part.

Ring has partnerships with companies that work directly with the federal government. There have been several recorded instances in which Ring has handed over video and photographic data to Law Enforcement entities that did not present warrants. Especially under this administration, it would be silly to discount the potential use of AI tracking technology as a weapon of the state.

Now consider that it was your data, potentially your doorbell footage, used to develop this technology. You weren’t made aware of this, and you damn sure weren’t compensated. These people buried clauses in a thousand-page ‘Terms and Conditions’ document, allowing them full access to your personal information, with no prompt to opt out. Most people arent’t even aware of the data collection happening in the background.

Does a company that takes advantage of uninformed consumers sound like a company that will handle your data with care and tact? How comfortable are you with letting a highly advanced advertising firm see and process data that comes from your home?

The Important Part

There are many actions you can take to disincentivize the development of this type of surveillance technology. If you’re uncomfortable, I’d encourage you to take action rather than falling into nihilism as a coping mechanism. It can be easy to think change isn’t possible, but that mindset lets these companies win.

1. Be Vocal with your Wallet

   Consumer tech companies need our money to be successful. Enough vocal pushback and cancelled subscriptions in response to anti-consumer decisions are often enough to get companies to reverse negative decisions, often for fear of a declining stock price. Because of the disgusted reaction to Search Party, Ring has suspended its previous integrations with Flock, a surveillance company known for its solar-powered AI cameras and strong ties to the US Government. Companies respond to negative feedback when it starts to affect their bottom line.

2. Anonymize your data

   Many of the services that we use contain opt-out options when it comes to the collection of our personal data. They often bury it in many pages of settings, but all it takes is one query to Google to see exactly how to turn off or anonymize tracking.

3. Become Technically Literate

   These companies mostly take advantage of consumers who don’t know that they have other options available to them. The best way to beat them is to inform yourself. Learn about how to activate a VPN on your device, research privacy-centered browsers and email clients, and even learn to vibecode your own security camera system and host the data yourself. With generative AI, all of this is easier to learn than ever before.

Remember that complaining without personal action accomplishes nothing. We can all dictate our own reality with collective action. I was proud to see so many people deactivate their Ring cameras as a response to this new feature, and I hope this kind of vocal resistance can continue as these companies get more overt in their implementations of this technology.

I’ve spent the last few weeks really evaluating the utility of current AI technology. For a while, I’ve felt as if there has been this massive leap in machine learning capability, yet no real significant breakthrough in what we should be using that power for beyond process automation. In thinking about what that breakthrough might be, I kept circling back to one key industry. Healthcare.

The scientific community at large has been interested for decades in the possibility of detecting diseases before they present themselves. For most of human history, we have only been able to diagnose diseases through external symptoms; this is steadily changing. Since the late 18th century, we’ve been able to use advanced technology to identify cellular and genealogical differences between the sick and the well. This is how we obtained a greater understanding of how to prevent diseases and asses risk of infection, among many other things.

For the last half-century, we have been collecting petabytes of data regarding individual diseases, including MRI scans, blood pressure tracking, and even autopsy reports. It would be impossible for one teamof humans to parse all this data, even thousands of the world’s most qualified scientists have struggled with a modicum of that volume of information. So imagine what we could accomplish if we could effortlessly parse terabytes in minutes?

Dozens of tech billionaires are confident that the time has come. Sam Altman, CEO of OpenAI, would go as far as to say in reference to cancer, “If AI stays on the trajectory that we think it will, then amazing things will be possible. Maybe with 10 gigawatts of compute, AI can figure out how to cure cancer”. The ‘maybe’ is doing a lot of heavy lifting there. His take only considers technical factors, such as compute power, data-center support, financial backing, etc. It’s devoid of the more ‘human’ variables, one of which is the ability to understand and explain the rationale behind certain decisions. Here, AI tech is far behind the curve.

AI and ‘Understanding’

I think it’s important to characterize AI models correctly, as I don’t believe many people accurately understand what a Machine Learning model is capable of. An AI model can ‘think’ in a very rudimentary sense. Model size determines the complexity of the ‘thinking’ much of the time, but definitionally, AI models are able to recognize patterns.

Deep Learning models are larger, more complex webs of nodes that can process more data and recognize more complex patterns than standard models. This is the technology that empowers researchers to create AI models that can identify Alzheimer’s disease, various cancers, etc.

Now, just because these models can identify such patterns does not mean that these models are capable of ‘understanding’ the information that has been fed to them. These models are given rounds of ‘training data’, which show the model a set of information to analyze for patterns. After the model is trained, it cannot expand its ‘knowledge’ until it is trained again. This opens the possibility for the model to respond unintelligibly if asked to process information it hasn’t seen before.

Even without that possibility, AI models can still be incorrect. Just because a computer processes information differently than us does not mean it’s incapable of error. I feel like I see contradictory rhetoric too often online, so I feel the need to stress the following: Unless these models are used exclusively within the context it has been trained for and are heavily refined, preferably by an expert, these models can be frequently wrong.

Accuracy rates of 90% or above are difficult to come by, often only achieved by models with extremely simple or niche applications or by teams of experts tirelessly training models for months. Even if high accuracy is achieved, one of the biggest problems with this technology still prevails.

Deep learning models alone are uninterpretable by human beings. The technology behind these models’ reasoning is often too complex to track, meaning that you’ll have no idea why the model made the decision that it made. To you, you just gave an input, and then an output appears out of thin air.

For better and for worse, Deep Learning Models are a Black Box.

Trust the Black Box

If I asked you to place your hand into a box you can’t see into, would you do it? Would you hesitate? How much trust can you have in something you can’t know the contents of?

Better scenario. What if I told you that I hand-picked a variety of medical documents related to BMI and overall health and placed them into a magic box that would summarize them all and give you medical advice based on your information. How much do you trust that box? You know the box is an inanimate object, that it cannot understand the true meaning of its inputs beyond the syntax and order of the numbers. You’ll never know how the box makes its decisions, you don’t even know what information the box was fed!

To be clear, this is not a new phenomenon. The above scenario is just a tech-infused version of the reality that many of us already face. Black and brown people within the United States and across the world have been subject to misdiagnosis and poor medical treatment due to poor representation of us in research and academic literature. To this day, the Body Mass Index does not accurately account for the average differences in muscle and bone density within people of African descent. I bring this up to say that this issue is not only important now, but that it has always been of importance.

The existential problem here is that you cannot reason with an AI model. You cannot convince it of anything, as it is only capable of ‘understanding’ within the bounds of its training data. I believe that alone makes this technology a threat to everyone if used without caveats. Expert or layman, the Black Box is not to be uncritically trusted.

Experts and Technology

In any industry, but particularly in scientific fields, experts are encouraged to dive headfirst into research to gain a deep understanding of a topic before they are given the authority to form any conclusions about a topic. This means those experts tend to gravitate towards tools that provide a level of explainability to the conclusions formed.

To a breast cancer researcher, a model that can predict breast cancer is nothing more than an automation tool without the ability for that model to say exactly why it made that prediction. I say that not to demean automation tools, as they absolutely have their place. What I’m trying to point out is that the breakthrough discoveries that tech billionaires are touting will come from this underlying technology, won’t come from the underlying technology. It will come from the many teams of people fervently trying to make the black box into a glass one.

The way that researchers are doing so is deceptively simple. We just need acetone.

Stripping the Paint

Even though the nature of Deep Learning models is to be closed, ML Researchers from across the world have helped develop early technologies to make prediction models more opaque in their reasoning.

There is an entire subsection of the tech industry focused on developing Explainable AI (XAI in short), which is a set of processes and methods to enhance the explainability of Machine Learning models. There exist frameworks and tools like Local Interpretable Model-agnostic Explanations (LIME) and SHapley Additive exPlanations (SHAP), both of which have been featured in several research papers and been deemed promising by the scientific community. While this technology is impressive, it’s far from mass adoption and will require significant funding, research, and time until it’s seen in clinical settings.

The truth is, there’s a long way to go until the black box is fully glass, but we’re on the way. In the meantime, it might be good to reconsider how we think about and use AI technology in our daily lives. I think the public should be informed about what will actually get us to the future we want, instead of being led into trusting the men who have every monetary incentive to lie to our faces.

Please, until the box is glass, don’t put your hand in it.