Our discussion of How to Assess Artificial Intelligence Impacts in International Development? at the recent Technology Salon DC had its very premise questioned at the start.
We should not be asking how we can assess AI when international development organizations decide to adopt the new technology. We must start assessing AI now. Why?
Because we are all already using AI in our personal and professional lives.
We engage with AI and train everything from neural networks to large language models with every email, upvote, and click online. Our firms already use AI to identify needs, deploy resources, and track outcomes. AI will never be used less than today, and it will be everywhere tomorrow.
That was the core message from this Technology Salon: The AI genie is out of the bottle, and it’s never going back. It’s time we think through what we mean when we ask ourselves and others to use AI responsibility, in ways that are trustworthy.
Is AI Inherently Different From Other Tech?
We already have a large body of standards for technology adoption by the government, civil society, and the private sector. There are ISO, IEEE, NIST, and many more that look at the maturity of the technology and the organizations who want to use it, and guide usage.
Is artificial intelligence any different?
Those that might answer “yes” need to acknowledge that we’ve been working on AI since the 1950s, if not earlier, and there are many things that could be seen as human augmentation for greater insights that go back even further. For example, eyeglasses are human augmentation that improved reading starting in the 10th Century AD.
However, the concept of generative AI, which doesn’t always replicate the same answer each time, can cause problems when we try to consistently evaluate it. For example, we can calculate with extreme precision the outcome of most technologies (say, electricity or lasers), but GenAI often answers differently when asked the same question. Semi- and fully autonomous technologies do the same, and we are struggling with this entire class of solutions.
How to Assess AI Now?
There are many ways to test, evaluate, verify and validate the many forms of AI. The usage of these four terms may be new to international development practitioners since they seem to come from the defense world.
- Test requires a system to perform its stated activities and produce evaluative data
- Evaluate answers the question of, “Did the solution perform as expected?”
- Verification confirms that a technology meets its system-level specifications.
- Validation answers the question of, “Is it the right solution to the problem?”
We can use these four ideas when assessing AI solutions for international development programs. We also try approaches like Black Mirror exercises to envision what can go wrong. Regardless, we must always remember that humans should “be in the loop” on all decisions, and that humans need to be ultimately responsible for any AI decision.
AI Failures Are Not Shared
One issue with assessing AI is our lack of understanding around its impact when failures inevitably happen.
When new technologies inherently fail, there is a culture of silence across the Federal government – from defense to diplomacy, and certainly in development. Failure is not just the F-word in international development. That is why no one likes to talk about failure with AI. Like with other technology, admitting failure brings on many fears, including:
- Fear a loss of prestige – they will be known as failures.
- Fear the loss of funding – why give money to failures?
- Fear the loss of collaborators – why work with failures?
Hence, we are ignorant of what doesn’t work, and destined to repeat the failures of others in our ignorance. When can we celebrate AI failure like we did ICRC’s cybersecurity failure?
A Call for Enlightened AI Policy
I was pleasantly surprised by the many calls for artificial intelligence policies at the Salon. Several participants wanted the US government to regulate AI usage across civil society, the private sector, and of course, the government itself.
Policies are decisive tools, since they are often formulated by public sector politicians subject to regulatory capture from high pressure industry lobbyists. Yet I personally believe that we need government intervention, even if it’s Executive Orders on Trustworthy AI that only apply to the government.
I do wish for constructive and positive Congressional action in the USA that pushes us, as a nation, towards Responsive AI practices, like those in other countries. A person can dream for Section 230 reform that extends to artificial intelligence solutions.