We are in the middle of the generative artificial intelligence (genAI) hype cycle. As soon as ChatGPT 3.5 was released in late 2022, it caught the imagination of the general public—and the international development community. It remains to be seen if genAI will reshape the world as much as some proponents think, but DAI is not waiting to find out. In late 2023, we embarked on an ambitious pilot project to explore if—and how—genAI can safely enhance the political economy analysis method across two of our donor-funded governance projects, one in Colombia and one in Georgia.

What is a Political Economy Analysis?

According to the U.S. Agency for International Development (USAID), a political economy analysis is “a field-based methodology that can improve the effectiveness of international development assistance by helping development practitioners to focus on not only how things happen but why things happen… and what realistic opportunities are there to change incentives and to effect change.”

Conducting a political economy analysis is about understanding the underlying dynamics and interests that can hinder or accelerate reform and shaping projects to work with those interests rather than against them. These analyses—especially those in the democracy, human rights, and governance sector—may be exhaustive and wide-ranging. Project teams often conduct them at the beginning of a project, then update them on a regular basis (sometimes even on a daily basis in the case of the so-called “everyday political analysis”). Because they involve a combination of desk research, such as reading local newspaper articles, and extensive stakeholder engagement, such as formal interviews with government counterparts and informal conversations with community members, they can be extremely time-consuming—even though they produce rich insights that can reshape the overall direction of the activity and specific interventions. This presents an opportunity for genAI to streamline the process without sacrificing depth.

What Did Our genAI Solution Look lLike?

We partnered with a U.S.-based small business Exchange Design on this pilot project, using its custom genAI application that uses multiple models hosted on Microsoft Azure, including (but not limited to) OpenAI’s GPT-4. This gave us the benefit of ChatGPT’s advanced ability to analyze information and respond in plain language to specific queries. Exchange Design’s genAI tool allows users to create their own closed datasets, rather than pulling from ChatGPT’s own publicly available large language models (LLMs)—which are deeply problematic from a data privacy and copyright perspective. This allowed us to create a retrieval augmented system curated with two separate datasets (one for Colombia and one for Georgia) that combined publicly available information about USAID’s approach to political economy analysis, project-specific information such as reports on security incidents, previous analyses, project quarterly reports, news from local media in the program’s target municipalities, learning exercises conducted at the sub-national level, municipal government programs, mayoral performance reviews, and notes on political profiles of local leaders. The use of Azure-hosted LLMs, combined with our own data, added a layer of data protection and privacy to the system.

Other key advantages of this genAI tool include:

• Easy-to-use chatbot functionality: This tool uses a similar interface to ChatGPT, where the user types in a question and receives near-instant responses in the same chat window. Screenshot: The genAI tool.
• It reduces hallucinations: Exchange Design’s tool informs users when it cannot answer a question, unlike many publicly available genAI tools using LLMs.
• It can attribute sources: For every answer that this genAI tool provides, it can also tell you from which sources in the dataset it gathered this information, which allows users to easily see where its answers are coming from. This functionality is especially important for political economy analyses, where each source may have a different perspective or interpretation of key events.
• It works in multiple languages, including those with non-Latin alphabets: It can analyze datasets and respond to queries in multiple languages, including Spanish and Georgian, which was critical to the success of this pilot.

How Did the Pilot Project Go?

As with almost everything in tech, our team saw mixed results. The genAI tool was a big success with the Colombia program whose monitoring and evaluation team used it often and has incorporated it into their day-to-day work of preparing political economy analysis updates. However, the Georgia team did not move beyond initial testing. These differing experiences have taught us several lessons about using genAI for political economy analysis:

• GenAI for political economy analysis is more useful in a stable context than in a rapidly changing political environment. Our generative AI pilot ran during the first half of 2024, coinciding with the political upheaval and mass protests in Georgia around the proposed ‘foreign agent’ law. GenAI outputs are only as good as the dataset they pull from; Georgia’s political situation was evolving so quickly that their dataset would have required near-daily updates to produce high-quality outputs that could inform a political economy analysis. On a tactical level, the Georgia team also had little time to try out a new digital tool when the political context in which they worked was shifting daily. By contrast, Colombia’s relatively stable political environment among our project’s counterparts in early 2024, lent itself to a PEA pilot. Because they were not dealing with a political crisis, our staff members in Colombia also had more time to experiment with and learn how to use this GenAI tool.
• GenAI works better in widely spoken languages. Even though genAI can theoretically work in any language, it tends to work better in more languages that use Latin characters, because their training sets are typically larger and more robust. (This reflects the problematic predominance of English language content on the internet, which deserves its own blog post). The practical effect on our pilot was that the GenAI tool worked almost perfectly in Spanish, one of the most widely spoken languages in the world with nearly 500 million native speakers, but had limited utility in Georgian, a language with approximately four million speakers that does not use Latin script. The Georgia team was disincentivized from using the genAI tool because it did not work particularly well in the language that they needed.
• Using genAI for political economy analysis may lend itself better to granular analyses for individual regions or municipalities. While the Georgia team operates at a national level from a single location in Tbilisi, the Colombia team operates across and has staff in 18 municipalities. The Colombia team used the genAI tool to conduct a situational analysis of its target regions, reviewing changes in the socioeconomic and political landscape as well as risks to the project implementation. This included assessments of challenges and implementation obstacles, as well as security issues and potential future risks, in specific target municipalities. In other words, this capability gave Colombia staff members another way to identify shifts and trends in Colombia’s political economy at a highly localized level, beyond analog analysis and informal conversations with staff members in other physical locations. Conducting location-specific political economy analyses was less important to the Georgia team, which focused on national-level interventions.

Image created by ChatGPT.

Moving Forward

Our pilot taught us several important lessons, but there is still room to explore the broad applications of genAI. Here are some of the ways AI could enhance the way we conduct political economy analyses:

• Automating desk research. Any political economy analysis involves a significant amount of research and can use multiple genAI applications that social scientists are already using. GenAI can automate desk research and significantly reduce the time needed for document review and data extraction. By quickly analyzing news articles, reports, and interview transcripts, AI tools can summarize trends and key themes, allowing practitioners to focus on analysis, rather than manual research. Our Colombia team was able to optimize their desk research with their genAI tool, which helped them summarize and analyze a wide range of local news sources to inform the regional context, including risks that could impact program implementation. During the program’s closeout, program staff are using the genAI tool to extract information about program achievements, with a focus on key results.
• Streamlining stakeholder mapping. Once the primary data on program stakeholders is fed into a genAI tool, it can rapidly process information on power dynamics, influence networks, and social relationships, helping to map out key actors more efficiently. AI can also highlight hidden connections between stakeholders that human researchers might miss.
• Simulating scenarios and predicting trends. GenAI could be used to simulate political and economic scenarios, providing predictions based on historical data and current trends. This could help development practitioners anticipate shifts in power structures, policy changes, or economic shocks.
• Real-time monitoring and analysis. GenAI could be programmed to continuously monitor political and economic indicators, producing real-time updates for political economy analyses and thus contributing to everyday political analysis. This would allow project teams to adapt their strategies to changing contexts, a key advantage in volatile environments.
• Uncovering informal ‘rules of the game.’ GenAI has the potential to uncover hidden patterns that might elude human analysis, but its effectiveness is fundamentally tied to the quality and scope of the datasets it relies on and to the specific prompts it is given. In the context of political economy analysis—which often focuses on informal ‘rules of the game,’ such as personal influence, corruption, and vested interests—this presents a unique challenge. To fully leverage GenAI in political economy analysis, we must find ways to assemble richer, more structured datasets and draft highly specific, targeted prompts, enabling more precise and impactful responses to inform our political economy analyses.

One other key consideration is the cost of genAI tools. Some widely available genAI tools, like ChatGPT, are currently free. However, these types of genAI systems have demonstrated data privacy risks that make them ill-suited for the sensitive information contained in political economy analyses. GenAI tools that come at a cost, like the one we used for this pilot project, safeguard data privacy (because their datasets will not become part of any larger, publicly available LLMs). While genAI’s efficiency gains—in theory—outweigh the cost of any such system, it does not negate the upfront costs of a more secure genAI tool. Therefore, it is critical for donor-funded programs that want to use genAI for political economy analysis to factor this cost into their budgets. These concerns also raise an important question about the viability of our partner institutions in the Global South using genAI in their work and whether genAI will further widen the gap between development partners in the Global North and Global South.

Whether or not generative AI reshapes the world as we know it, this pilot project taught us that—like everything else in the international development sector—the success of genAI tools for political economy analysis is highly context-specific. What works well for one project in one part of the world, in one political context, may not work at all for a different project in an entirely different part of the world and entirely different political context. Our projects will take into account these lessons learned when evaluating the need and feasibility for using genAI tools in their political economy analyses. We will also continue thoughtfully experimenting with new genAI use cases in our sector as they emerge, prioritizing user safety and community safety at every turn.