One of the sad truths that emerged at the Technology Salon on ICTs and M&E was that failure in development is rarely about the project performance, but about winning the next contract. This means that monitoring and evaluation is less about tracking and improving progress towards social change and more about weaving an advertising pitch.

This is not for a lack of frameworks, tools, mapping measurements against a theory of change, or even the need for more real-time data in development. It is about incentives. What is incentivized at the macro level is getting big numbers on the board and nice clean upwardly-trending graph lines. Micro-level incentives for filing reports to fill out the monitoring side of things focus on report filing as a requirement for salary payments or other basic carrot/stick-driven models. Neither of these actually encourage accurate, honest data, yet only with that accurate data can we remotely hope to tweak models and make improvements.

So, let's break monitoring apart from evaluation.

Monitoring can be real time and deeply embedded into the activities of a project, reducing the need to waste program staff time on reporting (and removing the need to figure out incentive programs). Any project with an ICT4D component should be light years ahead on this, building in complex logging to their work as a default. These logs should themselves be as open as is possible, but at least to the funding and or parent organizations and/or relevant government agencies. Remove the fudging of numbers and reduce the reporting time from weeks or months to as often as there is Internet connectivity (which, admittedly, still might be weeks or months in some situations).

More complex monitoring situations may require additional work outside of logging - qualitative interviews, metrics that don't pass through the technology components of the systems, and so on. But I would argue that the body of data that does or could be tracked alone would provide powerful proxy indicators of usage, impact, trends and anomalies. Projects like Instedd and the UN Global Pulse - even Google's Flu Trends find ways to take raw data and compile them into actionable knowledge.

Evaluation then becomes two different things. Part of evaluation is a constant, ongoing process -- not something tacked on at the end. Constant attention to the real-time monitoring data, allowing some ongoing adjustments to test methods to improve the project - which is incentivized itself by the ongoing monitoring being more visible.

The wholistic evaluation of the project is no longer something that is a last-minute task to frame the project in the best light. Rather, it is a synthesis of the trends, adjustments, and real-time evaluations that have already taken place. It becomes a document discussing the learnings from the project, and can celebrate both failures and successes together, and it frees the document from being an endless set of tables to being able to highlight qualitative impact stories. Evaluation reports might actually be read.

All of this, of course, should be as open as responsibly possible. Obviously the monitoring data may need extensive cleansing for privacy, but imagine if as a sector, development could learn from itself in a rapid, evolutionary process instead of in slow arduous cycles of every organization learning what works in the current trendy topics on their own.

So, how do we start breaking this apart?

This post was originally published as Monitoring and Evaluation is broken. Let's really break it.

Last month, the M&E for ICT4D Technology Salon noted that we lacked quality tools to measure outcomes. We all intuitively know that ICT could be the basis for great M&E tools, but what about taking that feeling into action?

The Development Gateway develops web-based platforms that make aid and development efforts more effective. After introducing the AidData initiative and the Aid Management Program that provide access to development finance and project information, they are now looking at what ICT can offer in the area of monitoring and evaluation.

In this month's Technology Salon, we will hear from Stephen Davenport on his efforts with the Development Gateway to develop a M&E tool and data collection system that could be used by donors, the NGO community, and even the Government of Malawi, as an additional module to programs like Aid Data or as a stand-alone product.

Join Steve and your technology and development colleagues at the Development Gateway offices for this deep dive into ICT tools for M&E:

Effective ICT tools for quality M&E
August Technology Salon
8:30 a.m. – 10:00 a.m.
Tuesday, August 9, 2011
Development Gateway
1889 F Street, NW, 2nd Floor<-- note address
Washington, DC 20006 (map)

We'll have hot coffee and Krispe Kreme donuts for a morning rush, but seating is limited and the Development Gateway is in a secure building. So RSVP ASAP to be confirmed for attendance or you are on the waitlist.

No one ever fails in ICT4D. Isn't that amazing! Technologies come and go quickly - bye, bye PDA's, Windows Vista, and soon Nokia - yet in ICT4D, each project has impact and we never fail. We just have lessons learned. In fact, can you name a single technology program that has publicly stated that it failed?

This is Oscar Night Syndrome, the need to always look good, and ICT4D is deep in denial with it. At the Best Practices in Measurement and Evaluation Technology Salon we dove into the need for monitoring and evaluation in ICT4D and the tools that can help us do that. What did we find?

ICT4D does not have an M&E culture

Now ICT projects do not exist in a vacuum. Many funders have indicators they expect a project to impact, and they often require some level of M&E. But often this evaluation is an after thought at best, where inputs (number of trainings) and outputs (number of trained people) are counted but there isn't any qualitative analysis (how did the attendees mindset change after the training).

Add to this the need to show results to the donor, their minimum tolerance for failure or anything else that could be seen as waste, and the current climate of "accountability" in political circles, and be it the foolish organization that doesn't turn in a shiny result complete with great storylines and images.

Just think about all the lessons (re) learned in every project, listed deep in a report, while the picture of a woman smiling with a mobile phone is on the cover and everything is rosy in the press release.

Help us improve: take the Technology Salon Survey - you could win an Android smartphone!

How can we change that?

Our great focus at the Salon was how to change the current M&E climate in ICT4D. How to better monitor, measure, and evaluate the projects we work on to improve our outcomes and our profession. We identified four areas where could improve M&E in ICT4D.

Creating Space for Failure

While these are four tools we can use to build an M&E culture, we must change the mindset of ICT4D practitioners if we expect any of these tools to really be used. One way to do that is to have regular meetings where we can talk about what works and doesn't - which is the Technology Salon. Another way is a Fail Faire - a positive celebration of failure.

So coming this fall will be a second Fail Faire in Washington DC, building on last year's event and other internal Faires. If you wanna be one of the cool kids who helps organize it, be sure to email me today!

Together we can change this Oscar Night Syndrome and create a real monitoring and evaluation culture in the information and communication technologies for development community.

Everyone talks about measurement and evaluation (M&E) like it matters. Yet, few of us do it well or even at all with ICT projects. So why should we measure and evaluate? How can we go about it? And what are the industry best practices, applied to the uniqueness of ICT?

Our goal is to explore research design and data collection methods that are applicable to ICT within practical field realities like time, money, politics, and ethics, and emerge with efficient M&E tools and processes we can use in ICT proposals and projects. To get us there, we'll have two esteemed experts leading us in a discussion around M&E and ICT in our next Technology Salon:

1. Raul Roman is a specialist in research methods and evaluation approaches in developing country contexts, an Adjunct Professor of International Development at SAIS-Johns Hopkins, and co-founder of UBELONG
2. Neeran Saraf consults on impact and monitoring and evaluation systems and knowledge management for the likes of the World Bank and a number of international NGOs.

Please join Raul and Neeran in a vibrant conversation with your peers at a new Technology Salon location:

Best Practices in ICT M&E
July Technology Salon
8:30 a.m. - 10:00 a.m.
Thursday, July 21, 2011
UN Foundation - M Street
1615 M Street NW, 7th Floor - note address change
Washington DC, 20036 (map)

We'll have hot coffee and Krispe Kreme donuts for a morning rush, but seating is limited and the UN Foundation is in a secure building. So RSVP ASAP to be confirmed for attendance or you are on the waitlist.