Measuring the Intangible: Social Justice Evaluation

If you’re a social justice organization, how do you measure your impact? How do you assess advances in racial equity, gender parity, or equal access to educational or economic opportunity? How do you evaluate progress toward the unmeasurable?

This question of how one measures the intangible sits squarely at the intersection of the changes social justice organizations are trying to effect, and the growing demand for impact assessment. But the answers don’t fit tidily into boxes, just as societal problems are not easy to disentangle from the systems that generate them. Nonetheless, heightened awareness about the need for and significance of equity-focused evaluation has given rise recently to new strategies and resources, particularly from the philanthropic sector. These newer approaches provide welcome tools in aligning organizations’ day-to-day work and their evaluation methods. 

The evaluation community has similarly recognized the need for equity and cultural competence to be “baked into” evaluation from the outset, as the American Evaluation Association’s (AEA) Statement on Cultural Competence demonstrates. Jara Dean-Coffey, Jill Casey, and Leon Caldwell explain in their article, “Raising the Bar—Integrating Cultural Competence and Equity: Equitable Evaluation”:

Whether implicit or explicit, social justice and human rights are part of the mission of many philanthropies. Evaluation produced, sponsored, or consumed by these philanthropies that doesn’t pay attention to the imperatives of cultural competencies may be inconsistent with their missions…Because the act of evaluation is itself part of the intervention, an equity lens is paramount when evaluating a program whose goals touch on issues of equity or inclusion.

Here are four actions that funders and social justice organizations can take as they seek to include that equity lens in their evaluation efforts:

Understand Why Equity-Focused Evaluation Matters: My M&E, a platform managed by the United Nations International Children’s Emergency Fund (UNICEF) and the International Organization for Cooperation in Evaluation (IOCE), offers monitoring and evaluation information, including the purpose, need, and importance of equity-focused evaluation.  As part of its overview on evaluation and good practices, UNICEF offers a handbook, “How to design and manage Equity-focused evaluations,” which provides rationale for such evaluations, strategies for managing equity-focused evaluations, and information on evaluation design, framework identification, and real-world challenges. And the aforementioned article by Dean-Coffey and her colleagues presents an equitable evaluation capacity-building (EECB) approach that can help organizations normalize and institutionalize equity-focused evaluation in a manner consistent with social justice goals.

Challenge Assumptions and Intentions Early and Often: Evaluators bring assumptions into their work—often unconsciously, and often with the best of intentions. It is critical that we explicitly face these assumptions and biases, question them directly, and consider how they will impact evaluation efforts, the data collected, and the audiences with whom their results will be shared. Fabriders offers a list of Questions to Ask Frequently (QAFs) When Working with Data and Marginalised Communities that helps evaluators understand and respect the relationship that will develop between themselves and the communities they seek to assess. Racial Equity Tools’ Getting Ready for Evaluation provides resources for groups preparing for the evaluation process, including Tip Sheets for considering the why, who, and how of assessing marginalized communities. 

Consider Embracing (Rather than Avoiding) Intangibles: Some evaluators are directly embracing intangibles as part of their evaluation process. The Inter-American Foundation, for example, recognizes that the grassroots development work it funds supports impact at various levels, and that both intangible and tangible results are meaningful. Its Grassroots Development Framework, which is the foundation of its evaluation approach, values both tangible and intangible returns, and acknowledges that each type of return can be evidenced at individual/family, organization, or societal levels. By explicitly assessing intangible benefits of its grassroots development efforts, the Foundation is able to better assess the longer-term changes it seeks to create through its grant making programs.

Create Conditions for Success: There is a strong urge to apply laboratory-like, case-control standards to evaluation of social and policy interventions. But the truth is, evaluations in the real world seldom identify cause-and-effect pathways with absolute clarity. That fact doesn’t undermine the value of evaluations, however—it merely points to the enduring merit of identifying “contribution, not attribution,” as Grantmakers for Effective Organizations (GEO) and the Council on Foundations put it. Phrased another way, attempting to show cause and effect definitively may be an exercise in futility within complicated, poorly controlled real-world environments. Increasingly, as Soya Jung notes in her article, “Foundations Share Approaches to Evaluating Racial Justice Work,” sponsors and consumers of social justice evaluations recognize they may “need to let go of the desire to pin down causality altogether and to focus instead on creating the conditions that make social change more likely to take place.”

 

How does your organization incorporate equity and cultural competence in its evaluation efforts? How has doing so advanced the mission and vision of the organization as a whole?