In my last post, I posed a question: If you’re a social justice organization, how do you measure your impact? Hawaii Appleseed Center for Law and Economic Justice (Hawaii Appleseed), a nonprofit law firm dedicated to advocacy on behalf of Hawaii’s low-income communities, offers firsthand insights into the inherent challenges as well as the learning opportunities evaluation can provide to a social justice organization. A small but mighty force for change, Hawaii Appleseed’s efforts encompass: research on housing, health, education, immigrant rights, and economic justice; legislative and administrative advocacy to ensure that laws and policies impacting those in poverty are legal, fair, and effective; community education and outreach efforts; partnership with other community-minded groups in grassroots coalitions; and, when needed, litigation to protect the rights of low-income individuals and families.
Via email, I recently asked Gavin Thornton, co-Executive Director of Hawaii Appleseed, to describe the ways Hawaii Appleseed incorporates evaluation into its organizational work, and what evaluation makes possible for the organization and for those it serves. As he thoughtfully details below, evaluation has helped Hawaii Appleseed identify systems that perpetuate poverty, think strategically about where to invest its staff resources, ensure the organization’s day-to-day efforts are on the right track, and contribute to and build on existing knowledge about how to effectively increase the opportunities and well-being of those in poverty.
How has Hawaii Appleseed approached evaluation? What does evaluation make possible for a social justice organization like Hawaii Appleseed?
Hawaii Appleseed’s mission is to create a more socially just Hawaii, where everyone has genuine opportunities to achieve economic security and fulfill their potential. We change systems that perpetuate inequality and injustice through policy development, legislative advocacy, coalition building, and litigation. This is an ambitious mission, especially for a small organization with limited resources (currently with only three permanent staff and a budget of around $300,000).
To fulfill our mission, we search for projects that will maximize our impact and return on investment—projects that will get us the most bang for our buck. Much of the work that we do is based on what has worked well elsewhere—evidence-based practices or promising practices that have been demonstrated as an effective means of improving opportunities for self-sufficiency. To evaluate the impact of our projects, we collect data on what we are able given our limited resources, but often must rely on conclusions we can draw based on research done by others, as I describe in more detail below.
Evaluation is critical to our work because it allows us to strategically advocate for the changes necessary to achieve our mission. We need to know if what we are doing is working, and if it is not, we need to modify our approach or focus our energies elsewhere. Indeed, this really gets to the core of the organization. Our focus is looking at systems that are keeping people in poverty—systems that are broken, but continue to stumble along because no one has made the effort to step back and recognize the deficiencies and correct them. After identifying systemic problems, we figure out how to change the systems so they create opportunity instead of stifling it. For the most part, the policy analysis we conduct and the reports we write are evaluations of broken-down systems. Since these evaluations are at the core of our work, we recognize the importance of self-evaluation, and trying to ensure that we are making the most of what we have and that our work is accomplishing its intended purpose.
Because the legal or policy successes Hawaii Appleseed pursues can take months or years to occur, how does the organization know—on a day-to-day basis—whether its work is heading in the right direction?
Perhaps surprisingly, we can often see immediate positive change resulting from our work. One example is a case we filed on behalf of the tenants at the Mayor Wright Homes public housing project, where over 360 households had endured years of unsanitary and unsafe living conditions including a lack of hot water, vermin infestation, and dangerous criminal activity on the premises due to lack of upkeep of the property and inadequate security. Even while we were still working up the case prior to its filing, the attention drawn to the problem resulted in the commencement of a significant rehabilitation of the project. The hot water system was fixed shortly after the case was filed, and by the time the suit was settled with a commitment to complete necessary repairs at the property, over $4 million in repairs had already been made, dramatically improving the condition of the project.
In situations like the Mayor Wright case, we do not even need to win the case to achieve a successful outcome—while a loss in court on Mayor Wright would have limited the impact of our work, it would not have discounted the benefits already accrued to the tenants (unless, of course, it was an early loss, before the benefits accrued). However, some of our work is more of an all or nothing proposition. For example, for the most part, legislative advocacy does not result in significant change unless the bill gets passed. Yet even in the legislative context we can still evaluate our progress on a day-to-day basis. We look at the number of partners we have recruited to our coalitions and the extent of their engagement; we count the number of legislators supporting our bills; we look at the media coverage of the issue—the number and quality of the stories. It is not necessarily a scientific process, but it does provide a general sense of whether we are getting traction on an issue. Ultimately though, none of this matters if the bill does not pass. Yet these indicators are critical in deciding whether it makes sense to continue pursuing an issue year after year.
What is a particular challenge Hawaii Appleseed faces in its evaluation efforts, and how has the organization faced that challenge?
One recurring problem that we often have in evaluating our work is that of attribution—there is often a degree of uncertainty regarding whether it was our actions or something else that created the change we sought. Success in the legislature requires a group effort, and so it is hard to say who or what was responsible for the result—the answer is usually that all, or nearly all, of the participants had something to do with the outcome, as well as external factors. While it would be nice to be able to quantify the impact of our work in some way—we expended x dollars and y hours which resulted in the achievement of z benefit—it is not practical, and we need to be content with more vague assessments of our work. For example, in advocating for Accessory Dwelling Units to be permitted on Oahu, we developed a policy brief to start a discussion (that we believe was not being had prior to our work on the issue, though at least one academic researcher had been looking at the issue). Then we spearheaded an advocacy effort to get a bill passed through the city council. Our work was cited in the mayor’s affordable housing plan for Honolulu, and we helped craft the bill that was ultimately passed. Based on this, we felt it was reasonable to conclude that that change was a direct result of our efforts—that it would not have occurred without our work—but it is still difficult to say definitively.
This difficulty in attribution is similar for our litigation efforts. Frequently, the response to the cases we have brought has been, “We’ve been looking at this issue and working on it; this is something we were already in the process of fixing before the case was even filed.” I suspect that this response often reflects the genuine beliefs of person or agency making the claim. However, since nearly all of our cases relate to issues that had persisted openly for years, and on which no action was taken until we began work on the case, the claims of “we were going to do this anyway” are questionable. For example, in the Mayor Wright case, there had been multiple stories in the newspaper about the lack of hot water during the seven years preceding the filing of the case, and yet the problem was never adequately addressed. In a case we are currently working on regarding the inadequacy of payments made by the state for the care of children in foster care, the payments—which federal law requires to be regularly updated to account for inflation—were not increased for nearly 25 years, even after five years of advocacy in the legislature by foster parents and advocates seeking an increase to the payments. The payment was increased within months of filing our case, but the state claimed that it was something they were going to do anyway. In circumstances like these, where a problem has persisted for years without action, but is addressed (or at least partially addressed) shortly after we bring a case on the issue, it seems very likely that our work provided the critical push to create the action necessary, but we can never be 100% certain regarding what would have happened had we not taken action.
How does Hawaii Appleseed demonstrate community-level impact that may be difficult to quantify? How does the organization assess its progress, for example, in advancing self-sufficiency or economic security for low-income families?
This is incredibly difficult, especially for a small organization like ours that does not have sufficient resources for the type of evaluation necessary to evaluate community impact. Instead, we are often forced to draw conclusions based on research done elsewhere. The Mayor Wright case provides a good example of this. We know that we obtained over $4 million in repairs and improvements to the project, but obtaining improvements to public housing projects is not our mission. What we really care about is creating an environment that will allow people to become self-sufficient. What difference does it make that a child is growing up in a place with working hot water, no holes in the wall, no bedbugs, and that is relatively free from criminal activity, versus a place that does not have those things? We do not know. However, we do know that there is a strong correlation between healthy, safe housing and positive health and economic outcomes. If we had the resources, we might be able to quantify the impact of our work in those terms—though even with significant resources it would still be very hard to do. For now, we are left with more vague notions of, “we know that healthy and safe living environments are important, and we obtained an improvement” (which we have been able to assess, not only by dollars spent on repairs, but also surveys of the conditions at the project).
For many of our projects, we can get a much clearer picture of the ultimate results, but we are still required to draw conclusions based on studies done elsewhere. For example, we have a project that seeks to increase participation in the school breakfast program. Participation rates in the school breakfast program (i.e., the percentage of children that eat school breakfast) are not important to us in and of themselves. However, there is a strong body of research that shows that low-income children who eat school breakfast benefit in clear, quantifiable ways, such as improved academic performance and increased health—things which are correlated with self-sufficiency later in life. We gather whatever data we can, for example on participation rates, visits to the school nurse, absenteeism, perceptions of classroom behavior, etc. Then, based on research done elsewhere, we can get a rough idea of the increased opportunities for self-sufficiency that the children are likely receiving as a result of our efforts.
How does Hawaii Appleseed engage the clients it serves in its advocacy and research efforts? What has it learned from doing so?
Because of our limited size and our area of expertise, we often must rely on other partner organizations to engage with those that we serve. Nearly all of the issues that we work on come from the community through other community organizations that contact us seeking help to resolve a problem. For example, we recently worked on a case that resulted in the drivers’ exam being translated into multiple languages. The issue was brought to us by a faith-based community organizing group whose members/constituents had identified the inability to obtain a license due to language barriers as a significant issue that was preventing people from getting to work and supporting their families. The group had already engaged the community in a variety of advocacy actions prior to our involvement, and continued to work directly with the community while serving as a liaison between the community and our organization during the course of the case.
That type of structure is workable, but it requires good communication, strong relationships, and everyone doing their part to make it work well. There is another organization in the national Appleseed network—Nebraska Appleseed—that has community organizers on staff, as well as attorneys and policy analysts. That allows Nebraska Appleseed to have direct relationships with the community it is serving, while at the same providing policy development and advocacy expertise. That model is attractive because it makes communication easier and provides a lot more control over the work, but it requires significant resources. There have been times where we have been able to serve both roles, but it is very difficult to do so well given our size. As such, we recognize the importance of continuing to strengthen relationships with other organizations that have more direct contact with those we serve, and also of building capacity so that we can have more of that direct contact ourselves.
How has your organization faced the challenges of measuring social justice impact? What lessons has your organization learned in evaluating its efforts?