Transparency Talk

« Glasspockets Find: Knight Foundation Takes the Measure of Social Media | Main | The Giving Pledge Keeps Growing »

Valuing Beneficiary Feedback: Promoting Foundation Accountability and Programmatic Outcomes by Incorporating Recipient Assessments into Decision-Making
May 9, 2013

(Emily Keller is an editorial associate in the Corporate Philanthropy department at the Foundation Center.)

Emily KellerFoundation leaders who want to increase the accountability of their work should consider supporting efforts to solicit feedback from beneficiaries, say three experts in the field of conducting recipient assessments.

To succeed, the feedback must be representative, actionable, systematic, and comparable, said Fay Twersky, Phil Buchanan, and Valerie Threlfall in a webinar presented last month by the Stanford Social Innovation Review (SSIR). The webinar was based on the article, "Listening to Those Who Matter Most, the Beneficiaries," written by the webinar speakers and published in the spring 2013 issue of SSIR.

Despite its inherent difficulties, beneficiary feedback is poised for growth as a method for measuring performance and accountability within the social sector movement toward "big data."

Twersky, who is the lead author of the article and the director of the Effective Philanthropy Group at the William and Flora Hewlett Foundation, said gaining knowledge from beneficiaries, in addition to experts and crowdsourcing, is a moral issue and a smart thing to do to achieve effective program results. "I think if this is important to practitioners to listen systematically and to do it well, it will be important to funders who are responsive to their grantees...I don't think we have done a good job as funders of listening to those voices. I think we can do a lot better," she said.

A 2011 survey of CEOs by the Center for Effective Philanthropy (CEP) showed that just 19 percent of responding foundations use beneficiary focus groups or convenings to assess the effectiveness of their foundation's programmatic work, and 16 percent use beneficiary surveys to do so. Those who collect this information reported having "a better understanding of the progress their foundation is making against its strategies" and "a more accurate understanding of the impact the foundation is having on the communities and fields in which it works."

So why aren't more foundations supporting programs to solicit beneficiary feedback? The webinar speakers examined the issue by discussing benefits and success stories, challenges and criticisms, and best practices for establishing a feedback system.

Benefits and Success Stories
Twersky, Buchanan, and Threlfall drew on their experiences as co-founders of YouthTruth, a nonprofit organization that administers online surveys to high school students across the country, to exhibit an effective system for gathering beneficiary feedback. YouthTruth was created in 2008 by CEP, in collaboration with the Bill & Melinda Gates Foundation, to leverage student opinions into schools' decision-making processes. The questions focus on student engagement, school culture, student-teacher relationships, rigor of classes and instruction, and preparedness for the future. Two hundred thousand students from 275 schools have answered the survey, and 85 percent of participating administrators say they have used the data to make policy or programmatic decisions.

"In the case of YouthTruth, we have seen real benefits — courageous students and schools that have participated in the process — in that it has really opened up new areas for discourse and I think changed both adults' and students' expectations about their involvement in decision-making," said Threlfall, a senior advisor at YouthTruth.

In the healthcare sector, the push for using beneficiary feedback to improve outcomes has focused around patient-centered and accountable care, as measured through initiatives such as the Hospital Care Quality Information from the Consumer Perspective (HCAHPS) survey. The publication of "Crossing the Quality Chasm: A New Health System for the 21st Century" by the Institute of Medicine in 2001 spurred an increase in the collection and use of recipient feedback for this purpose, the authors explained. The Patient Protection and Affordable Care Act continues in this direction with goals for measuring patient experiences during hospital stays and incorporating consumer assessments in determining insurance reimbursements. "When patients have better communication with providers, and when they understand treatment options and feel that they have some say in their own care, they are more likely to follow a treatment regimen and improve their health," the authors wrote in the SSIR article.

They pointed to work by the Cleveland Clinic to increase nurse check-ins as a result of patient feedback, as part of a successful quest for a 90 percent satisfaction rate using HCAHPS; and an initiative by the California HealthCare Foundation to gather feedback from patients outside the commercially insured population that has been the traditional focus of data collection.

The education and healthcare sectors provide unique opportunities for collecting feedback, as the service structure enables providers to track recipient populations and compare information across institutions, the speakers said.

Criticism and Challenges
Despite positive outcomes, experts say that beneficiary feedback is under-utilized by philanthropic organizations for many reasons. According to the CEP study, 73 percent of the responding foundations provided some funding to assist grantees in understanding the effectiveness of their programs, but only 9 percent did so for all of their grantees.

The speakers identified a range of challenges that may explain why beneficiary data is not more commonly used. According to Buchanan, who is president of CEP, soliciting feedback can be expensive and difficult to gather, particularly for populations that are hard to reach, and power dynamics between recipients and service providers can create a barrier to candid information sharing with a high response rate. Twersky cited low literacy rates, trust issues among vulnerable populations, and access to technology as other potential barriers. Allocating funding for surveys may be viewed negatively as an administrative cost that takes resources away from the direct provision of services. Another criticism is that placing an increased focus on outcome metrics can impede innovation and risk-taking. And although recipients may be viewed as customers of a business, they are not the ones paying for the services they receive, the speakers explained, which makes it easier for service providers to overlook their opinions.

Best Practices for Gathering and Incorporating Beneficiary Feedback
Twersky, Buchanan, and Threlfall offered a series of recommendations for collecting and utilizing beneficiary feedback effectively, including the following:

  • Seek Feedback When it Matters

The speakers recommended initiating the survey process before or during a program rather than only after it ends, enabling the data to have the most impact. Twersky compared this to leading indicators used in business.

  • Design Surveys for Impact

The speakers recommended developing a process to integrate feedback early on and to consider the use of focus groups before or after administering surveys, as well as establishing requirements for a high response rate. Buchanan stressed the importance of detailed survey design and methodology and suggested working with consultants or providers if necessary.

  • Strive for Candid, Representative Responses

Threlfall made the recommendation: "Check for non-responder bias to make sure certain populations aren't left out." This requires cultural awareness of the population being surveyed. For example, with smallholder farmers, "Men tend to have disproportionately more access to mobile phones than women, so whose voices are we hearing?" Twersky noted.

  • Prepare for Negative Results

In a YouthTruth video shown in the webinar, Dr. Brennon Sapp, principal of Scott High School in Taylor Mill, KY, described receiving difficult feedback. "One question that will haunt me to my grave is the question that was ‘do your teachers care about you?' We rated bottom 1% in the nation on that one specific question and it really hit hard. It's hard to swallow, hard to take, hard for my teachers to take," he said in the video. Following the survey, Sapp said he worked with staff to change policies and shift the culture of the school, which led to a decline in the failure rate and increased teacher intervention.

  • Collaborate with Other Organizations

Working with other groups enables providers to generate comparative data. In the CEP survey, 26 percent of respondents said they were currently using coordinated measurement systems with other funders in the same issue area, and 23 percent were considering doing so. According to Twersky, developing benchmarks through collaboration will help organizations to interpret the data.

Beneficiary Assessments in the "Big Data" Movement
Despite its inherent difficulties, beneficiary feedback is poised for growth as a method for measuring performance and accountability within the social sector movement toward "big data." The push for evidence-based social programs utilizing impact evaluations was echoed in the Obama Administration's launch of the Social Innovation Fund (SIF) in 2009 to provide million-dollar matching funds to nonprofit organizations chosen by grantmakers that are working in the areas of economic opportunity, healthy futures, and youth development. The SIF's "key characteristics" include: "Emphasis on rigorous evaluations of program results not only to improve accountability but also to build a stronger marketplace of organizations with evidence of impact."

While some challenges will remain insurmountable — as Twersky pointed out during the webinar, "When the intended beneficiary is the earth, how do we listen to the earth?" — there are more than enough resources available to start speaking to more of its inhabitants today.

What are the biggest challenges your organization has faced in collecting and incorporating beneficiary feedback into decision-making? What role should recipient assessments play in the "big data" movement? How can foundations and nonprofits use beneficiary feedback to enable greater accountability and effectiveness? Please provide your comments below.

-- Emily Keller

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Comments are moderated, and will not appear until the author has approved them.

Share This Blog

  • Share This

About Transparency Talk

  • Transparency Talk, the Glasspockets blog, is a platform for candid and constructive conversation about foundation transparency and accountability. In this space, Foundation Center highlights strategies, findings, and best practices on the web and in foundations–illuminating the importance of having "glass pockets."

    The views expressed in this blog do not necessarily reflect the views of the Foundation Center.

    Questions and comments may be
    directed to:

    Janet Camarena
    Director, Transparency Initiatives
    Foundation Center

    If you are interested in being a
    guest contributor, contact:
    glasspockets@foundationcenter.org

Subscribe to Transparency Talk

Categories