Evaluating the Impact of Social Media: Are We Wasting Our Time?

(Claire Gibbons, Ph.D., M.P.H., is a senior program officer in the Research & Evaluation Unit at the Robert Wood Johnson Foundation. She spends most of her time managing R&E projects for the Quality/Equality team.)

Last month Steve Downs and I discussed some of the Robert Wood Johnson Foundation’s (RWJF) experiences using social media and our first steps towards evaluating the impact of RWJF’s social media use in a webinar for Council on Foundation members (you can view the slides here). In response to our evaluation discussion, a webinar participant asked whether it makes any sense to evaluate something as spontaneous and fun as social media. This was also a question raised by Allison Fine in her blog. 

Allison expresses concern that a logic model “…misses the essence of what makes social media so unique, the serendipity and fun that are essential parts of “being” social”. This is an interesting and valid question- by creating a stodgy old logic model do we defeat the purpose of social media?

Before I share some thoughts on this question, let me describe briefly what we’ve been doing at RWJF. The staff at the Foundation is using social media, and many are enthusiastic about its potential to increase our impact, but until recently, no one had sat down to elicit exactly what we expect social media to impact. In fact, we engaged in a fairly lengthy period of discussion and experimentation before we began to plan for evaluation. One of RWJF’s initial steps was to form a working group to consider how the Foundation could best take advantage of Web 2.0 tools, and what it would mean for the Foundation if it did use these tools. This working group released a report internally to all Foundation staff in December of 2009. This began a period in which all staff was strongly encouraged to experiment with social media. Just signing up for Twitter and listening to the conversation by following others was encouraged, for example. A second Web 2.0 group was formed after some time passed that was charged with getting some sense of whether RWJF was moving forward with its use of social media and sharing lessons across program areas. It was at this point that we began to focus our attention on evaluation.

We decided that the first step in evaluating our use of social media should be to develop a logic model. We did this, with the help of consultant Victoria Dougherty, based on interviews with staff that were knowledgeable and involved in our social media efforts and on review of documents about our social media philosophy.

RWJF created a logic model to help evaluate the impact of social media.

View a PDF of the logic model »

The logic model has two pathways: the first describes how RWJF can approach its work over the next five years and the second describes some of the outcomes of the work. For RWJF to realize the potential of social media and eventually reach its long-term goals for being a more effective agent of change, and being a connector and facilitator that spurs broad participation in our work it must first position itself as a Web 2.0 organization and work to become more open and nimble. Social media use may also lead to creating new connections outside the Foundation and in turn lead to a greater ability to gather information from a broad network that can result in more effective programming. See my earlier blog post here for a more in-depth discussion of this logic model.

So, back to our earlier question: Does creating a logic model to drive an evaluation of the impact of social media defeat the whole purpose?

I don’t think so. But I’m pretty sure you guessed I was going to say that, given that I’m a Research & Evaluation Officer! So let me share my thinking.

1. Yes, social media in many cases is driven by spontaneity. Videos that go viral on YouTube are completely driven by spontaneous interest. But not all social media use is purely spontaneous. We believe that social media can be used strategically to further our programmatic goals. That means we can plan ahead to use a social media tool or tools. For example, staff at RWJF used a virtual forum to create an open platform for discussion and idea-gathering about teen dating violence prevention. They received thoughtful input from people working in the field, teens who had experienced dating violence and parents who lost a child due to dating violence, as well as many others. Read more here.
2. A logic model does not squash innovation! It describes it. Logic models are made to be broken and expanded and changed over time. The presence of a logic model is not meant to limit anyone’s activities to something that happens to be featured in a little box in the model. The logic model doesn’t dictate our programming- it’s simply a way to describe what we are doing and what we think the result will be.
3. RWJF’s use of social media in the workplace is predicated on the idea that it will help us achieve our goals. We could be wrong. We won’t know if social media is getting us anywhere good, or anywhere good faster, unless we measure some outcomes that we think are related to our programming activities. And one very useful tool for eliciting expected programmatic and policy outcomes is a logic model.

This isn’t to say that we think all possible pathways between use of social media and some good outcome are contained in our logic model. Absolutely not! This is just the best picture we could come up with at this point in time. New tools will become available, and we will use them. Staff will continue to innovate in ways that we haven’t imagined yet, and we welcome that innovation.

We still have a long ways to go in our journey to use social media in a way that helps us reach our strategic objectives, and in measuring and evaluating our use of social media. We certainly don’t have many answers, but think we’re on the right track. What do you think?

-- Claire Gibbons

Comments

[Brainbasedbiz.blogspot.com]

Claire, You have developed an amazing logical model here. Social media can tap into both logic and relationships depending on how you use it. In doing so it brings out a holistic view. No matter how we try to measure, would you agree that there is an element of ambiguity involved?

[Sksmith1881]

Great post, thanks for sharing! What social media tracking tools do you find useful?

[Miftacenzie]

Fantastic. Great approach by the city cocunil. How else do we discipline Nairobians and keep at least the CBD clean and habitable. Anything to get us neat, and forget the stench, squalor and garbage in all the lanes of Nairobi during the nyayo-years. Well, I hope I dont get caught for the petty offences If the askaris/cocunil-police take petty bribes, its coz we are indisciplined