Social Impact Measurement, Part II: a Fit-For-Purpose Mindset Towards Data

This post is about data – and buckets.

First, let’s talk data. We are in the midst of what many (such as the United Nations) call a “data revolution.” This revolution includes an astounding growth in the generation and availability of data. Likewise, it includes an accelerating demand for using data to inform decision-making. Think “evidence-based practice.” Or think about that list of metrics that your funder just released that you need to include in your next evaluation report. (If you are a funder, I am talking about that list of metrics you just sent to grantees.)

To elaborate, let me share a story about buckets. The moral is that the for-profit sector often reminds us to take a fit-for-purpose view towards data and metrics. You do not always need more or better data; you need just enough to inform the decision at hand and facilitate your mission.

For some time I have been consulting with CompostNow, a for-profit social enterprise. Its customers (households, restaurants, and offices) put food scraps into buckets; CompostNow collects the scraps and creates nutrient-rich compost for local farms.

Some months ago, CompostNow was preparing for an important transition. Household customers in Atlanta had for years been putting their scraps into 5-gallon buckets. For various reasons, it became necessary to switch to 4-gallon buckets.

As an evaluator I often look several steps down the line, and so I anticipated an important question: how many 4-gallon buckets should each customer receive? While most customers would be fine with one 4-gallon bucket, some had routinely filled their 5-gallon bucket to near capacity and therefore may need two 4-gallon buckets to fit all of their scraps moving forward.

It seemed that we could and should make a data-driven decision here. Since the company logs food scrap collection weights for every customer, we needed to look at the historical range of collection weights for each customer and decide accordingly how many 4-gallon buckets they would need.

I brought my insight up during a meeting:

Me: …and then we’ll determine how many buckets each customer gets, based on the historical distribution of food scrap weights we have collected from them.

CEO: Why?

[Pause.]

Me: Why—what do you mean?

CEO: Why don’t we just give each customer one bucket? They can tell us if they need more.

See, from the CEO’s perspective, analyzing bucket weight data for each individual customer was not necessary. We would instead simply deliver one bucket to all customers and rely on a more direct and relevant source of information – customer requests/feedback submitted through their online portal – to deliver additional buckets as necessary.

I share this story because those of us working to promote the greater good all have decisions to make, small and big. Maybe the question is: “How many buckets?” Maybe it is: “Should we continue this program?” So how we should use data to guide us?

In wrestling with how to use data effectively, there is wisdom in a “fit-for-purpose” mindset that I have found to predominate in for-profit settings. Below I provide an overview of two related aspects of this mindset, drawing upon a 2016 conference that convened leaders in the for-profit sector and the evaluation profession.

• “Enough precision for the decision.” This phrase, from Jeremy Nicholls, emphasizes that the purpose of data is to support strategic decision-making. Accordingly, the importance of the decision at hand (and the consequences of getting it wrong) should determine how much time and money to invest in good data and analysis. This is a Goldilocks approach to data – not too little, but not too much, for the decision at hand.
• Adaptability. Many panelists touched on this, including Markese Bryant, who spoke about adaptive philanthropy. They described a process of innovating, collecting quick data, and making changes accordingly. Panelists juxtaposed adaptive businesses (which value quick learning) with risk-averse nonprofits/governments (which value adherence to plans/protocols). While the idea of innovation to some may provoke ethical concerns about experimenting with people’s lives, several panelists noted the ethical pitfall of being risk-averse: when we say that we need more/better data before making a change, we implicitly assume that the status quo is acceptable in the meantime.

Let’s bring these concepts back to the bucket example. Deciding how many 4-gallon buckets to give each customer did not require any more precision than the knowledge that most customers’ old 5-gallon buckets were typically not overflowing. And the company’s customer service system provided adaptability – customers could easily submit a request for an additional bucket if needed.

While this fit-for-purpose mindset towards data is by no means exclusive to for-profit settings*, the for-profit sector’s unbending emphasis on efficiency provides a useful compass when it comes to data. Rather than distracting or overwhelming us, data should serve our decision-making processes and our ultimate purposes of making the world a better place.

* See references ² and ³ for overviews of a couple of frameworks common in the public/nonprofit realm that emphasize that data’s usefulness is in terms of informing decisions and solving problems.

1. W.K. Kellogg Foundation. (2017). The Step-by-Step Guide to Evaluation: How to Become Savvy Evaluation Consumers. Available at https://www.wkkf.org/resource-directory/resource/2010/w-k-kellogg-foundation-evaluation-handbook
2. Carnegie Foundation. Using Improvement Science to Accelerate Learning and Address Problems of Practice. 2018; Available from: https://www.carnegiefoundation.org/our-ideas/
3. Better Evaluation. Utilization-Focused Evaluation.  5/3/18]; Available from:http://www.betterevaluation.org/en/plan/approach/utilization_focused_evaluation