EMQ » January–March 2022 » Volume 58 Issue 1
[memberonly folder=”Members, EMQ2YearFolder, EMQ1YearFolder, EMQLibraryInstitution”]By Paul T. Penley
Almost every ministry has a big vision, good intentions, and new ideas for what’s next, but few can demonstrate the degree to which their mission has been achieved so far. Leadership teams enjoy revising their mission and values statements but often struggle to devise specific indicators for tracking progress toward the achievement of that mission. Recognizing this dynamic leads us to ask the question: Does it really matter if a ministry measures its results rather than just staying faithful to its calling?
The Consequences of Not Measuring
Ministries Miss Opportunities to Improve
Without a reliable method for tracking what is and is not working and why, ministry leaders cannot effectively steward financial and human resources toward the most effective strategies. Leaders are left to adjust strategies based on recent conversations with constituents or requests for help, but they cannot assess whether the adjustments were just changes or actual improvements. A good outcomes measurement system tracks factors affecting the results to show leaders which changes to make for better future results. It creates a learning organization.
Donors Lose Confidence
In 2019, 67% of donors said they are more likely today than they were five years ago to favor not-for-profits that provide measurable results on what they are accomplishing.[1] Expectations are rising. However, only 42% of nonprofit leaders think most donors expect them to do impact evaluations.[2] There’s a gap, and nonprofits need to catch up with donor expectations. So far, 52% of nonprofits have paid for an external evaluation of their program impact. That means if a ministry hasn’t completed an independent impact evaluation, they’re already behind. But it’s not just a matter of following a contemporary trend. It matters because ministries who don’t measure impact may poorly steward God’s resources without even knowing it, and more and more donors are starting to care about that important oversight.
Guiding Principles for Measuring Mission Achievement
Measuring something isn’t always better than measuring nothing if you are measuring the wrong things. Let’s establish four guiding principles for what to measure to determine the degree to which a mission is being achieved:
Universal Metrics Don’t Exist
All metrics are values laden. No one-size-fits-all model can account for cultural complexity or divergent opinions about what matters. Organizations have different goals and different values, which means they must customize their impact scorecard. Whereas the Effective Altruism movement might prioritize the number of deaths you can prevent for the least dollars,[3] someone else will question whether extending life on earth is the ultimate priority. If a poverty alleviation effort doubles someone’s income, should we celebrate the impact of getting richer if the money gets spent on beer and brothel visits instead of family healthcare and education? Establishing key performance metrics requires a nuanced conversation that goes beyond the typical mission, vision, and values brainstorm session.
Christian evangelism ministries have for decades celebrated the number of indicated decisions for Christ, but few have tracked the percentage of converts who get connected to an ongoing Christian community. If God’s dream all along has been to form a special interdependent community of people who embody his values, then is an individual’s internal decision a success if they remain in social isolation from other believers and unable to live out the fifty-three one another commands in the New Testament? We will each answer these questions differently based on the current shape of our values and hard-earned wisdom. Ministries can assist potential donors by translating their mission and values into specific measures of effectiveness so that they can clearly discern whether their values align by reviewing the accomplishments that are measured.
Activities and Outputs vs Outcomes
Reporting that a ministry was busy doing a lot of activities with a large number of people won’t cut it anymore. Serving one hundred children through five hundred counseling sessions, having two thousand people complete your curriculum, or exceeding one hundred thousand attendees at conferences doesn’t mean anything will be better a year later. Those are just activities and outputs. For example, reporting 120,000 volunteer hours produced 168,000 warm meals does not give evidence of any problem being solved. If an organization celebrates the 168,000 warm meals was up 20% from last year, it could be proof that the problem is getting worse, not better. To survive, more people need more meals that they cannot afford on their own.
To discern if positive progress is being made, a lasting outcome needs to be measured separately from all the inputs, activities, and immediate outputs, and after a person has been independent of the organization’s services for a significant period:
- Input: 120,000 volunteer hours spent cooking meals
- Activity: 168,000 warm meals served to 7,000 clients
- Output: 180 clients completed the “Get Your Life Back” program
- Outcome: 61% of program graduates have jobs and pay their own rent and food costs six months later
Reporting inputs, activities, and outputs in a visually enhanced annual report helps people understand the scope of a ministry’s work, but not its significance. All the big numbers could be a flash in the pan, a distraction from what matters, unless sustained improvements can be observed months and years later in people’s lives as they continue to do the right things without ongoing dependence on the ministry. Other examples include:
- Educating parents on how to guide teens through social media isn’t a success until parents start conversations and ultimately see their teen delete an unproductive account or add time limits to their usage – which Axis has done.
- Training rural farmers to use advanced agricultural techniques doesn’t matter until farmers employ those techniques and double their income and daily meals consumed – which Plant with Purpose has accomplished.
As long as ministries track their activity but don’t measure outcomes, neither the staff internally nor supporters externally can know if all the time and money has been stewarded wisely.
Baselines and Benchmarks
Measuring meaningful outcomes requires the comparison of results to a baseline or benchmark. Simply reporting a 90% high school graduation rate for kids in a mentoring program or tracking how two churches are planted in the next three years after pastors are trained doesn’t mean an intervention had any impact. We first need to know a counterfactual – what would have happened to people in the program if they had not participated in the program. A high school ministry celebrating its 90% graduation rate needs to realize that 79% of the public school students in its program would have graduated anyway. So, they can only take credit for helping 11% to graduate.
If a church-planting ministry observes its trained pastors starting two new house churches within three years of completing their training but doesn’t track or report the two house churches planted in the three years before their training, the ministry might simply be taking credit for what the pastors would have done anyway without the training. Every measured and reported outcome must be delivered with a baseline or benchmark that describes what happens to the same kind of people who don’t access that program.
I still remember the first time I read an impact report from a youth ministry that celebrated how 12% of their program participants went to jail. That doesn’t sound like good news you want to announce. However, the benchmark for similar youth who were already court-involved in Massachusetts predicted that 60% would end up in jail within three years of release. So, the ministry was reducing incarceration rates by 80%! Without such benchmarks, outcomes are meaningless. It may be too expensive or complex for small and medium-sized ministries to build experimental counterfactuals through pipeline methods or propensity matching with non-participant pre/post surveys, but some type of historical baseline or contemporary benchmark is necessary for reported outcomes to hold any significance.
Righteousness over Religiosity
For Christian ministries, spiritual impact must be measured. If you have quickly assumed spiritual transformation can’t, or even shouldn’t, be measured, ask this question: Could Jesus radically transform a person or community without any observable evidence? It may be inadvisable to track how God works inside someone’s heart, but we can see the ultimate fruit of his Spirit. The difficulty is selecting the appropriate measures with biblically informed discernment.
Religion questionnaires do exist (e.g., Duke University Religion Index or the Santa Clara Strength of Religious Faith Questionnaire). However, questions about religious service attendance, efforts to please God, and daily spiritual disciplines can measure a legalistic form of religion that doesn’t match Jesus’ values. So positive spiritual impact may actually coincide with negative religious growth.
An instructive critique of religiosity and a helpful clarification of God’s preferred metrics for spiritual maturity occurs in Isaiah 58. God criticizes people who worship the right way at the right place on the right day, but who treat people unfairly and mercilessly. Instead of putting on the right clothes and bowing down low to pray while they fast, God says, “No, this is the kind of fasting I want: Free those who are wrongly imprisoned; lighten the burden of those who work for you. Let the oppressed go free and remove the chains that bind people. Share your food with the hungry and give shelter to the homeless. Give clothes to those who need them, and do not hide from relatives who need your help.”
Righteousness in God’s kingdom, whether defined by Isaiah 58 or by Jesus in Matthew 25:31–46, is characterized by compassionate action and legal justice, personal generosity, and ethical business practices. So spiritual impact must go beyond measuring religious activities to assessing volunteer work, business practices, financial generosity, legal advocacy, and care for one’s family. Faithfulness to Jesus could see a decline in religious services attended each week so that adherents can care more for people!
Survey instruments, such as the German SpREUK-P SF17, more carefully differentiate five unique spheres of religious and spiritual expressions: Religious practices, Prosocial-humanistic practices, Existentialistic practices, Gratitude/Awe, and Spiritual (mind body) practices.[4] Any useful set of spiritual metrics should similarly differentiate knowledge, attitudes, and behaviors expressed individually, in Christian community, and in the world so that the interplay of intrinsic and extrinsic personal and group factors can be assessed to determine what type of spiritual or religious formation is happening. The last thing any Christian ministry would want is an increase in personal or public religious behaviors that does not coincide with internal gratitude, emotional peace, and spiritual connection with Christ leading believers to loving engagement with people in need.
What Can Be Measured
Understanding the guiding principles and purposes behind impact measurement can lead to the practical question: What really can a ministry measure beyond (1) people served and (2) activities completed that is compelling to both internal and external stakeholders?
As I have helped numerous ministries over the past decade set up outcomes measurement systems and then reviewed the outcomes of hundreds more, some of the most impressive results have been:
- Haggai International tracked over five years how its trained Christian leaders doubled the number of converts from their evangelism efforts and trained five times more people to do the same, compared to their baseline trajectory.
- AWANA alumni in the United States (ages 18–74) were twice as likely to share the gospel last year compared to the average American Christian – 82% sharing their faith versus 37% of American evangelicals.
- AXIS users who educated themselves on tough topics that teens face today and then discussed it with the youth in their life saw a concrete behavioral improvement on at least one issue 60% of the time – behavior changes included adding porn blockers to a phone, deleting a social media account, stopping drug and alcohol use, ending self-harm, graduating from anxiety meds, etc.
- Plant with Purpose program participants and their neighbors experience a 60% reduction in poverty indicators (inadequate food, unsafe home, etc.) when they apply better agricultural practices and start small businesses, while nearby communities are not improving at all.
- World Relief’s Church Empowerment Zone in Rwanda reduced the number of men who feel justified in beating their wife by 69% compared to neighboring communities.
This quick sample of what’s been done is meant to raise your expectations of what is possible. Serious impact measurement work can demonstrate how people served by ministries are doing (1) better than they were before and (2) better than their peers.
An International Case Study
To get a more detailed look of what impact can be measured and what it takes to assess it, let’s focus on a massive 5-year study that I completed in 2020 on John Stott’s legacy ministry, Langham Partnership. Langham operates three major programs that actively serve people in over one hundred nations in the Majority World: Langham Scholars, Literature, and Preaching programs. The diversity of programs and locations created the need to custom-build the evaluation cycles and instruments to assess impact. The custom design process involved circumnavigating the globe doing qualitative research and implementing a quantitative results-gathering process from Bible scholars, authors, colleges, publishers, and preachers. After the arduous process of managing data collection over five years – including multiple random reliability checks for data reported across language and cultural barriers – Langham now has a comprehensive understanding of verified results for its three programs. The highlights below show what can be measured after a scholar earns a PhD, literature is distributed to Bible colleges, and preachers are trained.
Langham Scholars Program
Using a logically constructed counterfactual from each scholar’s previous teaching record, Langham discovered that an earned PhD triples the number of students that a Langham Scholar will teach during a 25-year career as a professor. The competency to teach more courses and invitations to teach at more places give scholars with a doctorate access to almost five thousand more students than they would have been able to reach without a PhD.
However, Langham scholars in the Majority World do much more than teach: two-thirds become board members of international organizations in the first five years – which is five times more board roles than before they completed a PhD.
- 51% launch a new ministry within 10 years of graduation, serving over 700 people each
- By year 15 of their career, three-quarters become senior school administrators and grow student enrollment by over 100 students under their leadership.
- During a full 25-year career, where scholars with PhDs write four times more books and articles than their previous pace without a PhD, two-thirds produce the very first book on a key Bible or theology topic that literally no qualified Christian from their country has ever written before!
Langham Literature Program
Creating and distributing books produces easily measured outputs, but how do you know if all the activity leads to meaningful outcomes? To answer that question, Langham employed annual and retrospective assessments of both the book problems that the Majority World faces when teaching the next generation of Bible teachers and the degree to which Langham’s solution is working (Table 11.1).
| PROBLEM | BASELINE | PROGRESS | SIGNIFICANCE |
| Foreign author annual book sales (for twenty-two Langham Majority World publishing partners) | 91% | 82% (10 years later) | Christians are buying twice as many books by qualified indigenous authors (instead of foreign authors) who can relate culturally and speak to their heart issues. |
| Bible colleges and seminaries can’t afford the books they need | 84% | 78% (3 years later) | The need for distributing free books is slowly declining, so Langham can focus book grants on schools where it has the most impact. |
| Bible colleges and seminaries can’t offer courses because they don’t have the books | 41% | 32% (3 years later) | Langham’s book programs enabled 773 Bible colleges to add 842 courses in 2019 so that now one third of all courses, or 13,295 courses, require students to use a book provided by Langham in the past. Those colleges reported that 23,348 of their 204,536 students are only able to be trained today because of Langham. |
Langham Preaching Program
Since the Langham Preaching program changes participants’ understanding of terms and practices, a retrospective study was done to assess the following growth from a baseline (Table 11.2).
| PROBLEM | BASELINE | ENDLINE | SIGNIFICANCE |
| Sermon prep time | 6 hrs. (1 out of 4 did no prep) | 15 hrs. (6 years later) | Sermon prep time more than doubles as preachers work to deliver the message of the passage rather than just their opinion. Use of five specific sound Bible study skills tripled. |
| Preaching verse-by-verse thru Bible books | 14% | 54% (6 years later) | Audience learns how each passage fits into the original author’s complete message, rather than just hearing the preacher’s favorite topics. |
| Preachers ask audience if their sermon was clear, relevant and faithful to the Bible’s teaching | 12% | 63% (6 years later) | Humble requests for critical feedback grows fivefold as preachers demonstrate more humility. |
The improvement of Langham-trained preachers is so drastic that 74% are asked to preach in new places. So, the quality and quantity of their preaching grows.
Continuous Stewardship Improvement
Measuring ministry impact is doable, though difficult. It requires custom-building metrics to match the mission and tracking progress from a baseline or benchmark over time. But measuring meaningful outcomes that remain after beneficiaries are no longer dependent on the ministry is mission critical. Good leaders and wise stewards always want to know what is and is not working. They don’t pretend to have perfect organizations; they create learning organizations, and preferably ones that learn fast.
Every ministry should be able to provide a 1-page summary contrasting what happens to people with and without their intervention. They should be able to provide a copy of their latest external evaluation and cite the one or two key improvements they made based on the results. Our missions are too critical, and people’s lives are too important for us to settle for anecdotes and unknown results.
Don’t be satisfied with good intentions and an inspiring vision to solve a global problem; make sure the solution works. People have enough opportunities to donate nowadays; it’s time to provide evidence-backed opportunities to make a measurable difference.
Dr. Paul T. Penley is Managing Director of the Research Division at Excellence in Giving – a national philanthropic advisory firm that enables faith-based philanthropists to invest about $100 million a year in the best ministry solutions to the world’s toughest problems. He built the Nonprofit Analytics platform to evaluate the organizational health and performance of thousands of ministries so donors could make wise giving decisions and has designed an outcomes measurement process for dozens of ministries who want to prove what works and improve what doesn’t.
EMQ, Volume 58, Issue 1. Copyright © 2022 by Missio Nexus. All rights reserved. Not to be reproduced or copied in any form without written permission from Missio Nexus. Email: EMQ@MissioNexus.org.
[1] Penelope Burk, Burk Donor Survey (2019), 11, https://cygresearch.com/wp-content/uploads/2019/12/the-2019-burk-donor-survey-executive-summary.pdf.
[2] According to three thousand non-profit leaders surveyed. William Meehan and Kim Starkey Jonker, Engine of Impact (Stanford, CA: Stanford Business Books, 2018).
[3] See https://www.effectivealtruism.org.
[4] Arndt Büssing, Franz Reiser, Andreas Michalsen, and Klaus Baumann, “Engagement of Patients with Chronic Diseases in Spiritual and Secular Forms of Practice: Results with the Shortened SpREUK-P SF17 Questionnaire,” Integrative Medicine: A Clinician’s Journal 11, no. 1 (2012): 28–38.
