Process with purpose: Looking beyond the short-term with program evaluations

Program evaluations are critical tools for municipal leaders, whether they work for a big city or a small village. Program evaluations allow planners and leadership to determine if and how their projects and investments are meeting the needs of their residents. These evaluations can be deployed to measure both ambitious programs—like revitalizing an entire downtown—or more modest programs, like repaving neighborhood streets.

However, if evaluations fail to take a long-term perspective, it can often lead to challenges for municipalities. That is, taking a short-term approach to studying a program, gleaning a few necessary improvements, and then moving on will not allow municipalities to track and communicate the long-term impact of their efforts. Short-term evaluations fail to understand or plan out the long-term trajectory of a program, and thus can become outdated and unhelpful quickly.

Indeed, program evaluations most often yield real, tangible improvements if they have a long-term approach. If a municipal leader really wants to understand whether that downtown revitalization strategy is working effectively and equitably, or whether the local streets are being maintained in a way that best serves the community, they need to take a long-term approach. By taking a long-term approach to evaluation, leaders can make big changes, like budget redistribution, goal reorientation, process development and new staffing with confidence they will make an impact.

So, how can municipalities ensure that their program evaluations have a sufficiently long-term scope? Here are six strategies:

Dive deep. At the core of any successful long-term program evaluation is a comprehensive audit and data-gathering exercise. Dive deep into all the processes that underlie your program, from budget and spending, to staffing, to decision-making. Take the time to fully understand what makes your program tick. Then, study related processes that exist outside the specific program, but still affect it. For example, big-picture budget decisions that fund the program, or municipal hiring decisions that determine how many people work on it.

Listen to everyone. As you dive deep and gather data, ensure you’re collecting feedback from everyone involved. Municipal leaders shouldn’t be talking just to program leads. Instead, they should be talking to all staff who work on the program, any contractors or partners, and the residents impacted by it. Everyone who is touched by the program should have a voice, as every individual could possess vital insights. In practice, this means talking not just to the head of your Department of Public Works, but also the workers who pave the streets and the residents who drive on them.

Collect the right data by asking the right questions. As your program evaluation unfolds, ensure you’re gathering data that you can actually put to use. The information coming in should always be clear, direct and actionable—every data point should be a pathway for program improvement. Do this by asking clear, direct and actionable questions from the get-go. Determine what kind of queries will get you the most fruitful answers. For example, instead of asking residents, “How do you find the quality of our city streets?” Ask them, “Which city streets are the best and worst to drive on, and why?” or “Have you felt that our street care has improved or worsened over time, and why?”

Begin program evaluations at the right time. Long-term program evaluations require time and energy. For that reason, you should be strategic about when you conduct them. For example, if you are evaluating a program that’s funded by a specific grant, it is often wise to carry out a smaller evaluation at the midway point, in case you need to adjust course. A larger evaluation may lead you to changes outside the scope of the grant. Take on the broader evaluation at a time when your funding is less restricted, and your team can think more creatively about changes and transformations.

Pay attention to metrics… A key element of long-term program evaluations is meeting your goals. Goals are measured by metrics. That is, setting objectives at the start of the evaluation, and then determining whether or not you met those objectives. Work with your community to set those objectives and ensure they’re accurate. For example, if you are evaluating a downtown revitalization project, one objective may be whether a certain number of new businesses opened in a 12-month time frame. Or if you’re repaving neighborhood streets, an objective may be determining if complaints were reduced to zero.

… but understand that metrics are not law. Again, metrics are important, but they’re not carved into stone. As you learn more about your program throughout the evaluation process, you can update your goals and thus your metrics. That’s part of how you improve the program. For example, you may learn that it’s more important to measure not how many new businesses opened, but rather how long new businesses stayed open for.

Veteran municipal leaders know that program evaluations never truly end. They’re not something that should be conducted once in a blue moon. Rather, they should be ongoing so you’re constantly optimizing the work you’re doing. Using the strategies above, you can ensure each program is being evaluated thoroughly and running like a well-oiled machine. Over time, your community will develop a constant evaluation mindset that allows you to see and make improvements without the structure of a formal evaluation. This is true long-term thinking and will set your community up for success moving forward.

Celeste Frye, AICP is co-founder and CEO of Public Works Partners LLC, a WBE/DBE/SBE certified planning and consulting firm specializing in multi-stakeholder initiatives and building strong connections across the government, nonprofit and private sectors. For more information, visit www.publicworkspartners.com.