Do Our Policies "Work"? How Can or Should We Change Them? - My Thoughts on Evaluation and Change

Finally, we have reached the last stage of the policy process: evaluation. Once a policy has been passed and implemented, we generally want to know if the policy is successful (This is probably what you were interested in learning when you took this class). We have to develop measures to determine whether or not the policy is meeting its goals, if it was the best way to address a problem, and if we need to change our policy in some way. Evaluation and policy change are intimately tied. After all, it is hard to justify continuing to pay for an ineffective policy.

Like everything else in the policy process, program evaluation is harder than it seems. Generally, evaluations of public programs show that the program has no effect on targets. This may be because the program is ineffective, but it may also be due to the difficulties of measuring program success. Hopefully those of you who have taken a research methods class in the social sciences recognized some of these problems.

Peters starts with goals. He's discussed the goals of policy as problematic throughout the policy process. If you remember, in chapter four we learned that there are incentives to make goals vague and even contradictory in order to get bills passed by the legislature. When this happens, organizations who implement programs and analysts attempting to evaluate them do not know which goals are most important. Establishing success is impossible when successfully meeting one of the goals means failing to meet the others.

We also learn about the challenges of measurement. What time span should we use to show success or failure? Are participants reacting to our study? Would these participants have done just as well without the program? Which program is really affecting recipients of multiple programs? Are there unintended consequences of this program? How valid are the measures we have established? These are all questions that a good evaluation should ask, but some are nearly impossible to answer. They all affect the determination of whether or not a program is effective. Often, if we change a measure, the time span, or the method of participant selection we see completely different results. Next time you hear that policy X is a huge success or failure, I want you to remain skeptical and think about how the evaluators of the policy made that determination.

On a related note, we must always remember that political factors and values are affecting evaluation. Most pilot programs require some work in order to participate so they tend to attract the "best" of the eligible group. Often this may not be the group we actually wish to target. Organizations, policy analysts, and policymakers have self-interested reasons for wanting their programs to do well, and while they often do not intentionally bias their results, they may be gearing their studies to show programs in a positive light (of course, the converse is also true). Political partisans may be the most likely to seek biased results for political gain and will manipulate the evaluation process.

Government has become increasingly concerned with program evaluation since the 1990s. Evaluations using quantitative data with easy to cite statistics are considered particularly valid. Presidents have made various attempts at creating cross-program measures of efficiency and effectiveness to determine which programs to change and which to expand. One has to think about what types of programs will most likely show consistent results and which present more challenges. It is much easier for NASA to collect data and quantify the results of its activities than the public health department, for example. NASA will be much more likely to show outcomes (the effects of their research), while the health department will have to rely on illustrating inputs (funds) and outputs (the services they deliver).

Policy change can be thought of as the beginning of the policy stages cycle all over again. Based on evaluations, policymakers have to decide whether to maintain, terminate, or change policies. When government takes the time to evaluate and re-consider policies, they are rarely terminated or left in their exact same state. That being said, policy maintenance has become increasingly likely as Congress has struggled to pass meaningful legislation that seriously debates and considers policy. Even when programs have sunset provisions, they often are maintained because of a lack of political will to either end or change the program. For example, I worked for an organization in DC that was founded in 2000 to influence the reauthorization of the Workforce Investment Act, which was supposed to take place in 2003. Although there were many proposals over the years for WIA reauthorization, the program has been maintained without any changes for eight years beyond its reauthorization date. This happens to programs all of the time.

As Peters states, programs are rarely terminated. It is difficult for government to terminate a program once advocates, agencies, clients, and businesses benefit from their existence.

Policy change is generally the ideal option for policies, and tends to be the most likely outcome. We want policies to be changed as context and evaluation dictates. Sometimes changes are for the good and make the program better, but often change has just as many downsides. Peters discusses four types of change: linear succession, consolidation, splitting, and nonlinear succession. He gives examples of each. The consolidation of programs into the Department of Homeland Security is a good example of the pros and cons of policy change. It does seem as though consolidation has helped organizations and agencies in the sharing of information and anti-terrorism actions. On the other hand, FEMA has become a much less effective agency. Now that it is no longer independent, it is fighting for agency resources to deal with relatively common natural disasters in a department that is concerned primarily with terrorist threats. There is little reason to believe that advocating for FEMA is at the top of the Secretary of Homeland Security's priority list when it is often dealing with issues removed from the rest of DHS's mission. This is one of the consequences of the reactivity of policymaking in the United States. We often change policies in reaction to a visible, large-scale event that is judged to be a policy failure. In doing so, we may be setting the stage for the next policy failure in another area. Although there are many explanations of causation for the 2005 Katrina disaster, FEMAs consolidation into DHS certainly can be considered a contributing factor.