Program Evaluation

Assignment 2: Program Evaluation

In this assignment, you will be considering a case study and then applying what you have learned in this module about research and evaluation.

Case Study

Beginning in March last year, a year-long media-based campaign was aired on radio and television and in print in British Columbia. Aimed at smokers in the age group of twenty to thirty years, it is intended to influence them in ways that will lead them to stop smoking.

The logic model for the smoking cessation campaign is quite simple and reflects the prevailing views about program theory that are relevant to this kind of campaign. Given below is a part of that logic model, focusing on outcomes. The components, implementation activities, and outputs are not shown, but they focus on the amounts of media coverage on the radio and television and in newspapers. The campaign is intended to last a year and is intended to be implemented in segments. The first segment begins in March and ends in April, the second one is being implemented in July and August, and the third in October and November.

As you can see from the logic model, the campaign is intended to initially reach a group of smokers in British Columbia (awareness of the campaign) and, once they are aware of the campaign, to increase their knowledge of smoking-related issues that might appeal to persons in that age group.

One television advertisement, for example, shows a man standing outside a bar smoking a cigarette. As he inhales, he ages. As he exhales, he becomes younger again. The message is that if you quit smoking, most of the effects on your body are reversible. A website is provided so that viewers can get information to help them quit. Exposure to these messages is intended to affect attitudes toward smoking (to make them develop a view that smoking is not good for them and that they want to change their behaviors). Attitude change is expected to lead to changes in behaviors: attempts to quit, cutting back on smoking, and actually quitting.

The Evaluation

The evaluation of this province-wide program is intended to determine whether the program logic works as intended, i.e., whether people are aware of the campaign, whether the campaign increases their knowledge, whether their attitudes change, and, finally, whether they try or succeed in quitting smoking.

The evaluation of the planned smoking cessation campaign in British Columbia is focused on the effectiveness of the campaign. More specifically, the evaluation is intended to address these questions:

• What impact, if any, did the mass media campaign have on:

  Awareness of the campaign itself.
  Knowledge of the consequences of smoking.
  Attitudes toward smoking.
  Smoking-related behaviors.

The key issue for the evaluation design is to determine how the campaign affected the aforementioned attributes of persons who have self-identified as smokers.

Given that the campaign is province-wide, it will be difficult to find a control group that has not been exposed to the campaign. Given the costs of the campaign, the evaluation will need to focus on British Columbia and not try to find a comparison group in another province.

Population surveys of British Columbia residents suggest that approximately 15%–20% of the people in the age group of twenty to thirty years smoke. These smoking rates vary from one region to another and with sociodemographic characteristics of British Columbia residents.

Tasks:

Design an evaluation report that will be able to assess whether the smoking cessation campaign was successful. An important part of any proposed design is to assess whether the program theory actually worked the way it was intended, i.e., whether persons became aware of the campaign, whether their knowledge increased, whether their attitudes then changed, and whether attitude changes led to smoking cessation behaviors.

To ascertain whether the campaign impacts smokers in the way the program logic is intended, it is important to design an evaluation that actually tracks the way the campaign influences smokers over time. Ideally, the evaluation team wants to be able to see whether the program was visible; if it was, whether knowledge increased; if that happened, whether attitudes changed; and if attitudes changed, whether changes in behavior have been reported.

In a 2- to 3-page paper, explain the following:

• What are the main features of your evaluation design?
• How would it be implemented?
• How does the design deal (if it does) with possible rival hypotheses (internal validity threats)?
• How does the design address statistical conclusions, construct, and external validity?