Evaluation Research Report:

PURPOSE

The purpose of this research study was to determine how the video "Packrat" might be improved. I used the comments of two focus groups to gauge audience reaction to the video. It was my hope to find commonalities within the group responses that could be applied to a revision of the project. By utilizing these audience suggestions, I believe that the new version of the video will remain true to the intent of the piece while being more responsive to the expectations of the targeted audience.

QUESTIONS AND QUESTIONING PROCEDURES

In developing the questions, we first watched the video and discussed what kinds of questions would achieve our goal of getting feedback to improve the video. We then threw out a bunch of questions and wrote them up. We discussed how many questions to ask and thought about the length of time each could take. We discussed the relevance of the questions and we edited them. We wound up with eleven questions for the first group and added two additional questions and removed one for the second group.

Because one of the groups was contacted by mail, questionnaires were the only way to accomplish the goal. In the second group the opportunity arose to get more specific feedback in person.

All of the questions are aimed at determining story and structure. They go from broad to specific. They are not numbered in order to prevent the response of mentally ranking them and creating relationships between them.

SELECTION OF PARTICIPANTS

Intended audiences were:

-People who hoard and use support groups on the web or in person

-Users of the Obsessive Compulsive Foundation web site

-Attendees of the annual Obsessive Compulsive Foundation conference.

-Educators and students in media ethics, documentary theory, or psychology

-Anyone who comes into contact with a person who compulsively hoards

I will first state that videos and questionnaires have been distributed to all of these groups, but many are unable to respond within the timeframe to be included in this analysis. The groups that will be included in this analysis fall under the categories “Educators and students in media ethics, documentary theory, or psychology” and “Anyone who comes into contact with a person who compulsively hoards.”

To recruit educators and students, we used 'The List' and 'On Location' techniques. That is, we recruited Emerson College graduate students and faculty from the pool we had readily available at the college. While there was no official "list" we knew our primary recruits attended Emerson. We also recruited their friends outside the organization by using the 'Snowball Sample', and having participants bring friends along.

The second group is the group of participants who where queried when they watched the video in accordance with the project’s Human Subjects Review proposal. This meant we did not use a ‘Single-Category Design’ which I’ll come back to in the section on Limitations.

THE INTERVIEW ENVIRONMENT

The interview environment was not consistent between the two focus groups. One was conducted in a school classroom with seats arranged as in an auditorium which probably did not foster as much discussion as it could have. The moderator stood in front of the group. Participants did not learn each other’s names.

The second group consisted of individual survey questionnaires filled out in participants’ homes.

I would call both scenarios comfortable but structured by the questions themselves.

HOW THE INTERVIEW WAS ACTUALLY CARRIED OUT

For the first group, questionnaires were handed out prior to viewing the video. At the end of the video, specific faculty members responded verbally while others in the group filled out questionnaires, then participated verbally. There was some opportunity to query the viewers further.

For the second group, questionnaires were provided in advance as well. There was some opportunity for discussion and clarification in the form of phone conversations. It’s impossible to say whether participants’ in this group were hot or cold, or hungry or thirsty, but I feel I can assume they were relatively physically comfortable in their homes at the time of viewing.

The first group yielded about 15 questionnaires with discussion. The second group resulted in closer to 8 questionnaires at present.

ANALYSIS PROCEDURES

There was no gauge of homogeneity or screening, and no pre-questionnaire. We did not ask income, age, gender, or ethnicity screening questions.

Question 1: Does the story make sense?

This question gets right to the heart of the matter. Respondents in both groups universally agreed that the story was fragmented and needed focus. Characteristic responses were, “Seems like multiple stories. Pick one” and “It doesn’t tell a story as much as I’d like, but rather give more of a general overview.” Respondents in the first group answered no to the question and respondents in the second group answered yes but both groups qualified their answers in the same way.

Question 2: Whose story do you think it is?

Characteristic responses went in two directions. Either people answered that it was about Packratting in general or they thought it was about me and/or my father. Group One was split three ways between my story, my father’s story, and a general exposition of the condition. “The filmmaker’s story but I think it could be more defined.” In Group Two half the people answered “our story.”

Question 3: Do you understand who all the characters are and their relationships to each other? If not, where were you confused?

Everyone in both groups ha d a problem with the lack of chyrons but felt that would fix it. In Group One, two mentioned specifically a problem with the “expert” credibility. “Some I understood as your family—others I had no idea where they were coming from or what their expertise really is.” Everyone in this group had problems identifying family members.

Question 4: Is it confusing in other ways?

In Group One, two people left this blank. Two others said to cut the middle out with the “Man on the Street” style interviews. Others said it was choppy and disjointed and repetitious. “Repeats repeats repeats.” In Group Two the answers were more varied.

Question 5: How do you feel about “Packrats” after watching the movie?

In Group One the responses ranged from an analysis of the problem, to sympathy, to intrigue. Some said they learned a lot, such as “ I learned that it is a condition that can be triggered by medical complications” or “I learned a lot-but the movie feels ‘packrat’ed’ LET GO OF SOME OF YOUR FOOTAGE-THROW IT AWAY.” This is where the responses seemed to get emotional and where a couple of people admitted they had tendencies. In Group Two answers also ranged from what people learned, to sadness, to sympathy.

Question 6: What do you thing was the main point of the movie?

This question seemed easier to answer. In Group One, four people thought it was about living with my Dad. Ten people thought it was about “Packratting” as an issue although some of those were still unsure. “I don’t really know. My guess is that the movie is trying to say that this is a really hard, sad problem.” Others didn’t answer. The general complexion of this question was similar in the second group.

Question 7: What themes did you identify?

Here is a breakdown of the numbers of identified themes.

Family 7

Mental illness/health 5

No themes 4

Loss 4

Repetition across generations 4

OCD 3

Fear of lack of control 2

Confusion 2

Isolation 2

Difficulties it causes

How it can occur

Hoarding

Social/economic repercussion

Obsession

Experts/science

Depression

Perfectionism

Intelligence/education

Frustration

Cleaning up

Question 8: Is there a part that seems irrelevant?

It seems to make sense to do a numerical breakdown here too.

Hallway people 6

Repeated ideas 3

Repeated clips 2

Social worker 2

Disaster master 2

National Geographic guy 2

All experts 1

Question 9: Does it drag in any place and if so, where?

This is a slightly different breakdown, but also useful to see in numbers

Hallway people 3

Strokes 3

Disjointed, doesn’t drag 3

No answer 2

Repeated ideas 2

Repeated clips 2

Clinical explanations 2

All talk

Beginning

Question 10: What parts seem to work the best?

As above, I’ll do it by the numbers and add some emotional responses this time.

Action. “Action/animation is why they call it a “MOVie.”

Family/personal 7

Packrat Environments 5

Kris 5 “I think you, YOU analyze your father’s problem-makes for an interesting story.”

Packrat Man in workshop 4

Disaster master 3

No answer 3

Therapists 2

Stills 2

Question 11 (Group Two only): How did you feel about how you were represented in the movie?

Expected more than 20 seconds after 4 hours of interview

Accurately

I look so upset!

Third cut redundant and unnecessary

I wasn’t really in it and that’s totally fine

Question 11 (Group One only): How do you feel about how the issue was presented in the movie?

This question resulted in a lot of repetition about the lack of structure.

Question 12 (Group One only): Do you think this works as an educational piece?

Yes 8

Maybe 6 “I think you should focus on it as a personal piece. Viewers will still get an education but it will be more powerful.”

No 2

RECOMMENDATIONS

Based on the results here are some recommendations:

-Add chyrons.

-Define a topic or a story line and remove material extraneous to it.

Some possibilities include:

· A personal angle

· A clinical angle

-Make the story personal. This is a personal story and most if not all of the clinical and third person stories are extraneous and should be removed.

-Concentrate on the videomaker and her family’s stories.

LIMITATIONS

In doing only two focus groups I did sacrifice homogeneity by comparing a group of participants to a group of students/general audience. I was unable to achieve a 'Single-Category Design'. As participants I would expect certain biases to surface in Group Two.

The two groups are also different sizes and the questionnaires were distributed differently. Group Two is presently half the size of Group One although questionnaires continue to dribble in.

Questionnaires from Group One were missing pages. I ran out of staples and should have made an additional announcement to remind people there were three. A bathroom break coincided with distribution of the questionnaires so that would explain some of it. An additional problem with Group One was that the faculty advisors spoke before the rest of the group filled out their questionnaires, and I sometimes felt the group was parroting those faculty members.