Collecting Data for Analysis

Introduction

Collecting data for analysis is extremely important to our efforts as performance improvement professionals. Our goal is to understand the problem as it exists, and to understand where we want to go so that we can complete a Gap Analysis. This requires facts... not opinions, facts. Usually when we begin a project we are confronted with opinions about what the problem is, and what the solution should be. We can choose to use these opinions as a hypothesis - at most.

Once my Alignment meeting is complete and the stakeholders and I agree on the problem, my next step is to collect data for the analysis.

Data Collection Methods

To conduct data collection I typically need access to the following items/people:

1) Extant data (Organizational/Industry Processes, Procedures, Polices, and Guidelines, Org Charts, Personnel data, etc)

2) Subject Matter Expert (SME) (Someone who is considered an expert in the field)

3) Accomplished Performer (AP) (Someone who excels and is performing the task right now)

4) Stakeholders (People who can make decisions regarding the topic)

Of course, there are many ways to collect data. Below are five methods I use depending on the situation I'm in.

METHOD 1: IF I can get SMEs, APs, and Stakeholders in the same room at the same time for a length of time, I conduct a Focus Group. My data collection would go like this:

1) Collect Extant Data

2) Conduct Focus Group

This is the quickest way to collect data and reach conclusions. I've found it to also be the most difficult to set up, and perform. To begin, I ALWAYS collect Extant Data, ALWAYS. I want to walk into that Focus Group ready to speak intelligently about the subject. I need to understand the situation, the politics as well as the processes and procedures. Focus Groups are difficult because it's tough to get all of these key people in the same room and the same time for a period of time. If I manage to get a focus group together, the next difficult part is to facilitate the group.

METHOD 2: IF I can't get those individuals together for a focus group, my data collection would go like this:

1) Collect Extant Data

2) Interview SME

3) Interview AP

4) Conduct Observation

5) Verify with SME/AP

As I said, if I can't get a focus group, the next best thing is to personally interview the SMEs and APs (after collecting Extant Data of course). Once those interviews are complete, it's helpful if I can follow up with an observation of the performance. I find that SMEs and APs are usually not able to fully articulate a process. Once I've documented the process through Interviews, I use observation to figure out the details. Once my observation is complete, meet again with our SMEs and APs to discuss what I learned and/or deconflict anything that differed with what they told me.

METHOD 3: IF there are no Accomplished Performers or Observation, my data collection would go like this:

1) Collect Extant Data

2) Interview SME

3) Survey employee group

4) Verify with SME/AP

As with Method 2, I would begin with Collecting Extant Data and Interviewing SMEs. As I mentioned before, I've found that most SMEs are not able to fully articulate the process for one reason or another. Because SMEs are not performing the task right now, I want to verify what they know with what is currently happening. I take the information learned from the SME interview(s) and create a survey, which I launch to as many employees in the target audience as possible. Information from the SMEs will either support or detract from the original hypothesis. You'll use the employee survey to test your hypothesis. Once again, after the survey is complete I meet with our SMEs  to discuss what I learned and/or deconflict anything that differed with what they told me.

METHOD 4: Sometimes I am not able to interview SMEs OR APs or conduct observation. In that case, my data collection would look like this:

1) Collect Extant Data

2) Survey employee group

If I'm not able to conduct interviews or observation, I will Collect Extant Data and use that to create an employee survey. The survey is used to validate hypothesis I created from analyzing the Extant Data.

METHOD 5: Sometimes I do not have access to survey employees OR conduct an observation. In that case, my data collection would look like this:

1) Collect Extant Data

This is a worst-case scenario. If I only have access to Extant Data, I will make assumptions from that data - and make recommendations based on those assumptions. When this happens we usually end up piloting the intervention and doing the data collection process there. Intervention development becomes and iterative process, which can be time consuming.

Final Thoughts

Coming up with a data collection plan is important to do early on in any project. In my experience, as soon as you've agreed on the problem with the stakeholder, you should immediately begin thinking about how to collect data - and start reserving the needed resources.

You may have hypotheses from the alignment meeting. If you do not already have a hypothesis and  you're doing a focus group, it may be best to guide the key players to an understanding of the problem - this will create a higher level of buy-in on the solution. If you're not doing a focus group, your hypotheses will be developed between the Extant Data collection and your interviews, and tested during observation and surveys.

A common question asked is "how long do I keep collecting data?" My Needs Assessment professor from Graduate School, Dr. Don Winiecki always used to tell us "keep collecting data until you stop hearing new things." In other words, keep interviewing your AP(s) until you've heard all of the variations at least twice (time permitting of course). Once your interviews stop uncovering new details, you are ready to analyze the data and form your hypotheses. Ideally, the survey should be used to test or validate those hypotheses. If you're using a survey to uncover new things, it's because either your interviews were incomplete, or you were not allowed to do interviews in the first place. Surveys are typically a one-shot deal, so they need to be specific/targeted questions, ideally quantitative so it's easier to come to a conclusion about the results.