Posted December 16th, 2009 by Aale Roos

I suppose one way might be to offer better service, but that is not what I am writing about in this column. Improving survey results is easy and people do it all the time. Unfortunately, many do it unintentionally, so that they are not aware that they are manipulating the results.

Let’s assume that you know that your customers are not as satisfied as you would like them to be and you would like to be able to show results that are a little better than what you expect to get. For example, if you are a service desk manager and your bonus depends on the results, you might consider it unfair that you miss your bonus just because your staff is incompetent, wouldn’t it? Hmm, let’s not be too ethical.

There are four basic methods to reach better results. Nothing prevents you from applying them all, but I will list them in the order of importance:

1. Respondent selection
2. Respondent influencing
3. Question design
4. Analysis and reporting tricks

My examples are from a service desk point of view, but actually these methods apply to several service situ-ations.

1 . Respondent selection

Usually it is not a good idea to send the satisfaction survey to everybody. People generally do not like to get surveys often and therefore it is better to take a sample. Selecting the respondents is the best and most effective method of manipulating the results. In an ideal situation you can individually pick the people. The simplest way to do this is to tell your staff to select happy customers only.

Another way is to select only those who have contacted the service desk recently. Remember that they represent the part of the population who still think that the service desk can help them. Those who do not care to contact your desk are left out by this selection. Next to that, it is likely that those who like you best, call you more often. This means that if you are required to pick the respondents randomly, you can still improve the results. A random sample of callers during a short time period will favor the frequent callers. A person who calls you once a week has a much bigger the chance of being in the sample compared to a person who calls once a year.

2. Respondent influencing

Direct influencing means telling the respondents that your bonus or that of the service desk staff depends on how they answer. This seems to work: car dealers also use this method.

A second method is to contact everyone who has given a low mark and ask them for an explanation. This can easily be motivated as a quality improvement initiative. When using proper questioning style, the customers quickly learn not to give low marks. The downside is that this method will affect only future surveys.

3. Question design

A good question guides the customer to the right answer. If your problem is that your staff mainly consists of high school dropouts, a few older employees and one PhD that you have not been able to fire, you could formulate your question like this:

Our staff is highly educated up to PhD level and we have may years of combined work expe-rience. We work under high pressure to solve hundred of complicated problems daily. How would you rate our competence in IT and telecommunications technological matters? 

This question puts the user in their rightful place. Who are they to criticize the pros?

A second part of good question design is distorting the scale by making it one sided. Here is an example of a distorted scale:

5  Extremely satisfied
4  Highly satisfied
3  Quite satisfied
2  Somewhat satisfied
1  Completely dissatisfied

Or if you suspect that your interactive telephone system (Press 1 if you want…)  is not very popular you could formulate the question like this.

Our interactive telephony solution has been in use for one year and it has proved to be enorm-ously efficient in directing calls. For further development, what would you like to us to do:

1  Add new choices to the menu
2  Add new layers to the menu
3  The menu is good as it is.

It would be a surprise if you could not report that 100% of users are happy with the telephone system and X% would even like to add features to it.

4. Analysis and reporting tricks

It is quite easy to manipulate the results by combining answer options. If you used the distorted scale from previous paragraph, you could have a great satisfaction index value by combining options 2-5 as satisfied. Usually you do not see many 1’s in a survey so you should be able to reach 100% satisfaction.

 Let’s assume you had the above distribution on the distorted 1-5 scale. Honestly, it does not look very good and in truth, you do not have a single satisfied customer. But, instead of showing the graph above, you could just report that 97.3 % of customers were satisfied. Notice also the decimal. As you only have a couple hundred answers, you should not give three significant numbers but it looks so much more convincing than plain 97 %.  97.29 % might be even better.

As a final resort, you can work wonders with the scales in a graph. Here is a case. The satisfaction survey has been done quarterly six times and you have been in charge for the last 14 months, so the last four columns starting from period 3 represent your results.

The results look like this.

 

Now the situation does not look very nice. There are still a few tricks that can save the day.

Here I have widened the scale from 1-5 to 0-10. It is meaningless but even University professors do this all the time. It is extremely common to see people reporting 1-5 results with columns starting from 0. Also the 3-D picture helps to distort the data.

This is the best option. Now the old history has been left out so that period 1 is former period 3 and I have used Excel automatic scale so there is no conscious manipulation in that. Doesn’t it look fantastic how you have been able to improve the results? (And yes, all three pictures come from the same data).

Average: Select ratingPoorOkayGoodGreatAwesome

Your rating: None Average: 3.5 (4 votes)