More on Quantitative vs Qualitative Evidence

One perceived disadvantage of qualitative studies is that analyzing the data, for example, responses to interviews, is inevitably done within some particular  interpretative framework. In fact, quantitative data is also interpreted within some framework but it is less usual to state what it is.

For example, the Asset 2010 survey of academic staff in SET departments found that 17% of women compared with 14% of men had no provision for appraisals, 9% of women compared with 7% of men could have an appraisal on request, and 74% of women compared with 80% of men had appraisals as a matter of routine. These differences are statistically significant.

There are a number of ways of interpreting this:

1. There are departments which discriminate against women by providing men with appraisals but not women.
2. Women are not uniformly distributed across disciplines. Perhaps some disciplines are more likely to operate appraisal schemes than others.
3. Perhaps men in departments with routine appraisals were more likely to have been encouraged to complete the survey.
4. The appraisal process itself discriminates against women so departments with routine appraisals are less likely to attract and retain women.
5. If men are more likely to receive routine appraisals, it is they who are discriminated against since they have to spend time filling in appraisal forms while their female colleagues get on with their jobs. (Presumably the slightly over a quarter of academic staff who reported that they did not find their appraisal to be either useful or valuable might concur with this interpretation.)

My point is not that any of these interpretations is particularly likely. It is that the same quantitative data can be viewed in different ways depending on your perspective. Those who are keen to establish that women are hard done by might incline to interpretation 1. Those who believe that universities are meritocracies and hence fair to all might incline to interpretation 2 or interpretation 3. Those with a less than positive experience of appraisal might incline to interpretation 4 or interpretation 5.

What then should the response be to findings such as these? The standard scientific response of wanting to eliminate incorrect explanations in order to isolate a single best explanation has the disadvantage that it could mean that action to correct an unfair situation is delayed, possibly for years. The other problem with this response is that the holders of strong views, wherever they lie on the spectrum between 'an appraisal is a good thing and everyone should have one whether they want it or not' and 'appraisals are a bureaucratic waste of time foisted on us by HR', rarely base their beliefs on rational analysis of evidence. Consequently finding more evidence is unlikely to change their minds. The standard top-down response of asserting one interpretation to be correct and labelling anyone who disagrees as 'obstructionist' or 'a dinosaur' has the disadvantage of being ineffective. Academics either ignore top-down initiatives or find ways of getting around them. My own view is that, at around a quarter, the proportion of academic staff finding their appraisals to be neither useful nor valuable is unacceptable high. The course of action I would favour is:

•  Explicitly recognise that people have different beliefs and experiences. Expecting someone whose experience of appraisal has been negative to be enthusiastic about a new appraisal policy is counter-productive.
•  Articulate what  a departmental staff appraisal policy is supposed to achieve
•  Ask staff for suggestions for how best to achieve it
•  Give staff feedback on their suggestions. (Too often, people make suggestions only for them to apparently disappear. For example, a suggestion may be perceived as being too resource intensive to implement. If the person making the suggestion is given this feedback they may well be able to think of ways of achieving the same result more efficiently.)
•  Formulate a policy that is grounded in reality and recognises the constraints on people's time. (In an ideal world everyone would attend training courses in how to appraise/be appraised. In real life they do not, unless they are compelled to do so, in which case they turn up, resent being there and don't learn anything.)
•  Monitor your procedures, not just by ticking off whether everyone has completed an appraisal but by seeking feedback on whether the procedures are achieving the desired results.

Disclaimer: My own experience of appraisal has been positive, though personally I would not rate appraisal as having been particularly useful to my career development, which may have more to do with my moves between New Zealand and the UK than the process itself.