Over Interpreted Data 2

My second example of over-interpreted data (that is, facts that do not support the inferences that people attempt to draw from them) is from the Campaign for Science and Engineering in the UK Policy Document Number 8, May 2008, Delivering Diversity: Making Science & Engineering Accessible to All (available as a pdf). The statement is:

‘A 2005 survey of STEM workers in Higher Education, found that 41% of men had been interviewed by all male panels compared with 27% of women. This illustrates the importance of making sure that interview panels are appropriately diverse.’

The first question might be 41% of what and 27% of what? It makes a difference whether it is 7/17 = 41% and 3/11 = 27% or 410/1000 and 135/500.

The figures appear to be taken from the Athena Project Report Number 26: ASSET 2003: The Athena Survey of Science Engineering and Technology in Higher Education (available from www.athenaforum.org.uk/reports), which states, referring to appointment to a first lectureship:
‘

Of those appointed in the last two years, 35% of the men and 26% of women had been appointed by all male interview panels, an improvement on the past:
• 3 to 5 years ago - 41% of men and 28% of women were appointed by all male panels
• 6 to10 years ago - 55% of men and 28% of women were appointed by all male panels.’

Table A8 of the Appendix to Report 26 - Statistical Tables gives the numbers of those who reported having an all male interview panel by grade and gender and Table 4 gives the total numbers of respondents to the survey by grade. From these we can establish that out of 1,512 male survey respondents 606 (40%) reported an all male interview panel and out of 660 female survey respondents 200 (30%) reported an all male interview panel. These numbers show that it is statistically very unlikely that the categories (male, female) and (reported an all male interview panel, did not report an all male interview panel) are independent. There are at least four possible explanations for this observation:

1. All male interview panels exhibit bias in favour of men.
2. Women are likely to be put off accepting positions if they meet no or few women during the recruitment process.
3. Women perform less well at interview if there are no women on the interview panel.
4. Women are not uniformly distributed among different subjects (see Table 3 in the Statistical Tables). Subjects with few women may be more likely to have all male interview panels and a preponderance of male appointees.

In options 1-3 an all male panel affects the selection process. In option 4 the association of male interview panels with male appointees is a result of a common factor, namely the relative numbers of women in different subjects. Unfortunately the data as presented in the Athena Report do not allow us to distinguish between these possibilities.

My point here is not that the composition of interview panels is not important. My point is that the data quoted do not, as is claimed, illustrate that importance. In fact, I have not been able to find any data relating to the effect of the composition of interview panels on the outcome of the appointment process. I found numerous exhortations to ensure that interview panels are appropriately diverse but no evidence that this makes any difference. (I tried Google and Google Scholar. If anyone knows where such data are to be found, please would you let me know?)

The data on unconscious bias show that it depends on the gender of those being evaluated not on the gender of those doing the evaluating (see Why so slow? The advancement of women by Virginia Valian,MIT Press 1999 ) It might be that having a women on the interview panel reduces the effect of unconscious bias but if that woman has had to be imported from another department it may reinforce it.

There is evidence that women are put-off by a recruitment process where the interview panel is all male or they meet no or very few women while visiting the prospective employer, at least in the graduate recruitment process (S.L. Rynes, R.D. Bretz, B. Gerhart (1991), "The importance of recruitment in job choice: A different way of looking", Personnel Psychology, Vol. 44 pp.487 - 521. digitalcommons.ilr.cornell.edu).

Irrespective of whether there exist data showing that the gender composition of interview panels affects the outcome of recruitment, if we are genuinely trying to attract a diverse workforce we would ensure that both interview panels and the people that candidates meet during the recruitment process are visibly diverse. Surely we would want the people coming to interviews, our potential colleagues, to feel welcome and to see that our jobs are not restricted to white men (or, in the non-academic context, white women).