How Sausage Is Made

The Philosophical Gourmet recently sent around its draft list of faculties for its 2006 rankings.

These lists will then be shown to professors who will be asked to rank "faculty as a whole", and in some cases faculty by area of expertise, according to the following scale:

5 - Distinguished
4 - Strong
3 - Good
2 - Adequate
1 - Marginal
0 - Inadequate for a PhD program

On the PGR website, I looked under the section "Methods and Criteria", to find the criteria by which these numbers are to be assigned, and yet I couldn't find any. It's not even clear whether 'distinguished' means in fact or by reputation. (Elsewhere on the site the surveys are referred to as 'reputational surveys', so you draw your own conclusion. Presumably, also, being merely 'good' is by reputation only.)

The evaluators in ancient philosophy for 2004-6 are apparently (or at least they are so identified):

Julia Annas
Sarah Broadie
Dan Devereux
Brad Inwood
Lindsay Judson
M.M. McCabe
Stephen Menn
David Reeve
Chris Shields
Gisela Striker

Given that so many students in philosophy consult the Leiter rankings, I wonder if these scholars, and the other evaluators, would be willing to publish the reasons for their rankings, for instance, declaring the books and papers which they have studied with sufficient care, which were written by the professors of the 50+ faculties they are asked to evaluate, and then the reasons why the work of one scholar is better (generally?) than that of another. (Of course we acknowledge that that sort of judgment is only one factor contributing to their judgment on the faculty 'as a whole'.)

The section on 'methods and criteria' contains, at the end, this passing acknowledgment of the importance of independence:

The school from which the evaluator received the PhD (or equivalent) is listed in parentheses. Note that faculty were not permitted to evaluate their own department or the department from which they had received their PhD.

Thus it is recognized that the evaluators must be independent. But it seems unreasonable to hold that 'independence would appear to be impaired' (the term of art in the code of professional conduct for accountants) only through these two associations. Wouldn't other such things as collaboration, friendship, prior assistence given or taken, etc. also 'appear to impair independence'--even more so?

One might object that no judgment could be thus independent. Not so. But, in any case, that sort of independence is what we should look for in a judgment we expect to be accepted simply on authority, when the grounds for the judgment are not publicly supplied or defended.