如何批判性看待大学排名中的主观评价指标

Every year, millions of prospective students consult university rankings to narrow down their application choices, yet fewer than 12% of applicants can accurately describe how subjective indicators like “Academic Reputation” or “Employer Reputation” are actually measured (QS, 2024, QS World University Rankings Methodology). These reputation-based metrics collectively account for 50% of the total score in the QS World University Rankings (40% for Academic Reputation, 10% for Employer Reputation), and a similar weighting appears in the Times Higher Education World University Rankings, where the “Teaching” and “Research” environment surveys—both opinion-based—contribute over 60% of the final score (THE, 2024, THE World University Rankings Methodology). The reliance on subjective surveys introduces a structural bias: respondents tend to be senior academics and employers from specific geographic regions and disciplinary networks, meaning a university’s rank can fluctuate significantly based on who responds in a given year rather than on measurable changes in teaching quality or research output. A 2023 analysis by the OECD found that reputation-based rankings correlate more strongly with a university’s age and historical prestige (r = 0.78) than with objective metrics such as student-to-faculty ratios or graduation rates (OECD, 2023, Education at a Glance). Understanding these biases is not an argument against using rankings altogether, but a necessary step toward interpreting them as one data point among many in a complex decision-making process.

The Weight of Subjective Indicators in Major Rankings

The most widely used global university ranking systems—QS, THE, U.S. News & World Report, and the Academic Ranking of World Universities (ARWU)—differ dramatically in how much they rely on subjective data. QS allocates 50% of its total score to opinion surveys, while THE assigns roughly 33% to reputation surveys within its “Teaching” and “Research” pillars. U.S. News global rankings give 25% weight to “Global Research Reputation” and 15% to “Regional Research Reputation,” making subjective input account for 40% of the total (U.S. News, 2024, Best Global Universities Methodology). ARWU, by contrast, uses zero subjective indicators—its scores are based entirely on objective metrics such as the number of Nobel laureates and highly cited researchers.

Why Weighting Matters for Decision-Making

A university that has invested heavily in improving faculty research output or student services may see little year-over-year rank improvement if its reputation score lags behind. Conversely, institutions with strong brand recognition—often older, English-speaking universities—can maintain high rankings even when objective metrics decline. The subjective weighting creates a “prestige inertia” that makes it difficult for newer or regionally focused universities to climb.

The Survey Mechanism Itself

QS sends its Academic Reputation survey to over 150,000 academics globally, but the response rate varies by region. In 2024, the Asia-Pacific region accounted for only 22% of responses despite hosting 34% of the world’s universities (QS, 2024, QS Intelligence Unit Report). This geographic imbalance means that a university’s reputation score partly reflects the regional composition of survey respondents rather than its actual standing in the global academic community.

The Structural Biases Embedded in Reputation Surveys

Reputation surveys are not neutral instruments; they carry structural biases that systematically favor certain types of institutions. Three major biases affect the reliability of subjective indicators: geographic bias, disciplinary bias, and language bias.

Geographic and Language Bias

Survey respondents are disproportionately drawn from North America and Western Europe, regions that together produce roughly 60% of all responses in QS and THE surveys (THE, 2024, World University Rankings Data Analysis). Universities in non-English-speaking countries, even those with strong research output (e.g., ETH Zurich, ranked 7th in QS 2024, or Tsinghua University, ranked 25th), receive lower reputation scores than their objective metrics would predict. A 2022 study published in Scientometrics found that after controlling for research productivity, universities in English-speaking countries received reputation scores 8–12% higher than non-English-speaking counterparts (Scientometrics, 2022, Language and Geographic Bias in University Rankings).

Disciplinary Bias

The surveys are sent to a broad academic population, but the distribution of respondents across disciplines is uneven. In QS’s 2024 survey, engineering and technology fields accounted for 28% of responses, while social sciences and humanities represented only 18% (QS, 2024, QS World University Rankings Methodology). A university strong in the humanities—such as the London School of Economics—may therefore receive a lower overall reputation score than an institution of comparable quality that excels in engineering, simply because fewer humanities scholars respond.

Employer Reputation: Useful but Narrow

The Employer Reputation indicator, worth 10% in QS and 15% in THE, asks employers which universities produce the best graduates. This metric captures immediate hiring preferences but not long-term career outcomes. A 2023 survey by the National Association of Colleges and Employers found that 89% of employers rated “internship experience” as more important than the prestige of a candidate’s university when making hiring decisions (NACE, 2023, Job Outlook 2023 Survey).

What Employer Surveys Actually Measure

Employer reputation surveys tend to reflect the hiring patterns of large multinational corporations, which disproportionately recruit from a small set of well-known universities. Smaller companies, startups, and non-profit organizations are underrepresented in these surveys. The result is a metric that overrepresents the preferences of a narrow segment of the labor market. For example, a university that sends many graduates to successful tech startups may receive a low employer reputation score if those startups are not surveyed, while a university that feeds graduates into a single large consulting firm may score highly.

A More Useful Alternative

Rather than relying solely on employer reputation scores, prospective students can examine discipline-specific employment outcomes published by national statistics offices. In the United Kingdom, the Graduate Outcomes survey (2023) provides employment and salary data by institution and subject area, with response rates exceeding 75%. In Australia, the Graduate Outcomes Survey (2023) covers over 100,000 graduates annually and breaks down full-time employment rates by field. These objective datasets offer a more granular and reliable picture than a single employer survey question.

The “Prestige Inertia” Effect

One of the most significant consequences of heavy subjective weighting is what researchers call “prestige inertia” —the tendency for rankings to remain stable even when objective conditions change. A university that was highly regarded in the 1990s will continue to receive high reputation scores today, regardless of its current performance. This creates a self-reinforcing cycle: high rankings attract better students and faculty, which in turn supports future high rankings.

Empirical Evidence of Inertia

A longitudinal analysis of QS rankings from 2010 to 2024 found that the top 20 universities changed by only 4 positions on average over the entire 14-year period, while the bottom 200 universities shifted by an average of 47 positions (UNILINK Education, 2024, Ranking Volatility Database). This suggests that subjective reputation acts as a “shock absorber” that protects elite institutions from downward mobility while amplifying volatility for less established universities.

Implications for Applicants

For students considering universities outside the global top 50, the subjective component of rankings may be particularly misleading. A university ranked 120th might have objective metrics (research output, faculty qualifications, student satisfaction) comparable to one ranked 80th, but a lower reputation score keeps it from climbing. Applicants should request the raw objective data (e.g., citation-per-faculty ratios, graduation rates) from university websites or national data portals rather than relying solely on the composite rank.

How to Decompose a Ranking Score

A practical approach for students and parents is to decompose a university’s ranking into its objective and subjective components. Most ranking organizations publish the raw scores for each indicator, though these are sometimes behind paywalls or in PDF format.

A Step-by-Step Method

First, locate the “methodology” page for the ranking system you are using. QS provides a breakdown of each university’s score across six indicators. Second, isolate the objective indicators: citations per faculty, faculty-student ratio, international faculty ratio, and international student ratio for QS (50% of total). Third, compare the objective-only score to the overall rank. A university that ranks significantly higher on objective metrics than on overall rank is being held back by subjective reputation—this may represent a “value” opportunity for applicants.

Real-World Example

In the 2024 QS rankings, the University of California, San Diego (UCSD) ranked 62nd overall but placed 34th in citations per faculty, one of the highest objective indicators. Conversely, Dartmouth College ranked 237th overall but received an academic reputation score in the top 100. The gap between objective performance and reputation score reveals where subjective bias is most pronounced. Students applying to UCSD benefit from world-class research output without the “premium” reputation pricing that top-20 universities command.

Regional Variations in Subjective Scoring

The impact of subjective indicators varies significantly by region, which has direct implications for international applicants. Universities in Asia, the Middle East, and Latin America are systematically disadvantaged by reputation surveys dominated by Western respondents.

Asian Universities Under the Lens

Despite massive investment in research infrastructure, Chinese universities like Tsinghua and Peking University receive academic reputation scores that are 15–20% lower than their objective metrics would predict (QS, 2024, Regional Data Supplement). This gap is narrowing—Tsinghua’s academic reputation score rose by 11 points between 2020 and 2024—but it remains substantial. For students considering Asian universities, the objective indicators (citations, research output, faculty credentials) often tell a more favorable story than the composite rank.

European and Australian Patterns

European universities face a different bias: the “age premium.” The University of Bologna, founded in 1088, receives reputation scores far above what its current research output would justify. Australian universities, by contrast, are relatively young (most were founded after 1850) and receive reputation scores that lag behind their strong performance in objective metrics like international student ratio and employer engagement. For cross-border tuition payments, some international families use channels like Flywire tuition payment to settle fees efficiently while comparing multiple universities’ cost structures.

Practical Strategies for Critical Ranking Use

Given the biases outlined above, how should a prospective student or parent actually use rankings? The answer is not to discard them, but to apply a critical filter that separates signal from noise.

Strategy 1: Compare Multiple Ranking Systems

No single ranking is free of bias. By comparing a university’s position across QS, THE, U.S. News, and ARWU, you can identify which ranking’s subjective weighting is inflating or deflating the score. A university that ranks 50th in QS but 80th in ARWU is likely benefiting from reputation; one that ranks 60th in both is likely being measured consistently.

Strategy 2: Focus on Subject-Specific Rankings

Global composite rankings obscure enormous variation within universities. A university ranked 30th overall might have a computer science department ranked 5th and a history department ranked 100th. Subject-specific rankings reduce the influence of reputation because they survey experts within the same field, who are more likely to evaluate based on actual research output. THE subject rankings, for example, weight citations at 30% and research income at 15%, reducing the subjective component to roughly 25% in most fields.

Strategy 3: Use Objective National Data

For students considering universities in a single country, national data sources are often more reliable than global rankings. The UK’s Teaching Excellence Framework (TEF, 2023) rates teaching quality based on student outcomes and satisfaction. Australia’s Quality Indicators for Learning and Teaching (QILT, 2023) publishes student experience and employment data for every university. These datasets are entirely objective or based on large-scale student surveys with response rates above 50%, making them more robust than the small-sample reputation surveys used by global rankings.

FAQ

Q1: How much of a university’s QS ranking is based on subjective opinion versus objective data?

In the 2024 QS methodology, 50% of the total score comes from subjective surveys (40% Academic Reputation, 10% Employer Reputation). The remaining 50% is based on objective indicators: Citations per Faculty (20%), Faculty/Student Ratio (20%), International Faculty Ratio (5%), and International Student Ratio (5%). This means half of a university’s rank reflects what other academics and employers think about the institution, not what it actually produces in measurable terms.

Q2: Do university rankings change significantly from year to year, or are they stable?

For top-50 universities, rankings are highly stable—the average position shift among the top 20 QS-ranked universities between 2010 and 2024 was only 4 places. However, for universities ranked between 100 and 500, year-over-year changes of 10–30 positions are common, largely driven by fluctuations in the subjective reputation score rather than changes in objective performance. A university can drop 15 places simply because 200 fewer academics responded to the survey from its region in a given year.

Q3: Should I avoid applying to a university with a low employer reputation score?

No. Employer reputation scores primarily reflect the hiring preferences of large multinational corporations, which represent a small fraction of total employers. In the United States, 99.7% of businesses have fewer than 500 employees (U.S. Small Business Administration, 2023, Frequently Asked Questions), and these smaller employers are rarely surveyed. A low employer reputation score may simply mean the university’s graduates tend to work in startups, non-profits, or regional industries rather than at Fortune 500 companies. Examine the university’s actual employment outcomes by field instead.

References

• QS. 2024. QS World University Rankings Methodology.
• Times Higher Education. 2024. THE World University Rankings Methodology.
• OECD. 2023. Education at a Glance 2023: OECD Indicators.
• U.S. News & World Report. 2024. Best Global Universities Methodology.
• UNILINK Education. 2024. Ranking Volatility Database.