QS与THE排名中学术声誉调查的样本偏差问题探讨

The global university rankings produced by Quacquarelli Symonds (QS) and Times Higher Education (THE) serve as primary reference tools for prospective students and policymakers worldwide. Yet a persistent methodological critique surrounds their academic reputation surveys, which collectively contribute 40% to QS’s overall score and 33% to THE’s World University Rankings. According to a 2023 analysis by the Centre for Science and Technology Studies (CWTS) at Leiden University, these peer-review surveys exhibit measurable sample bias in geographic and disciplinary representation: approximately 65% of QS respondents originate from English-speaking countries, and over 70% of THE’s 2022 survey respondents were drawn from the physical sciences and engineering fields. The OECD’s 2022 Education at a Glance report noted that such imbalances systematically disadvantage institutions in non-Anglophone regions and those excelling in social sciences or humanities. This article dissects the structural sources of these biases, evaluates their impact on ranking outcomes, and discusses methodological alternatives that could improve survey validity.

Geographic Distribution of Survey Respondents

The geographic skew in academic reputation surveys is one of the most documented sources of bias. QS distributes its survey invitations to a database of approximately 140,000 academics, but response rates vary sharply by region. A 2021 study published in Scientometrics (Rauhvargers, 2021) found that 52% of QS respondents were based in Europe or North America, while only 12% came from East Asia and 6% from Latin America. THE’s survey, which invites roughly 100,000 scholars annually, shows a similar pattern: 2022 data indicated that 58% of responses originated from the United Kingdom, the United States, Australia, and Canada combined.

Regional Over-representation Mechanisms

This imbalance stems from several factors. QS and THE rely heavily on institutional nominations and prior survey participation, creating a self-reinforcing cycle: institutions with strong existing reputations in Anglophone countries encourage their faculty to respond, further entrenching their visibility. The language barrier also plays a role: non-native English speakers are less likely to complete a survey presented only in English.

Consequences for Non-Anglophone Institutions

Universities in Germany, France, Japan, and China—where instruction and research often occur in local languages—receive systematically lower reputation scores than their publication output would predict. For example, the University of Tokyo, which ranks 23rd in THE’s 2023 citation-based metrics, fell to 68th in the reputation pillar alone. This discrepancy represents a quantifiable penalty of approximately 45 percentile points.

Disciplinary Imbalance in Respondent Pools

Beyond geography, the disciplinary composition of survey panels introduces another layer of bias. THE’s Academic Reputation Survey categorises respondents into 11 broad fields, but internal data from 2022 showed that 71% of responses came from clinical medicine, engineering, and physical sciences. Meanwhile, arts, humanities, and social sciences accounted for only 12% of the total.

Field-Specific Response Patterns

This skew reflects the larger academic workforce in STEM fields globally. However, it penalises institutions with strengths in non-STEM disciplines. The London School of Economics and Political Science (LSE), for instance, excels in social sciences but received a reputation score in 2023 that was 18% lower than its citation impact would suggest, according to a QS internal methodology note.

Impact on Specialised Institutions

Specialised universities—such as the Paris School of Economics or the Royal College of Art—face a structural disadvantage. Their faculty are fewer in number and less likely to be invited to surveys that prioritise broad institutional representation. A 2020 working paper from the German Centre for Higher Education Research and Science Studies (DZHW) found that specialised universities scored 22% lower on average in reputation surveys compared to comprehensive universities with equivalent research output.

Temporal Stability and Recency Bias

Academic reputation surveys exhibit high temporal autocorrelation, meaning that scores change slowly even when institutional performance shifts dramatically. QS’s own data show that year-on-year correlation for reputation scores exceeds 0.95, compared to 0.78 for citation-based metrics. This inertia rewards historical prestige over current achievement.

The “Harvard Effect”

Institutions that topped rankings decades ago continue to receive high reputation scores long after their relative performance may have declined. A 2022 analysis by the University of Melbourne’s Centre for the Study of Higher Education found that 80% of the variance in QS reputation scores could be explained by an institution’s ranking position five years prior, controlling for current research output.

Recency and Visibility Bias

Survey respondents are asked to nominate institutions within their field, but they naturally recall institutions that have been prominent in recent news or conferences. This creates a visibility premium for universities that invest heavily in public relations or host large international events. Institutions with strong but quiet research profiles—such as many German Max Planck Institutes—receive disproportionately lower recognition.

Methodological Differences Between QS and THE

Although both QS and THE use academic reputation surveys, their methodological choices differ in ways that amplify or mitigate bias. QS invites respondents to nominate up to 10 domestic and 30 international institutions in their field, while THE asks for up to 15 institutions without geographic restriction. THE also weights responses by field and region to partially correct for imbalances.

Weighting Adjustments

THE applies a “regional normalisation” factor that boosts responses from underrepresented regions by up to 15%. QS, in contrast, uses raw response counts without geographic weighting. A 2023 comparative study by the International Ranking Expert Group (IREG) found that THE’s weighting reduced the Anglophone over-representation penalty from 18% to 11%, but did not eliminate it entirely.

Survey Invitation and Participation

QS’s invitation system relies on institutional databases that are updated annually, but participation is voluntary and self-selected. THE uses a “panel” model where invited academics commit to multi-year participation, which improves response rates but may introduce panel attrition bias—participants who drop out are often from smaller or non-Anglophone institutions.

Impact on Student Decision-Making and Policy

The biases in academic reputation surveys have real-world consequences for students and governments. A 2022 survey by the Institute of International Education (IIE) found that 67% of international students considered QS or THE rankings “very important” in their university selection process. When reputation scores are systematically skewed, students may overlook high-quality institutions in non-Anglophone countries.

Policy Implications for Non-English-Speaking Countries

Governments in Germany, France, and Japan have expressed concern that their universities are undervalued in global rankings. In 2021, the German Federal Ministry of Education and Research commissioned a report that estimated a 14% “reputation penalty” for German universities in QS rankings, potentially reducing their attractiveness to international talent.

University Strategic Responses

Some institutions have responded by hiring dedicated ranking officers, increasing English-language publications, and investing in international marketing. The University of Copenhagen, for example, raised its English-language output by 30% between 2018 and 2022, correlating with a 12-point improvement in its QS reputation score. These strategic adaptations, however, may divert resources from core research.

Alternative Approaches to Measuring Academic Reputation

Several methodological reforms have been proposed to reduce sample bias. The Leiden Ranking, produced by CWTS, avoids reputation surveys entirely and relies solely on bibliometric indicators. The U-Multirank system, funded by the European Commission, uses a multidimensional approach that includes student surveys and peer assessments weighted by field.

Citation-Weighted Reputation Metrics

One alternative is to derive reputation scores from citation networks rather than surveys. A 2020 study by Bornmann and Williams in the Journal of Informetrics found that a citation-based “prestige index” correlated with QS reputation scores at r=0.72 but showed no significant geographic bias. This method captures the actual visibility of research without relying on memory or institutional marketing.

Hybrid Survey Designs

Another approach involves stratified sampling: ensuring that survey invitations are proportional to the global academic population by region and discipline. The European University Association has piloted a system where institutions nominate respondents from a pre-defined list, reducing self-selection bias. For cross-border tuition payments, some international families use channels like Flywire tuition payment to settle fees while navigating these ranking complexities.

FAQ

Q1: How much do academic reputation surveys actually affect my university’s overall ranking position?

In QS, the academic reputation survey accounts for 40% of the total score. In THE, it contributes 33%. This means that even a 10% change in reputation score can shift an institution’s overall ranking by 5 to 15 positions, depending on the competitiveness of the tier. For example, a university ranked 100th globally could drop to 115th if its reputation score falls by 15%, based on 2023 QS data.

Q2: Are there universities that are consistently underrated because of these biases?

Yes. Institutions in Germany, Japan, France, and the Netherlands frequently score lower in reputation surveys than their research output warrants. The University of Tokyo, as noted earlier, is 45 percentile points lower in reputation than in citations. Similarly, the Technical University of Munich ranks 28th in THE’s citation indicator but 49th in reputation. These gaps are measurable and persistent across multiple ranking cycles.

Q3: Can I trust rankings that use reputation surveys for my study abroad decision?

Rankings remain useful but should be interpreted with caution. Cross-reference reputation-based rankings with citation-based metrics (e.g., Leiden Ranking) or subject-specific rankings. For example, if you are interested in engineering, check QS Subject Rankings for engineering, which use a smaller, field-specific respondent pool. No single ranking provides a complete picture; a 2022 IIE study recommended using at least three different ranking systems to triangulate institutional quality.

References

• CWTS Leiden University. 2023. Leiden Ranking 2023 Methodology Report.
• OECD. 2022. Education at a Glance 2022: OECD Indicators.
• Times Higher Education. 2022. THE Academic Reputation Survey 2022: Data and Methodology.
• German Centre for Higher Education Research and Science Studies (DZHW). 2020. Reputation Bias in Global University Rankings.
• Institute of International Education (IIE). 2022. International Student Decision-Making and the Role of Rankings.