Ever wonder how the QS World University Rankings work? Or the reasoning behind this? Here's some answers to some common questions regarding the methodology we use.
QS has been producing university and business school research for the past 20 years. During this period, we have introduced innovations such as the use of global employer and academic surveys as a cornerstone of our approach.
In that time, QS research has become highly respected and every year is referenced in roughly 1000 different newspapers, journals and web sites – a Who’s Who of the best media around the world.
Since 2004, QS has been producing the World University Rankings. They have become the most widely used basis for comparing universities across borders, with over 20 million web visitors viewing our rankings results in 2009 alone and many others seeing them in print.
Rankings are inherently complex, and no ranking system will ever claim to be perfect. But we are proud of the level of accuracy and the level of acceptance we have achieved for our rankings. Professor Alan Kantrow, former Editor of McKinsey Quarterly says “QS…..is responsible for the best global ranking of universities.”
Why does QS produce the World University Rankings?
QS’s mission is “to enable motivated people to fulfill their potential through educational achievement, international mobility and career development."
It became obvious in the late 1990s that more and more young people want to study at undergraduate and postgraduate level outside their home country. The numbers have grown from 2.5 million to over 4 million people studying outside their home country in the last decade. A basis for comparing universities across borders is much-needed, and lies at the heart of QS’s mission.
Our primary focus is on serving the information needs of these young people, whilst at the same time recognizing that our rankings have huge interest for academics, university leaders and employers.
How has QS designed its methodology?
QS began the process of ranking universities internationally by identifying the primary missions of world class universities: Research Quality, Graduate Employability, Teaching Quality and International Outlook. It then sought ways of measuring each of these.
QS entered into consultation with many university rectors in compiling the rankings methodology and identifying these core missions. Subsequent consultation with almost 8000 academics in face-to-face seminars have shown strong support for our focus on these primary (though by no means exclusive) missions of world class universities.
Next, QS sought to develop a tried and tested approach to conducting an expert Peer Review Survey of academic quality. It brought in statistical and technical experts to ensure that our survey design could not be ‘gamed’ and provided valid results. The survey design is founded on the principles of many online political polls, which have become increasingly accurate at anticipating election results.
Sir Richard Sykes, rector of Imperial College London, felt that “an academic review is an appropriate way to compare universities. It takes smart people to judge smart people.”
What is the QS methodology?
QS selected the following indicators and weights. A much more detailed review of the methodology is available at the QS Intelligence Unit's website.
- Academic Peer Review 40%
- Global Employer Review 10%
- Citations Per Faculty 20%
- International Student Ratio 5%
- International Faculty Ratio 5%
- Faculty Student Ratio 20%
Is the methodology accurate and reliable?
The peer review conducted by QS has been independently validated by a leading statistician. Paul Thurman, Clinical Professor at Columbia University and a specialist in public health statistics, has found “The research QS is doing is more advanced than many other qualitative or simple ranking-based methods seen today.”
He adds, “Your method eliminates most response bias and you have controlled your sample, to be broadly representative by discipline and region. Your results are certainly statistically valid. With a 2% sample response yielding 9,500 responses, the margin of error is only 0.98% at the 95% confidence level.” This margin of error is as good as or better than most political polls conducted around the world today.
Have there been criticisms of the QS methodology?
Following the end in 2009 of a six-year collaboration between the two organisations, Times Higher Education has launched a campaign of criticism of the QS World University Rankings. QS owns all the intellectual property of the World University Rankings results and methodology.
It seems that THE believes the only way to legitimise producing its own new rankings is to pretend dissatisfaction with QS. Martin Ince was Rankings Editor at THE for six years, overseeing work with QS, and was previously deputy editor of THE. He says “I can honestly say that the rankings produced by QS, were regarded as an outstanding piece of work….accurate, insightful and absolutely fit for purpose.”
Professor Alan M. Kantrow, Former Editor of McKinsey Quarterly, adds “Global competition in higher education necessitates some basis for quality comparison across borders… QS’s Academic Peer and Employer Reviews rightly place the basis for comparison in the hands of experts - academics and employers around the world.”
What are some of the limitations of the methodology?
There are limitations to every ranking. There is certainly no single right answer, which is why QS believes in competition in rankings to provide consumers with multiple points of view. Some of the most common criticisms are the following:
- Institution size. Rankings have been criticized for favouring large universities. The Shanghai Jiao Tong Academic Ranking (SJT) certainly favours large, well funded institutions with an emphasis on science, and most of its indicators are not adjusted for size. In QS’s World University Rankings, by contrast, all the hard data indicators are adjusted for size. The unprompted nature of our survey design also means that respondents have to think about the universities which they actively know produce great research, and employers have to actively think about the universities they seek to recruit from.
- Bias towards the natural and life sciences. Measures such as citation counts do favor universities which are strong in the fields of medicine and natural sciences, where there is a strong publishing and citation culture. Nevertheless, within our Peer Review, QS normalises across five broad subject areas: Life sciences, natural sciences, IT and engineering, the social sciences, and the arts and humanities, as well as across geographies. This ensures that universities that are strong in the humanities or social sciences have almost as good an opportunity to feature in our results as those strong in the sciences.
- Anglo-Saxon bias. A common criticism of global rankings is that they favor universities which publish in the English language, because most journals counted by bibliometric databases (counts of papers and citations per university) are in English. In addition, Anglo-Saxon academics have a greater culture of citing each other’s work than academics in other countries. But the QS Peer Review is independent of any such language bias, and QS has gone to great lengths to produce our surveys in a range of languages, so as not to disadvantage non-native English speaking academics. We accept that some bias remains in the citation per faculty count, but we are encouraging the inclusion of as many foreign journals as possible. Our supplier of citations data, Scopus, has a database which is generally regarded as being less prone to language bias than its main rival, Thomson Reuters, which is used by THE.
How do you verify the data you collect?
QS goes to great lengths to verify all the data we collect, using multiple sources. Citations data is collected from Scopus (provided by Elsevier) and together we ensure accurate grouping of the data. Each affiliated college of each university needs to be incorporated into the analysis – for example, MIT has 1741 different affiliations.
Most of the hard data criteria are collected directly from universities, using a system which records the time of the entry and the person making it. All data is verified against government data statistics as well as against university web sites. Any inconsistencies are followed up directly with each university and resolved. This is a huge task, requiring a team of researchers speaking many different languages.
Do you classify different types of university?
QS has introduced a system which categorises each university on 12 different measures, based on whether the university specialises, is a general university and/or has a medical school, its number of students, and finally the research productivity of the university.
Arts and humanities-focused universities produce many fewer research papers than science-focused ones. In this way, sub-rankings can be drawn by classification. QS also publishes sub-rankings by the five broad subject categories defined earlier: life sciences, natural sciences, IT and engineering, the social sciences and the arts and humanities.
How transparent is the QS World University Rankings methodology?
All the hard data utilised in the rankings is published in various formats on TopUniversities.com. QS publishes detailed breakdowns of the respondents to our academic and employer surveys, which we feature in seminars attended by academics and employers around the world.
How well accepted is the QS methodology?
Over 8,000 academics have attended seminars specifically debating the QS World University Rankings methodology. Amongst attendees there has been almost universal acceptance of the QS ranking criteria.
The weightings applied to those criteria are subject to individual opinion and QS will be introducing a ‘personalised ranking’ system in 2010. This will allow people to apply their own weights and create their own ranking. For our overall rankings, QS believes in stability and changes are made only gradually, such as the introduction of recruiter review in 2005.
What are some of the benefits of the QS World University Rankings methodology?
Different stakeholders point to different benefits. The most commonly quoted benefits amongst universities and academics are the following:
- Strategic planning. QS World University Rankings provide a trusted an objective measure whereby university leaders can measure their own efforts to develop the performance of their university. The rankings are not an end in themselves, but simply a measuring tool. They can be used alongside data from other rankings, and internally generated data, to measure performance against goals.
- Benchmarking. Six years of prior results enable university leaders to benchmark the evolution of their university compared to other institutions.
- Measuring quality. Our use of citation data provides an indicator of the quality of research output. Universities can then go into greater depth by utilising tools provided by Elsevier to identify distinctive research competencies and research productivity at the individual faculty level.
- International recognition and brand awareness. QS reputation surveys provide the best measure available today of the strength of a university brand amongst other academics and employers. In a global education market place, this is a vital measure.
There are similar benefits for employers. The QS World University Rankings enable employers to plan their recruitment campaigns amongst highly rated universities anywhere in the world. In the past, a company expanding into a new market would have had to rely on local word of mouth when planning to recruit graduates. Today it has a reliable basis for its choice of target university. Indeed the rankings are encouraging many employers to try hiring at universities previously unknown to them.
For students and parents, the benefits of the World University Rankings are obvious. Any candidate seeking to study abroad no longer has to rely on the advice of an agent who may be on commission to recommend a particular university. Rather they can use objective data, compare universities for their different strengths and make informed short lists. QS publishes the Top Universities Guide and e-Guide providing further information on over 500 universities in 40 countries reviewed by the rankings.
Government officials and policymakers all around the world are keen users of the QS World University Rankings, which were set up partly in response to a suggestion by Her Majesty’s Treasury in the United Kingdom. The Rankings are an objective measure of a country’s university system. They are a powerful tool for spotting areas of strength in higher education as well as weaknesses that must be addressed.