Getting real about rankings

Creating a grading mechanism for Indian universities from scratch, particularly in a large, complex, and a disorganised system, is a massive challenge

Human Resources Development Minister Smriti Irani has announced that India will develop a national rankings system for its universities. In principle, this is a good idea. International rankings do not entirely suit Indian realities, and India has done abysmally in them. Further, providing benchmarks to measure the productivity of Indian universities and creating a sense of movement and competition among them is laudable.

An optimistic exercise

The challenge of actually creating rankings that will be based on real and relevant data is immense. It is worth thinking about the problems before plunging into uncharted territory. The experience of some other countries is not especially favourable. A few years ago, the Russians, stung by the poor showing of their universities in international rankings, created an international ranking system of their own. Unsurprisingly, Russian universities did quite well. However, no one, even in Russia, believed the results of this ranking and the project disappeared. The Bertelsmann Foundation in Germany has been working for almost a decade on a non-ranking compilation of German institutions that has been widely praised. But it has taken a long time. Even the influential U.S. News and World Report ranking in the U.S., now in its 30th year, is regularly criticised for methodological and other failings. The goal of creating and implementing an Indian ranking, for release in December 2015, seems overly optimistic.

It is crucial to “get it right” the first time. If the Indian ranking system is seriously flawed in its design, methodology, data, or interpretation, it will be picked apart and immediately lose credibility. Creating a ranking from scratch — particularly for an academic system as large, complex, and in many ways as disorganised as India — is a massive challenge.

The challenges

Reports indicate that the main responsibility for developing the ranking, scope and methodology will be in the hands of the Indian Institutes of Technology (IITs). While they are distinguished, they are not universities but rather specialised technology institutions, and are unfamiliar with the broader university context. Further, they have no special expertise on higher education — a requirement to develop a good ranking system. A very substantial problem in India is the lack of data on many aspects of higher education. Even basic up-to-date statistical information is often lacking. Without reliable data on all of the aspects of ranking from all of the universities included, the rankings will have limited value.

Will the rankings include India’s 35,000 colleges? The vast majority of students attend these institutions. Almost all are affiliated to a university, but there are significant variations in quality, focus, and orientation among the colleges affiliated to any single university. Many will lack good data. It is impossible to generalise about more than one hundred colleges affiliated to a university. If the rankings include only universities, they will be of limited relevance to much of the public.

Some of the metrics that are proposed for measurement (such as ranking of social roles of universities) are impossible to measure. How will social roles be defined? Further, there is no data available, regardless of the definitions. Other global and national rankings have struggled with measuring teaching quality; no one has solved this dilemma. Some of the rankings use such proxies as the teacher/student ratio and similar relatively easy factors, but these do not measure actual quality.

All of the global rankings include publications and research funding as key productivity factors for universities. Most rankings count articles in internationally respected journals, included in the Science Citation Index or their humanities or social science equivalents. They then include citation rates and other criteria of actual use of the publications. The problem is that few Indian journals are included in the international indexes and there are no Indian equivalents that can be easily included. It would be possible to create such indices, but this will take both time and money. At present, there is no accurate way of evaluating either the scope or the influence of the publications of Indian academics.

Similarly, there are not easily available data sources for research funding or patent development, although these would be easier for universities to develop if careful criteria are put into place. The data collection challenges for universities will be quite substantial — there is only limited information currently available. In many countries, most universities have institutional research offices that are responsible for data collection and analysis and are able to provide information on a range of topics required by ranking agencies, quality assurance authorities, and the government. India does not have a tradition of institutional research — although building such offices is a key requirement of professionalising the work of universities. Internationalisation will be one of the criteria for excellence in the rankings. Indian universities are just now recognising the importance of internationalisation and will score poorly, at least in the short run. Few have a strategy to engage with the rest of the world, and the numbers of international students and staff in most institutions are quite small.

Will the private higher education sector be included in the rankings? A few of the private institutions are innovative and may score well, although most will not. These universities may have less data available than their public counterparts, and some may be reluctant to report accurate statistics.

A lost cause?

The idea of rankings is a good one. Rankings will stimulate the further professionalisation of Indian academe. Rankings will create a sense of competitiveness in the system; they will help build a differentiated academic system with a few internationally recognised research-intensive universities and a much larger number of institutions that will focus mainly on teaching. But implementation will not be easy. Those involved must be realistic about what is involved, what the costs will be, and how much time and energy will be required. If published reports and public statements are any guidelines, realism is not part of current thinking or planning.