Photo courtesy of SocialScienceSpaceDotCom

Earlier this month, the Ministry of Human Resource Development (MHRD) released India Rankings 2016. This is the first-ever effort by the government to rank Indian colleges and universities. Whether or not it should have taken up this task is a matter of debate. However, the more important question—since the government has already released its rankings and plans to do so on an annual basis—is whether students (and their parents) should consult India Rankings during this admissions season to make informed decisions about choosing a college or university for their education.

Magazine rankings versus government rankings

As students and parents already know, each year a few newsmagazines publish college and university rankings of various kinds. While some are better than others in terms of reliability, there are three issues in general that students and parents should keep in mind while consulting these rankings.

First, since all these publications are for-profit and extremely generous in carrying advertisements from ‘non-profit’ private universities of various kinds, including some very dubious ones, there is a clear conflict of interest. This is, of course, not acknowledged up front.

Second, unlike private universities, public institutions do not get to show off in the special college and university rankings issues of the magazines because they do not have a budget for showing off. They can at most advertise for admissions, jobs and so on. On the other hand, private universities often advertise quite aggressively and reach out better to potential students than public institutions even though most are clearly inferior to them.

Finally, there are limitations with the quantity and quality of information that newsmagazines are able to procure and verify and the methodologies they employ to prepare rankings.

With the exception of a small number of established private institutions, which are much sought after, others are actively seeking more students and tend to exaggerate their assets, whether it is infrastructure or faculty, in their effort to gain more students. Since newsmagazines benefit directly from advertisements by private universities, they have a clear disincentive to not seek to thoroughly verify the information provided by ‘donor’ institutions. As far as public institutions are concerned, among other things, newsmagazines have no tie-up with the government or with academic institutions so they cannot demand information from the institutions or verify it if need be.

The government, in contrast, has the authority and greater means to procure data and demand it from academic institutions when necessary. And of course, the government is also in a better position to verify the information provided if so needed. Indeed, overall, one of the advantages of government rankings is that it has the potential to be more complete and reliable than that prepared by newsmagazines.

The rankings exercise, however, is more complicated than collecting and interpreting data (however good). It is fraught with numerous measurement problems and methodological issues. Over the years, several problem areas have been exposed even in well-regarded rankings by Quacquarelli Symonds (QS) and Times Higher Education (THE) of which some have been resolved but newer ones keep cropping up. Indeed, the impulse behind preparing India Rankings 2016 was that the government was unhappy with the ranking parameters used by QS and THE. Given this context, it should not be surprising that rankings efforts by Indian newsmagazines certainly fall short in bigger ways. They simply do not have the required resources or the incentive to do a thorough-enough job of collecting information, verifying it and interpreting the collected data. Of course the next obvious question is: Can the government do better? Not this time.
 

India Rankings 2016

The National Institutional Ranking Framework (NIRF)was entrusted with the task of ranking academic institutions across six different categories: Universities and colleges; Engineering; Management; Architecture; and Pharmacy. As discussed below, the numbers of participating institutions were low and there are other limitations with the rankings as well. Students and parents should therefore see the rankings at best as a ‘good effort’ and treat it with skepticism and caution.

1. The number of participating institutions: The total number of participating institutions was on the lower side. This was not entirely unexpected since this was the first time that the government was carrying out such an exercise and participation was voluntary. India’s universities have in the past not shown themselves eager to share data and participate in university ranking, and this is something which the government wants to change with its new rankings system.
   
   In all, 233 universities, 1438 engineering institutions, 609 management institutions, 454 pharmacy institutions, 803 colleges and 28 architecture institutions submitted relevant data for the rankings. The numbers of participating colleges and architecture institutions were so few that it was decided not to rank institutions in these two categories. Now, a very large number of students—approximately 40 per cent of the total—head to colleges to earn degrees in arts/humanities/social sciences. India Rankings 2016 has nothing for them. Even in the case of universities, only 233 of the 757 universities—or 30 per cent—participated in the exercise. Given these low numbers, and allowing for the possibility that more than a few many good institutions may not have participated in the rankings, the rankings should at best be seen as provisional.
    
2. The quality of data: The preface of India Rankings notes the importance of “honest” and “reliable” data in preparing the rankings of academic institutions. It further calls for giving “deeper attention” to the problem in the future. Finally, because of problems with data and other issues, the report admits to “pitfalls” and “deficiencies”
   
   As already noted, institutions in two categories—colleges and architecture institutions—were not ranked due to non-submission of data by institutions. In addition, there are problems with the quality of data to an extent where it has skewed the rankings. In an open letter to Smriti Irani, the minister of human resource development, Maheshwar Peri pointed to 20 B–schools that are wrongfully included and another 20 that are wrongfully excluded in the list of top management institutions. Characterizing the effort as a case of “callous rankings with sparse information,” he quizzed: “How can you explain a public perception score of 834 for IIM (A) and 191 for IIM (B).”
    
3. Issues with categorizing institutions: Another problematic aspect of India Rankings is how it fits different kinds of higher education institutions into the six categories mentioned earlier. It was decided that higher education institutions would be ranked in one category only even when many could be a good fit for other categories as well. In the FAQs section of the NIRF, the following is stated:
   
   It was seen that institutions applying in multiple domains was leading to a situation that was creating confusion. Therefore, it was decided that institutions would be considered for ranking only under one domain. For this, institutions that had applied in 2 or more domains were communicated with and asked to choose one domain in which they wanted to be ranked. In some cases where it was clear which domain the institution would fall in, (for example IITs in Engineering), a decision was taken and communicated to them.
    

This decision is questionable. The IITs, for example, compete in world university rankings as universities even though they are primarily known as engineering institutions. Given that one of the objectives behind preparing these rankings was to help Indian institutions compete better in world university rankings, and the IITs have been competing as universities over the past many years, the IITs should have been allowed to compete both as universities and as engineering institutions.

Imposing the rule of ‘one institution, one category’, has also meant that some institutions have been placed and ranked under a less appropriate category. For example, the Indian Institute of Foreign Trade (IIFT) was forced to compete as a university when for all practical purposes, it is not one. Similarly, the Institute of Chemical Technology, Mumbai, is placed among the top 10 universities but it is a specialized institution which should be competing as an engineering institution and not as a university.

The NIRF should also have considered that the participation of institutions such as the IITs in more than one category—i.e. in the universities category—would not only have enhanced competition in that category but also given the IITs and the central universities in particular a better idea about how they compare to each other within the approximate framework of world university rankings. With the IITs ranked separately, many universities may have ranked high because of limited competition in the Universities category.
 

Treat with caution

Students and parents should proceed with extreme caution in taking any decision that is based on India Rankings 2016. That being said, they should also keep in mind, as many perhaps already do, that the rankings published by newsmagazines are not the whole truth either, even though many of them have been engaged in the exercise for a long time and may have become better at it over time. What they need to do is utilize some combination of the following in deciding which college/university to attend: college and university rankings by newsmagazines and NIRF; word-of-mouth reputation of institutions; visiting the webpages of highly-ranked institutions, especially the private ones to verify how good they are, especially with respect to faculty qualifications. They should certainly not get swayed by slick advertisements that private universities place in newsmagazines, online or on television.
 

Pushkar(@PushHigherEd) is Assistant Professor, Department of Humanities and Social Sciences, BITS Pilani-Goa.