Academic

US News Rankings and the Data Disclosure Dance

By Roger Dennis

US News rankings week always leads to a major use of Internet band width.  And The Faculty Lounge is no exception.  As Dan Filler has already noted, some of the most thoughtful comments in this cycle come from Judge Pollak who offers an alternative method for assessing law school quality.  And there is the usual kerfuffle about possible US News mistakes and the potential for gaming the system.  This year?s controversies are well summarized by Paul Caron.  (Update: Paul has information about more possible errors here.)  The securities regulation lawyer in me got to thinking about whether there is a way to respond to Judge Pollak while increasing reporting reliability and satisfying the commercial interests of US News.

One partial answer to Judge Pollak?s call for a better focused ranking system could be a more extensive disclosure of data by US News.  Under the current system, each law school provides US News the data it requests.  This component data is not made broadly available by either US News or the law schools.  US News then notifies each school about how its own data is converted into points which result in its overall score.  Disclosure by US News of every school?s component points and the underlying data used to create component points would enable users to better evaluate factors that are important to them individually.  For example I am imagining that ranking point differences that are derived from library volumes and expenditures per student could become less significant while bar passage rating points, a currently low point value item, would become more relevant to users. And US News could charge for this data dump as part of their premium online service. 

More disclosure should also lead to better reliability.  Anomalies would be more obvious and ex ante ought to be driven from the information marketplace.  Because US News has become such a high stakes game there is also a role for regulation of information disclosure.  It?s time to be more systematic about the way the data is collected.   A relatively low cost solution comes to mind.  Key data is derived from bar examiners and information provided to the ABA and LSAC by law schools.  Law schools should agree to allow these organizations to provide the data to US News directly. At a minimum, this should create more data consistency.  Moreover, the legal education community ought to take taking very seriously its obligation of providing accurate data to US News.  Such an obligation already exists in the ABA accreditation standards (specifically Interpretation 509-4).  US News requires deans to sign their school?s submission.  Maybe the ABA should pile on and require a Sarbanes Oxley like certification to it that the law school is playing fair with US News.

Full post as published by The Faculty Lounge on April 27, 2009 (boomark / email).