NRC FAQs
My question is about...
- Purpose and history of the study
- Criteria for participation in the assessment
- Data collection and calculation
- Significance and use of the results
- Questions about my program’s data
Purpose and History of the Study:
Q. What is the purpose of the study?
The 2006 National Research Council’s Assessment of Research Doctorate Programs is an effort to help universities improve the quality of their doctoral programs through benchmarking and to provide potential students and the public with accessible, readily available information on doctoral programs nationwide.
These illustrative rankings should not be interpreted as definitive conclusions about the relative quality of doctoral programs, nor are they endorsed as such by the National Research Council. Rather, they demonstrate how the data can be used to rank programs based on the importance of particular characteristics to various users.
Q. Why is this study important?
To date, this is the largest dataset available to faculty, administrators and funders to compare, evaluate and improve programs. It offers an unprecedented collection of data on more than 5,000 doctoral programs in 62 fields at 212 universities in the United States It is also the first time that information on doctoral programs will be made widely available to prospective students and the public.
The study speaks to the importance of doctoral education as a key component of our system of education as well as the graduate education community’s efforts at continuous improvement. The results of this study come at a time of increased scrutiny of higher education and serves as a benchmark foundation for future assessment efforts in the ongoing work to improve doctoral education in America. The quality of doctoral education plays a vital role in developing U.S. expertise in a variety of fields and to enhancing U.S. competitiveness and innovation.
Q. Where can I get a copy of the report and data?
The assessment includes a report describing the approach used and general findings about U.S. doctoral education, as well as an Excel spreadsheet containing the data and illustrative rankings. The report and spreadsheet can be downloaded free of charge at www.nap.edu/rdp/, along with both the original and revised guide to the study methodology.
Q. Why do I see references to both a 2009 and a 2010 methodology?
In 2009, NRC released its methodology guide for this study, but subsequent changes were introduced into the methodology in the 2010 report. Therefore both are offered as reference.
Q. What is different between this NRC assessment and previous ones?
The National Research Council has conducted assessments published in 1982 and 1995. The previous studies gathered data that described doctoral programs and relied on a reputational survey.
The most recent study is a large collection of quantitative data that looks at 20 characteristics of doctoral programs. These characteristics have been weighted according to the importance placed on each by faculty within each field. The new study employs a much more sophisticated statistical analysis, in which results are presented as ranges of illustrative rankings.
Note: The 1995 methodology is different from the 2010 methodology in enough ways that the two are not strictly comparable.
The new study also expands the number of doctoral disciplines included in the study to capture emerging fields. The method of counting publications and citations also differs from previous studies. And it includes separate illustrative rankings for specific dimensions of doctoral study – faculty productivity, student support and outcomes, and diversity – which were not included in the 1995 study.
Q. What are the 20 characteristics used to assess programs?
- Publications per allocated faculty member
- Citations per publication
- Percent faculty with grants
- Percent faculty interdisciplinary
- Percent non-Asian minority faculty
- Percent female faculty
- Awards per allocated faculty
- Average GRE-Q
- Percent first-year students with full support
- Percent first-year students with external funding
- Percent non-Asian minority students
- Percent female students
- Percent international students
- Average Ph.D.s 2002 to 2006
- Percent completing within 6 years
- Time to degree
- Percent students in academic positions
- Student work space
- Health insurance
- Number of student activities offered
Q. Why aren’t the rankings reported as a single number?
NRC’s objective was not to produce an authoritative declaration of the “best programs” in given fields; the study committee concluded that no single such ranking can be produced in an unambiguous and rigorous way. Instead, the study illustrates how the data can be used to rank the quality of programs based on the importance of particular characteristics to various users. Users can generate many different sets of ranges of rankings, depending on which characteristics they wish to place the most importance.
Q. How was this study funded?
Financial support was provided by the National Institutes of Health, the National Science Foundation, the Alfred P. Sloan Foundation, the Andrew W. Mellon Foundation and the 220 participating universities.
Q. Why was the release delayed?
The 2006 study is more quantitative and is the first study of its kind. It required careful analysis of the data for more than 5,000 programs and continued scrutiny of the methodology to assure a quality outcome.
Q. When will the next NRC assessment begin?
The NRC hasn’t announced any information regarding future studies.
Criteria for participation in the assessment:
Q. What criteria were used to determine which fields to include in the assessment?
A field must have produced at least 500 Ph.D.s over the most recent five years and have at least 25 universities with programs that that produced three or more Ph.D.s in the last three years. Exceptions were made for a few fields that no longer met the criteria, but were included in past assessments. The number of fields included in the study increased from 41 to more than 60 and a new category of “emerging fields” is included reflecting the growth of interdisciplinary programs across universities (see http://sites.nationalacademies.org/PGA/Resdoc/PGA_044478).
Q. Which institutions participated in the assessment?
Two hundred and twenty-two institutions with about 5,000 academic programs participated in the NRC study, including cross-university programs. A full listing of institutions and programs is available at http://sites.nationalacademies.org/PGA/Resdoc/PGA_044749. The Graduate College also has created a searchable Web-based tool at http://www.grad.illinois.edu/nrc/NRCSelect.cfm.
Q. Which Illinois programs participated in the assessment?
A list of Illinois programs that participated in the assessment is available at www.grad.illinois.edu/nrc/taxonomy.cfm#mapping.
Q. Why was data collected for my program but results are not released?
Sixty-five Illinois programs originally were identified for inclusion in the assessment, and 58 of those were rated. The seven programs for which results are not released fell into one of the following three categories:
--They mapped to an emerging field (determined by NRC criteria) that was not rated.
--They were too small based on the number of PhD's awarded over a five-year time span.
--The programs that mapped to the NRC field Languages, Societies, and Cultures were not rated due to the broad nature of the field.
See www.grad.illinois.edu/nrc/taxonomy.cfm#mapping for more details.
Q. Why weren’t business, education, and social work included in the assessment?
The NRC Assessment, first begun in the early 1980s, focused on what were, at that time, considered traditional research-oriented Ph.D. programs. More recently, NRC has been expanding this focus with each assessment, primarily through the inclusion of programs like agriculture and expansion of the life sciences. Education, business, and social work remain outside the realm of this study at this time. Future NRC assessments may expand to include these disciplines.
Questions about data collection and calculation:
Q. When were the data collected?
The data were collected from academic year 2005-2006 through questionnaires sent to those identified as doctoral faculty by their institutions, as well as through questionnaires sent to the heads of doctoral programs, administrators, and students. Information on characteristics such as publications and citations came from public sources and uses a considerably longer timeframe. (See the 2009 Methodology Guide)
Q. How were the data collected?
The dataconsist of answers to questionnaires that were sent to the universities, the doctoral programs, the faculty of these programs, and to advanced doctoral students in five fields (physics, English, chemical engineering, economics, and neuroscience), as well as data from the National Science Foundation, the Institute for Scientific Information, and from 224 scholarly and honorary societies.
Q. How were GRE scores collected and used?
GRE scores were collected as part of the program questionnaire. The 2009 Methodology Guide (pp. 50 and 61) provides details about how these were factored into the assessment.
Q. How were the ranges of rankings calculated from the data?
Two approaches were taken and then combined to calculate ranges of rankings. One approach asked faculty about the importance of program characteristics independent of any particular program. The second approach asked about the quality of specific programs without specifying particular characteristics. The results of the two approaches were then combined to form one set of weights.
In the first approach, doctoral faculty members were asked directly what they considered to be the most important characteristics that contributed to program quality. The faculty responses, calculated separately for each field, constituted a first set of weights.
In the second approach, programs were rated on their perceived quality by samples of faculty separately for each field. From a statistical analysis of these program quality ratings, weights were derived for each of the characteristics. These weights constituted a second set of weights for each field.
The weights calculated by each method were similar but not identical. These two sources of “importance weights” were then combined to give a final weight for each of the 20 variables.
Uncertainty was allowed for by taking 500 samples of raters, since rater opinions may vary, and by allowing for the variance of each characteristic within a range. The ratings were then arranged from highest to lowest (that is, ranked) for all the programs in a field. For each program, the middle quartiles of rankings were then calculated. This calculation results in a “range of rankings”for each program rather than a single rank order. Similarly derived ranges of rankings are calculated for each of the supplementary measures.
Q. Why are illustrative rankings given in ranges instead of as a single number?
The committee felt strongly that assigning to each program a single number and ranking them accordingly would be misleading, since there are significant uncertainties and variability in any ranking process. Uncertainties arise from assumptions made in creating a ranking model based on quantitative data on program characteristics. Even with such a model, variability arises from numerous sources, including differences in the views among the faculty surveyed, fluctuations in data from year to year, and the error inherent in estimations from any statistical model. The ranges reflect some of this uncertainty and variability.
Q: What do the S-rankings mean?
The S (or survey-based) rankings reflect the degree to which a program is strong in the characteristics that faculty in the field rated as most important to the overall quality of a program. In a survey, faculty were asked about the importance of 20 characteristics – for example, publications per faculty member and students' time to degree – in determining the quality of a program, and each characteristic was assigned a weight accordingly. These weights varied by field, since each characteristic is not valued to the same degree in all fields; the percent of faculty with grants was valued more highly in biology than in history, for example. The weights were then applied to the data on these characteristics for each program, resulting in the ranges of S-rankings for the program.
Q: What do the R-rankings mean?
The R (or regression-based) rankings are based on an indirect approach to determining what faculty value in a program. First, a sample group of faculty were asked to rate a sample of programs in their fields. Then, a statistical analysis was used to calculate how the 20 program characteristics would need to be weighted in order to reproduce most closely the sample ratings. In other words, the analysis attempted to understand how much importance faculty implicitly attached to various program characteristics when they rated the sample of programs. Weights were assigned to each characteristic accordingly – again, these varied by field – and the weights were then applied to the data on these characteristics for each program, resulting in a second range of rankings.
Q: Are the R-rankings the same as reputational rankings?
The R- rankings are not the same as reputational rankings. The R ranking ranges are based on data about the programs; the reputational ratings for a sample of programs in each field were used to determine the weights, but were not used directly to rank programs.
Q: What do the "5th percentile" and "95th percentile" mean in the illustrative rankings?
The degree of uncertainty in the rankings is quantified in part by calculating the S- and R-rankings of each program 500 times. The resulting 500 rankings were numerically ordered and the lowest and highest five percent were excluded. Thus, the 5th and 95th percentile rankings -- in other words, the 25th highest ranking and the 475th highest ranking in the list of 500 -- define each program's range of rankings, as shown in the Excel spreadsheet.
For more information on the methodologies used to calculate the S-ranking and R-ranking ranges, see Chapter 4 of the report, as well as the revised methodology guide.
Q: Why are there differences between the R-rankings and the S-rankings?
Although each approach was based on the program data, different sets of weights were applied to the data, yielding different ranges of rankings. In the S-rankings, for example, faculty in most fields placed the greatest weight on characteristics related to faculty research activity, such as per capita publications or the percentage of faculty with grants. Therefore, programs that are strong in those characteristics tend to rank higher. Such characteristics were also weighted heavily in the R-rankings for many fields, but program size (measured by numbers of Ph.D.s produced by the program averaged over five years) was frequently the characteristic with the largest weight in determining these rankings.
The National Research Council is not endorsing either ranking or any ranking as the best indicator of program quality, but instead is providing the R- and S- rankings as illustrations of how rankings can be created by applying weights to data on program characteristics. The degree of importance attached to each program characteristic depends on how the rankings are to be used. The program data are being made available so that users can compare programs based on the characteristics that are most important to them.
Q: Why is the methodology different from that described in the 2009 Methodology?
NRC had originally planned to combine the R- and the S- rankings into a single range of rankings, which is the approach outlined in the 2009 Methodology Guide. The production of rankings from measures of quantitative data turned out to be more complicated and to have greater uncertainty than originally thought. As a consequence, it was decided that the two measures should not be combined, and instead NRC has presented them as two illustrative rankings. Neither one is endorsed or recommended by the National Research Council as an authoritative conclusion about the relative quality of doctoral programs. It was also decided to include a broader range of rankings from the 500 calculated for each program. The range in the 2009 Methodology Guide excluded half of them (the highest 25 percent and the lowest 25 percent); now only 10 percent – the highest 5 percent and lowest 5 percent – are excluded.
Q. How was faculty research activity measured?
This dimensional measure relates to various ways to gauge the contribution of research: publications, citations (except for the humanities), the percent of the faculty holding research grants, and recognition of scholarship as evidenced by honors and awards. Specifically, the components of the research activity dimensional measure are: average publications per allocated faculty member, average citations per publication, percent of core and new doctoral faculty respondents holding grants, and awards per allocated faculty member. Publishing patterns and the availability of research funding and awards for scholarships vary by field, but the weight placed on publications per faculty member is remarkably consistent – about 30 percent – across fields (see the 2009 Methodology Guide, p. 12).
Q. What if a faculty member has an appointment in more than one program?
In most cases faculty productivity was prorated by percentage of appointment. Faculty allocations are posted in the Graduate College Web-based data tool at https://www.grad.illinois.edu/nrc/results/ (bluestem login required). It is important to note that the algorithm NRC used for calculating faculty appointments differs from the calculation of faculty appointments typically used on campus.
In a few cases, such as the assessment of diversity, only core and new faculty were included in the measures (see the 2009 Methodology Guide, Appendix E).
Q. How was diversity measured?
The measures that are included in this dimensional measure [Diversity of the Academic Environment]: the percent of faculty and percent of students who are from underrepresented minority groups, the percent of faculty and the percent of students who are female, and the percent of students who are international (that is, in the United States on a temporary visa). In terms of field differences, most fields place the highest weight on the percentage of students from underrepresented minority groups. In the health sciences, social sciences, and humanities, relatively high weights are also placed on the percentage of faculty who are underrepresented minorities. The percentage of international students was not highly weighted, except for the physical sciences (see the 2009 Methodology Guide p. 13 and Appendix F).
Q. Do the weights differ between programs?
The weights were determined by faculty responses to questionnaires that asked faculty which characteristics they believe are most important. Weights were then calculated for each field and used consistently across institutions.
For example, the characteristics identified as most important to faculty in chemistry programs are not identical to those identified as most important by history faculty, and will be different. However, the weights used for the chemistry program at Institution X are the same as those used for the chemistry program and Institution Y.
Significance and use of the results:
Q. Who is the intended audience for this data?
The purpose of the study is twofold: To provide doctoral programs with data to compare themselves to similar programs, and to provide accessible data about program characteristics to prospective doctoral students.
Q. What if the results differ significantly from other assessments or from rankings such as U.S. News & World Report?
This survey does not cover undergraduate or masters programs at all. It is solely about doctoral research programs. Also unlike U.S. News this survey does not include a subjective “peer assessment” score.
Q: Why did I hear a news story that said that ______________University is ranked number one by the NRC?
Because the report does not rank programs, department or schools, some (reporters/institutions) have attempted use the data to determine rankings. By weighing the categories differently, it is possible to create a nearly infinite set of ranking outcomes.
Q. How meaningful are the results of this study given the length of time between data collection and release of the results?
This third NRC assessment began data collection in July 2006. The results provide the most comprehensive and valid analysis of research doctorates that is currently available.
Q. How will our campus use NRC data?
The purpose of the NRC Assessment is to provide information to doctoral research programs in order to help programs understand their strengths and weaknesses compared to their peers. The data should prove useful for directors of graduate study, deans, and faculty within programs.
Q. Can I use NRC data in marketing and publicity for my program?
Marketing decisions should be made on a unit by unit basis in consultation with your chief communications officer, who will consult with Public Affairs as needed.
Q. How can a student like me use the data and rankings to help me evaluate various programs?
Students can pick out the programs of interest and compare them on characteristics such as percent of students funded in the first year, time to degree, and placement rate. Comparing programs based on the dimensional ranking "Student Support and Outcomes" -- which is a single measure that combines these characteristics -- may be helpful as well. A tutorial giving an example of how students can use the spreadsheets to select and compare different programs will be available at www.nap.edu/rdp.
In addition, PhDs.org, an independent website not affiliated with the National Research Council, is incorporating data from the assessment into its Graduate School Guide. Users of the Guide will be able to assign weights to the program characteristics measured by the National Research Council and others, and rank graduate programs according to their own priorities.
Prospective students should not use the NRC data alone in their evaluation. They should talk to faculty in their field of interest and students who are now pursuing doctoral study in the programs that interest them as they make their evaluation.
Questions about my program’s data
Q. Where can I get a copy of my program’s data that were submitted to NRC?
Each program’s data are made available by the Graduate College at https://www.grad.illinois.edu/nrc/final/index.cfm (bluestem login required).
Q. How do I find out how the variables were weighted for my program?
This information is included in the Web-based data tool provide by the Graduate College.
It is accessible at https://www.grad.illinois.edu/nrc/results/ (bluestem login required).
Q. What does it mean if my program moved significantly up or down from the 1995 rankings?
Because the new study uses a very different methodology it is impossible to make direct comparisons between the two studies. See Chapter 7 of the 2010 Report.
Q. Can I find out which faculty in my program completed surveys or at least what percentage?
Names of individuals who completed the survey are confidential. Overall, however, Illinois had an extremely high response rate for completion of questionnaires.
· Program completions rates for Illinois were 100%, and 89% nationally.
· Faculty completion rates at Illinois were 83%, and 73% nationally.
· Student completion rates at Illinois were 71%, and 53% nationally.
Q. Who do I ask if I have questions about the NRC data?
E-mail the Graduate College at gradnrcresults@illinois.edu.
Q. How can I learn more about how to work with NRC data?
The Graduate College will host a series of workshops for deans, directors and faculty, who wish to learn more about working with NRC data. The Graduate College will send notification of workshops to Directors of Graduate Study, Graduate Program Contacts, and Graduate Faculty. You can also check the Graduate College Faculty & Staff Events Calendar (http://www.grad.illinois.edu/events) for upcoming workshops or contact the Graduate College at gradnrcresults@illinois.edu.