Originally post on May 19, 2011
The Math-NSERC liasion committee has received the following letter from 16 members of the EG 1508:
May 19, 2011
Dr. Suzanne Fortier
President, NSERC
Dear Dr. Fortier,
We would like to draw your attention to the distressing results of the 2011 Discovery Grants Program in Mathematics, a competition in which all the signatories of this letter played a role, as members of the Evaluation Group for Mathematics and Statistics (1508). We have strong reasons to believe that the outcome of the competition in 2011 is not consistent with NSERC’s clearly stated goal to have a “process that is responsive to merit and is objective, transparent and fair”, as quoted from your recent presentation of the competition summary.
We detail below each aspect that seems seriously problematic in the hope and expectation that NSERC will rapidly correct the situation for the applicants from the recent competition of 2011 or, at the very least, ensure that these issues are adequately resolved in time for next year’s competition.
Lack of fairness: The members of the Evaluation Group (hereafter, the EG) have been deeply concerned by the fact that there are huge funding differences for comparable files in the 2010 and 2011 competitions. Indeed, funding for the same bin in 2010 and 2011 dropped by as much as 40% ($12,000, in this particular case), even though we, as members of the EG, agree that rating standards for the two successive years were not significantly different. Researchers expect to be judged on the merit of their file when compared to the whole community, not only to the applicants in the same competition. Variations such as those observed between 2010 and 2011 are highly unfair.
Responsiveness to merit : In addition to the bin funding discrepancies between 2010 and 2011, we have observed that funding differences between successive bins (from \$1,000 between bins I and J, to \$9,000 between bins D and E) are extraordinarily large and appear, frankly, arbitrary from the perspective of merit. Researchers with remarkable achievements and outstanding potential are provided with insufficient funds to develop and carry out their research projects as a consequence of this extremely unbalanced funding model. As committee members, we were asked to use the full scale in our ratings, and we did so, but with the expectation that scientific merit would ultimately be rewarded with proportional grant amounts. The bin values selected by NSERC distort the rating scale in such a drastic fashion that the lengthy and rigorous evaluation process that took place was rendered almost meaningless.
Transparency: The 2011 competition and its aftermath also raises questions about the transparency of the present system in which critical funding choices are exclusively the purview of the Executive committee of the EG and NSERC staff, and we feel that Isabelle Blain’s letter to the applicants hardly addresses any of the concerns expressed by the EG members and the community.
The Executive of our EG was placed in the very difficult situation to have to choose between two unpalatable and unfair scenarios, a huge drop in success rates from a year to the other (from 62% to under 45%), or reducing drastically bin values in mid-range bins. All along this process as well as after it, for confidentiality reasons, they were unable to consult and explain in a detailed way the whole situation to the rest of the EG.
One example of how the general lack of openness affected the process and its aftermath is related to the decision made by the Executive and NSERC staff to have different funding levels per bin for mathematics and for statistics, even though the two subjects share the same EG. The difference in funding levels was swiftly explained in Ms. Blain’s letter by claiming a different interpretation of the merit criteria. However, there was no comparison of files with the same rankings in mathematics and in statistics during or after the end of the competition to justify such a broad claim – in fact, bins can not even be opened after the end of the competition. After noticing a markedly different bin distribution in mathematics and statistics, the Executive decided to split the budgets between these two groups, a decision justified by the fact that, historically, these two subjects have been separated one from the other in the funding process and the respective subgroups of the EG had worked in complete separation also in this evaluation exercise. As a consequence, the bin values in statistics became higher than in mathematics and reached levels comparable to those in 2010. As the confidentiality principle did not allow the Executive to explain to the EG, not to mention to consult with its members on,the full details of the various scenarios involved, this led to considerable unease in the EG and beyond.
Another example of lack of transparency is apparent in the fact that Ms. Blain’s letter hardly makes any attempt to justify grant funding differences between 2010 and 2011. Her letter indicates that one of the priorities in the current year was to fund the bin J. This is commendable and in keeping with the wishes of our community, as we understand them. But it does not explain why this led to hugely different results from 2010, when the same criteria were applied and the same bin J was funded, with a similar overall success rate. We are distressed that her letter did not refer to any of the other variables entirely in NSERC hands and involved in this outcome : first and foremost the budget, the values of the lowest funded and the top bins, and the gradations between funding amounts for different bins.
As members of the EG, we were involved in a process that is of great importance to our scientific community and, more generally, for the advancement of research and innovationin Canada. We have made considerable efforts to implement the new grant distribution system in ways consistent with the highest scientific standards in a context made particularly difficult by the chronic and increasingly severe underfunding of the mathematical and statistical sciences when compared with other NSERC supported disciplines. We established our ratings with the best of our scientific expertise and we naturally expected this to be reflected accurately in terms of budgetary allocations. This was far from being the case.
As reviewers, we were highly impressed with the quality of the 2011 applicants. Canada, through NSERC, has supported many of them in past years and can take justifiable pride in their achievements. Moreover, the Discovery Grants Program is essential for the progress of science in Canada and, for the outright majority of applicants reviewed by the EG, is the only available source of significant research funding. Unfortunately, the 2011 Discovery Grants Program in Mathematics failed many of the most promising of these applicants. As a consequence, our confidence in the program, as currently administered, is regrettably shaken. We urge you to take rapid corrective action so that the 2012 competition will not suffer from the same problems.
Yours Sincerely,
Mike Bennett (UBC)
Nantel Bergeron (York)
Lia Bronsard (McMaster)
Thomas Brüstle (Sherbrooke and Bishop’s)
Olivier Collin (UQAM)
Benoit Collins (Ottawa)
Octav Cornea (Montreal)
Alan Dow (North Carolina, Charlotte)
Hermann Eberl (Guelph)
Christopher Godsil (Waterloo)
Eyal Goren (McGill)
Robert McCann (Toronto)
Matthias Neufang (Carleton and Fields)
John Stockie (Simon Fraser)
Holger Teismann (Acadia)
Xiaoqiang Zhao (Memorial University)
cc: Ms. Isabelle Blain, Vice-President, Research Grants & Scholarships; Ms. Madeleine
Bastien, Team Leader, NSERC Evaluation Group 1508; Mathematics-NSERC Liaison
Committee.
Recent Comments