Originally post on June 30, 2011

NSERC President Suzanne Fortier has replied to the math/stats community’s public letter with a letter.

Here is a link to President Fortier’s letter in PDF format.

Originally post on May 27, 2011

Analytical overview of the three previous Evaluation Group 1508 meetings

Presentation at the May 20, 2011 conference call NSERC-Mathematics & Statistics Liaison Committee

Walter Craig
Department of Mathematics & Statistics
McMaster University

Introduction: The past three years of NSERC Discovery Grants have not been good ones for Canadian mathematics and statistics. Average grant amounts for their Discovery Grants (DG) are \$20K as opposed to the NSERC DG average of more than \$34K. The total budget allocated for mathematics and statistics Discovery Grants declines each year; indeed over a five year period this amount has decreased by 17.5% 1. And many award amounts for Discovery Grants do not make sense with respect to the international stature of the researcher. In general, the median grant does not support a research program that is reasonable and sustainable for an active high quality researcher. 2

On consultation with members of the Canadian mathematics and statistics community, there emerge three basic causes for this situation, pointing to actions that we must take:

(1) The Discovery Grant budget for Mathematics & Statistics (EG 1508) is insufficient. This has been recognized by the research community and NSERC alike, however action to remedy the situation is ever put off to the future. For this I quote Shakespere’s Hamlet ‘…tomorrow and tomorrow and tomorrow…’ as the response to the problem. We need NSERC to show a good faith effort for action on this problem today.

(2) The Evaluation Group (EG) and its chairs and co-chairs are expected to play a leadership role in allocating research funding for the community, while given no actual control, nor even any prior knowledge of the budget. The real role that they play at this point is of a scapegoat (en francais, le bouc emmissaire).

(3) The dynamics of the EG decision-making process under the current bin-conference model relies upon too few parameters to make its subtle choices, and is excessively and rigidly controlled through rules imposed upon it by NSERC.

All three of these causes need to be studied and ameliorated. Fortunately, item (3) can be addressed in a straightforward manner, with a certain amount of thought, planning and good will. I will talk about this today. At the end of this presentation I will make several suggestions for changes, all of the implementable within the bin-conference model, none of which is too big. The result will be a more fair, more stable, and more transparent process of Discovery Grant allocation. Regarding item (3), the EG must make difficult decisions about comparing the quality of grant proposals and funding them, and do so in as fair as possible way, as do all grant evaluation committees. In the current procedures, they are not given the tools to do so. This is the aspect that I am suggesting to change, and this can be accomplished with relatively minor modifications to the current methods.

Outline of the dynamics of decision making in the EG meetings:
Just as is the case with most other scientific grant awarding agencies, proposals are solicited and received. In mathematics and statistics there were about 274 this year (the numbers here and below are for the most part estimates, as the actual figures were not made available at the time of this meeting). And as is the case with each such grant competition, they fall into three categories.

1. high quality researcher, will be funded at some level (perhaps 20% of the proposals)
2. proposals which, despite their possible merit, will most likely not be funded because of the competitive level of the process. (perhaps another 20%)
3. proposals which would deserve to be funded, if the budget allows. (the remaining middle 60% of the proposals)

This division, explicit or implicit, takes place in every grant reviewing panel that I have experienced. In my mind, the third ‘middle’ category corresponds to the NSERC bins H (VVS), I (VSS), J (SSS) and K (SSM). The funding cutoff will have to be placed somewhere within this large group of proposals.

The current procedures of the NSERC EGs: proposals are inspected serially (by conferences or sub-committees made of the EG members) and rated (with three ratings in fact) placing them in a ‘bin’. As it is known that in the prior two competitions, bin J (SSS) is most likely to be the lowest bin that will be funded, while the budget is not known, it is very difficult for an EG member to give one of the middle proposals a rank lower than that for bin J. The factual evidence for this is that a full 33% of the successful grants in Math & Stats are in bin J. Indeed about 68% are in the combined bins H, I and J. In the 2011 Discovery Grant competition, this exceeds most other EGs by quite a lot, as one can check from the NSERC data release at

http://www.nserc-crsng.gc.ca/doc/Funding-Financement/DGStat-SDStateng.pdf

As the EG members rank proposals, there is basically only one pass, with no construction of a global picture of their spectrum of quality. It is certainly realized that proposals should be judged fairly with respect to each other, and with the new bin system there is an attention towards a ‘calibration’ as to the meaning of the different letter grades (EOVSMI). And true problematic case are allowed to be reinspected, but these are a small minority of the total. The hard decision, which is where to draw the grant cutoff line once the budget is revealed, is somewhere between bins H and K, and given its size, probably in the middle of bin J. However the rules are rigid as to a bin being indivisible. In my opinion this is a very important point, namely the EG executive committee is forced to decide about the inclusion or the exclusion of all proposals that have been placed in this critical bin, leading to the depletion of the budget, or respectively an unacceptably low success rate. This is precisely because the dynamics of the decision making process has resulted in so many proposals being allocated to bin J.

There are quite a few other (and larger) grant awarding agencies, and it is a good idea to discuss what procedures they have developed, and their sense of best practices. I will start with a comparison with the NSF of the United States 3. At a meeting of a NSF grants panel, all proposals are visible to all panel members, and can be read and judged. A subcommittee is charged to read and recommend each proposal (4 people, a ‘reporter’ and three ‘readers’), however the full panel plays a role in the decision. Each proposal is compared with all others. Indeed they are labeled on post-its and placed on a white board at the front of the committee room. And there are three columns, where the middle on corresponds roughly to the bins H, I, J, and K. In rank ordering the proposals, the panel makes at least three passes through the whole list. And indeed the tough decisions are made on the proposals in the middle column. This is where the panel spends most of its discussion time. At the end of the panel meeting (normally lasting 3 days) a summary review of each proposal is written by the relevant subcommittee, and the panelists approve it. The global ranking system, the transparency of the judging decisions, and the summary review are all part of the accountability of the NSF process.

The biggest difference between NSERC’s present procedures and the time-honored ones of the NSF, are that he latter offers a global picture of the spectrum of quality of the proposals. The current NSERC practice is weak precisely on this point. There is only effectively one pass through the proposal dossiers, and the internal quality of proposals within the same bin is not compared. If however the NSERC Evaluation Group members were able to construct a global picture before proposals were rated into bins, the difficult decisions would be able to be made. This would allow them to avoid the situation in which bin J contained approximately 58 proposals (as it did this year) where the funding decision rules require either all or none of them to be awarded a grant.

The funding agency that I know second best is the CNRS administered ANR grants of France. I served on this panel last year. There are two weeks of meeting, one in February and one in April, and in all there were five passes through all of the proposals, in which all panelists participated. And on questions of conflict of interest, Canada may have quite a few, but France has very many, due to the prominence of the École Normale Supérieure and the École Polytechnique. Yet they are able to handle them, in a relatively graceful way. As far as accountability goes, it is important in these sessions that the reader, that is the subcommittee chair, write and sign each proposal’s summary review.

There is one further comparison to make, this time something close to every university instructor, because we all have to mark final exams and give grades in large sections of freshman courses. The time honored `best practice’ is to assign each problem of the final a number of points, calibrated to its difficulty or importance. As the problems are graded, points are tallied for each exam script. A fair amount of effort goes into calibrating the standards of grading between different graders and between those marked earlier and later in the grading process. A global distribution is tabulated, often graphically. Only then are the grades ABCDF assigned. Finally marks are reviewed; any mark that does not make sense with respect to the known qualities of a given student is revisited. This procedure is considered the best way to mark a course exam fairly. Without this procedure, or something similar, an instructor of a large course knows that he/she will get into difficulties. Grading exams serially as A, B, … F, with no backward looks nor comparisons of the global distribution, would definitely lead to injustices.

Demands for action:

Regarding item (3) on EG procedures, we must make changes to the decision making process so that grant awards are fair and make sense. It is a procedural question that can be worked out, as long as good will is exhibited by all sides. It should contain the
following elements:

• EG members should be given a global picture of the proposals in the competition, which should be ranked according to global comparisons.
• rules governing proposal rating should be transparent, flexible and make scientific sense.
• The ranking system must require that EG members and NSERC staff be accountable for their decisions. In particular a summary review of the panels decisions should be forthcoming for every proposal.

Regarding item (2), which has to do with EG chairs and co-chairs, we have started to work on suggestions for members for next year’s committee, doing this in a correspondence with Patricia Rouillard of NSERC. We are ready to give more input to this process and further suggestions for new EG members. We appreciate that NSERC is consulting us, we are willing to respond; more is needed.

Regarding item (1), having to do with the Mathematics & Statistics EG 1508 budget:

• returning grants to EG 1508 are about \$3,828K, while we have estimated the outgoing budget of this year’s DG competition at about \$3,334K, this is a difference of almost \$500K. It is just about enough to restore the mathematics bins to values comparable to the 2010 DG competition. NSERC should do this immediately.
• We of the Canadian mathematics and statistics community need to have accountability of NSERC for its budget decisions regarding EG 1508. In particular we need to know the origin of such deep budget cuts as have occurred this year, and why they they were not signaled to the community nor to the Math & Stats EG during the main phase of the rating of our proposals.
• The separation of the mathematics budget from the statistics budget is controversial and divisive. Recognizing the deep level of commonality in mathematics and statistics, this should not have been done, and we should not be placed in our current position of two grant award values for the same quality standards, one for mathematics and another for statistics.

Respectfully,

Walter Craig

Chair, Mathematics – NSERC Liaison Committee

Originally post on May 19, 2011

The Math-NSERC liasion committee has received the following letter from 16 members of the EG 1508:

May 19, 2011

Dr. Suzanne Fortier
President, NSERC

Dear Dr. Fortier,

We would like to draw your attention to the distressing results of the 2011 Discovery Grants Program in Mathematics, a competition in which all the signatories of this letter played a role, as members of the Evaluation Group for Mathematics and Statistics (1508). We have strong reasons to believe that the outcome of the competition in 2011 is not consistent with NSERC’s clearly stated goal to have a “process that is responsive to merit and is objective, transparent and fair”, as quoted from your recent presentation of the competition summary.

We detail below each aspect that seems seriously problematic in the hope and expectation that NSERC will rapidly correct the situation for the applicants from the recent competition of 2011 or, at the very least, ensure that these issues are adequately resolved in time for next year’s competition.

Lack of fairness: The members of the Evaluation Group (hereafter, the EG) have been deeply concerned by the fact that there are huge funding differences for comparable files in the 2010 and 2011 competitions. Indeed, funding for the same bin in 2010 and 2011 dropped by as much as 40% ($12,000, in this particular case), even though we, as members of the EG, agree that rating standards for the two successive years were not significantly different. Researchers expect to be judged on the merit of their file when compared to the whole community, not only to the applicants in the same competition. Variations such as those observed between 2010 and 2011 are highly unfair.

Responsiveness to merit : In addition to the bin funding discrepancies between 2010 and 2011, we have observed that funding differences between successive bins (from \$1,000 between bins I and J, to \$9,000 between bins D and E) are extraordinarily large and appear, frankly, arbitrary from the perspective of merit. Researchers with remarkable achievements and outstanding potential are provided with insufficient funds to develop and carry out their research projects as a consequence of this extremely unbalanced funding model. As committee members, we were asked to use the full scale in our ratings, and we did so, but with the expectation that scientific merit would ultimately be rewarded with proportional grant amounts. The bin values selected by NSERC distort the rating scale in such a drastic fashion that the lengthy and rigorous evaluation process that took place was rendered almost meaningless.

Transparency: The 2011 competition and its aftermath also raises questions about the transparency of the present system in which critical funding choices are exclusively the purview of the Executive committee of the EG and NSERC staff, and we feel that Isabelle Blain’s letter to the applicants hardly addresses any of the concerns expressed by the EG members and the community.

The Executive of our EG was placed in the very difficult situation to have to choose between two unpalatable and unfair scenarios, a huge drop in success rates from a year to the other (from 62% to under 45%), or reducing drastically bin values in mid-range bins. All along this process as well as after it, for confidentiality reasons, they were unable to consult and explain in a detailed way the whole situation to the rest of the EG.

One example of how the general lack of openness affected the process and its aftermath is related to the decision made by the Executive and NSERC staff to have different funding levels per bin for mathematics and for statistics, even though the two subjects share the same EG. The difference in funding levels was swiftly explained in Ms. Blain’s letter by claiming a different interpretation of the merit criteria. However, there was no comparison of files with the same rankings in mathematics and in statistics during or after the end of the competition to justify such a broad claim – in fact, bins can not even be opened after the end of the competition. After noticing a markedly different bin distribution in mathematics and statistics, the Executive decided to split the budgets between these two groups, a decision justified by the fact that, historically, these two subjects have been separated one from the other in the funding process and the respective subgroups of the EG had worked in complete separation also in this evaluation exercise. As a consequence, the bin values in statistics became higher than in mathematics and reached levels comparable to those in 2010. As the confidentiality principle did not allow the Executive to explain to the EG, not to mention to consult with its members on,the full details of the various scenarios involved, this led to considerable unease in the EG and beyond.

Another example of lack of transparency is apparent in the fact that Ms. Blain’s letter hardly makes any attempt to justify grant funding differences between 2010 and 2011. Her letter indicates that one of the priorities in the current year was to fund the bin J. This is commendable and in keeping with the wishes of our community, as we understand them. But it does not explain why this led to hugely different results from 2010, when the same criteria were applied and the same bin J was funded, with a similar overall success rate. We are distressed that her letter did not refer to any of the other variables entirely in NSERC hands and involved in this outcome : first and foremost the budget, the values of the lowest funded and the top bins, and the gradations between funding amounts for different bins.

As members of the EG, we were involved in a process that is of great importance to our scientific community and, more generally, for the advancement of research and innovationin Canada. We have made considerable efforts to implement the new grant distribution system in ways consistent with the highest scientific standards in a context made particularly difficult by the chronic and increasingly severe underfunding of the mathematical and statistical sciences when compared with other NSERC supported disciplines. We established our ratings with the best of our scientific expertise and we naturally expected this to be reflected accurately in terms of budgetary allocations. This was far from being the case.

As reviewers, we were highly impressed with the quality of the 2011 applicants. Canada, through NSERC, has supported many of them in past years and can take justifiable pride in their achievements. Moreover, the Discovery Grants Program is essential for the progress of science in Canada and, for the outright majority of applicants reviewed by the EG, is the only available source of significant research funding. Unfortunately, the 2011 Discovery Grants Program in Mathematics failed many of the most promising of these applicants. As a consequence, our confidence in the program, as currently administered, is regrettably shaken. We urge you to take rapid corrective action so that the 2012 competition will not suffer from the same problems.

Yours Sincerely,

Mike Bennett (UBC)
Nantel Bergeron (York)
Lia Bronsard (McMaster)
Thomas Brüstle (Sherbrooke and Bishop’s)
Olivier Collin (UQAM)
Benoit Collins (Ottawa)
Octav Cornea (Montreal)
Alan Dow (North Carolina, Charlotte)
Hermann Eberl (Guelph)
Christopher Godsil (Waterloo)
Eyal Goren (McGill)
Robert McCann (Toronto)
Matthias Neufang (Carleton and Fields)
John Stockie (Simon Fraser)
Holger Teismann (Acadia)
Xiaoqiang Zhao (Memorial University)

cc: Ms. Isabelle Blain, Vice-President, Research Grants & Scholarships; Ms. Madeleine
Bastien, Team Leader, NSERC Evaluation Group 1508; Mathematics-NSERC Liaison
Committee.

Originally post on May 8, 2011

The following Canada Research Chairs have signed the
2011 public statement about NSERC Discovery Grants.

1. Alejandro Adem, UBC
2. Heinz Bauschke, UBC Okanagan
3. Nantel Bergeron, York (member 2011 Evaluation Group)
4. Vladimir Chernousov, UAlberta
5. Hugh Chipman, Acadia
6. Walter Craig, McMaster
7. Hermann Eberl, Guelph (member 2011 Evaluation Group)
8. George Elliott, Toronto
9. Jim Geelen, Waterloo
10. Andrew Granville, U. Montreal
11. Pengfei Guan, McGill
12. Adrian Iovita, Concordia
13. Steve Kudla, Toronto
14. Rachel A. Kuske, UBC
15. Francois Lalonde, U. Montreal
16. Mark Lewis, UAlberta
17. Edwin Perkins, UBC
18. Ian Putnam, Victoria
19. Anthony Quas, Victoria
20. Sujatha Ramdorai, UBC
21. Nancy Reid, Toronto
22. Alexander Shirelman, Concordia
23. Patrick Speissegger, McMaster
24. Stevo Todorcevic, Toronto
25. Nicole Tomczak-Jaegermann, UAlberta
26. Bálint Virág, Toronto
27. Jianhong Wu, York

To suggest additions or edits to this list, please send an email message to: nmlcadmin@nmlc.math.ca.

Originally post on May 8, 2011

The following Fellows of the Royal Society of Canada have signed the
2011 public statement about NSERC Discovery Grants.

1. Jim Arthur
2. Martin Barlow
3. Ed Bierstone
4. Tom Bloom
5. David Boyd
6. Walter Craig
7. Ken Davidson
8. Henri Darmon
9. Ivar Ekeland
10. George Elliot
11. Joel Feldman
12. Donald Fraser
13. John Friedlander
14. Nassif Ghoussoub
15. Andrew Granville
16. Pengfei Guan
17. Victor Ivrii
18. Lisa Jeffrey
19. Niky Kamran
20. Francois Lalonde
21. Jerry Lawless
22. Eckhard Meinrenken
23. Pierre Milman
24. Kumar Murty
25. Ram Murty
26. Edwin Perkins
27. Ian Putnam
28. Nancy Reid
29. Gordon Semenoff
30. I. Michael Sigal
31. Gordon Slade
32. Cameron Stewart
33. Mary Thompson
34. Nicole Tomczak-Jaegerman
35. Alfred Weiss

To suggest additions or edits to this list, please send an email message to: nmlcadmin@nmlc.math.ca.

Originally post on May 6, 2011

Statement from the Executive Committee of the Mathematics & Statistics Evaluation Group (EG 1508):

Mathematics & Statistics Evaluation Group 1508, Executive Committee

The Executive Committee is part of, stands with and advocates to NSERC for the mathematical and statistical community. It had the extremely difficult task to choose between different unsatisfactory outcomes. The following principles guided the Executive and NSERC during budget deliberations:

• maintaining success rates in response to the extensive feedback received from the community after the 2010 competition;
• ensuring researchers with strong evaluations across the board receive non-zero funding, again based on extensive feedback received in the past from the community;
• equitable treatment of all sections;
• keeping exceptional researchers funded at internationally competitive levels.

Trying to manage all of the above with the ever increasing pressure on our budget led to very hard decisions. The final scenario provides funding to the maximum possible number of deserving researchers within the above constraints.

While the Executive must abstain from signing the open letter due to the obvious conflict of interest, we will address the issues in a separate communication to NSERC.