Comment 087

"I believe the most important objective should be to spread the review burden across the year. A system

that achieved this but didn't achieve anything else would still add a lot of value. Assuming a fixed

(and identifiable) pool of high-quality reviewers, spreading across the year increases the probability

that a paper gets a high-quality review. In my experience as an action editor, ARR does not currently

achieve this - some months I get 10+ papers, other months I get zero. For better or worse, this is probably

not caused by infrastructure limitations but is more organisational - something like Jonathan Kummerfeld's

suggestion could be a step in a good direction. I am not sure why ""frequent deadlines"" and ""revise-and-resubmit""

are always coupled together, we could have frequent non-conference-specific deadlines without any support

for revise-and-resubmit. The most painful part of the process for me as an action editor is dealing

with large numbers of missing reviews and the last-minute rush to find emergency reviewers. This also

causes an uneven distribution of labour and stress over time. The problem seems to be much worse with

ARR than it was with traditional conference reviewing (perhaps because reviewers traditionally opt in?).

We need to find a way to know early whether some reviewers are not going to submit reviews, and have

some accountability for reviewers that fail to submit reviews without informing AEs early. Extreme idea:

if you fail to provide assigned reviews for N cycles, you are prevented from submitting a paper for

the next M cycles. Overall it seems incorrect to conclude that because a particular implementation of

rolling review could have been launched better, rolling review as an approach is flawed. I also do not

believe it would be better to throw everything out and invent yet another system which would only get

user feedback after a number of years (one version of Hal's proposal). I would suggest: - A formal review

of everything we have learned from the ARR launch, conducted by a team that includes some community

members that did not work on ARR - A strict timeline for the analysis (e.g. 2-3 months) - A proposal

for where we need to get to (do we know what the desirable end state is? if not, how do we identify

it?) and an analysis of whether the current ARR platform is the right starting point - A consultation

period open to the full ACL membership - A final decision endorsed by the ACL Exec and other relevant

stakeholders (e.g. conference PCs). ""Use a rolling review system for reviews, but not meta-reviews

(which would then be handled by an Area Chair within the conference)"" -> it's critical that area chairs/action

editors can interact with reviewers, e.g. to support reviewer discussion and to push reviewers to improve

their reviews. This proposal seems likely to prevent this (e.g. the AC may be looking at a paper that

was reviewed six months previously and the reviewers may have forgotten the details). ""Develop mentoring

for both reviewers and action editors"" -> a noble objective, but who does the mentoring? Good mentoring

can be a lot of work. I'm not sure whether I support or oppose without some detail about implementation.

""By default, authors sign up to be considered for reviewing assignments. They must opt-out if they

are unable to review."" -> This seems too extreme. Some coauthors may be very junior, or not from the

field - putting them on the spot adds friction. Maybe there could be a requirement that for every paper,

at least one author must confirm that they will review? "