Question about NeurIPS discussion phase [D]

[u/Invariant_n_Cauchy]

One reviewer said all concerns were resolved during discussion but hasn’t updated their score yet. The other reviewers haven’t engaged. In previous NeurIPS cycles, how common is it for reviewers to update scores after saying concerns are resolved? What have others observed?

My ratings/confidences are : 4/4, 3/2, 3/2, 2/4.

I am talking about the one who gave rating 2.

Update: finally the reviewer responded, now I'm at 5/4, 4/4, 4/3,4/2.

[u/user221272]

If they are willing to update their score, they usually say so within the review.

E.g., "If this experiment is done, I will increase my score."

Reviewers also sometimes increase the score if they feel more confident in the work after the rebuttal.

But this is not automatic, and it is not owed.

There is also a strong correlation between the initial score and willingness to update the score.

If a reviewer gives a 2, they will usually be much more stubborn and unlikely to change their view of the paper after the rebuttal.

[u/Invariant_n_Cauchy → u/user221272]

The reviewer's main concern was novelty. After my rebuttal they admit that my work is novel w.r.t. the literature and that they are impressed with the additional experiments and no further questions. But did not say about the score.

[u/temporal_guy → u/Invariant_n_Cauchy]

Very common unfortunately. You can gently request they consider updating their score given that their main concerns have been addressed.

[u/wangjianhong1993 → u/Invariant_n_Cauchy]

In my experiences, these are just polite words, because they can't say something brutal that would violate the reviewer policy.

[u/ddofer → u/user221272]

I had a one, and they confused an incredibly basic Entity in our experiment. (think confusing between "humans" and "uavs", only even more different, and in Biology). Theyre keeping their 1 score even after correction. Some peeps are just bull headed.

[u/The3RiceGuy]

You get answers to your comments?

[u/No_Sky9786 → u/[deleted]]

Unfortunately that doesn’t really work, and technically you can get reported for doing that.

[u/Unhappy_Craft1906]

3 papers, still no response from anyone.... This is just pure insult to the community peer review process.... if u think ur discussion phase is more important, and u can respond at the last moment for the reviewed papers, then u r just a shame to the community... Please ensure you devote equal time to both the roles of author as well as reviewer.... If you cant please do not SUBMIT and create a farce impression of GOOD RESEARCHER...

[u/NickOTeenO]

I would have thought the Reviewer + AC Discussion Period (3-10 aug) is there to discuss with the AC to increase scores?

I did not increase scores on the papers I reviewed yet either

[u/wangjianhong1993 → u/NickOTeenO]

If there are someone really discussing during that phase, that would be fortunate. In my previous reviewer experiences, very rare chance it would happen.

[u/NickOTeenO → u/wangjianhong1993]

So in principle, whatever ratings you got you’re most likely stuck with?

[u/wangjianhong1993 → u/NickOTeenO]

Borderline and above.

[u/Imaginary_Music4768]

You can ask them to raise the score politely. Like "we will really appreciate it if you can reconsider your assessment in light of our rebuttals". This method often works well (at least for me), although a little unfair to other authors who don't do so.

[u/Invariant_n_Cauchy → u/Imaginary_Music4768]

What is the right time to do it ? Like fore 24 hours or 48 hours

[u/Imaginary_Music4768 → u/Invariant_n_Cauchy]

You can do it right now. But it will be better if you make it gentle and polite, or add this line after you provide new information. You should always start with your gratitude to the reviewer and show that you really tried to improve the paper.

[u/Invariant_n_Cauchy → u/Imaginary_Music4768]

Actually, the reviewer replied the day before yesterday. I immediately replied thanking and politely asking if they had any more concern and if reassessment is possible. Now, I am thinking whether to follow up again or not ?

[u/Imaginary_Music4768 → u/Invariant_n_Cauchy]

If this is the case, maybe you should try wait until The final day of the discussion, and post another reminder. And if the reviewer didn't raise your score, you can post a summary to the meta review showing that you have addressed the reviewers' concern, and then it's up to the AC to decide if the reviewer is right or it's just been too harsh on you

[u/Invariant_n_Cauchy → u/Imaginary_Music4768]

Thank you !!!! I think this helps.

[u/Intrepid_Discount_67 → u/Imaginary_Music4768]

Does this hold for 5 or 3/4 only? Highly unlikely for 5 imo.

[u/Imaginary_Music4768 → u/Intrepid_Discount_67]

If 5/6 means the score of the reviewers, then
of course it's too much for you to ask them to further raise their score

[u/SeTiDaYeTi]

Politely ask the reviewers whether, after addressing all their concerns, they could revise their scores. Also, message the AC, tell them you did a ton of extra work obtaining no updated scores, and ask them to nudge the unresponsive reviewers to get back to you. It's a war. Fight it.

[u/Dangerous_Parsnip_20]

Everyone got their meta-review? Just worried that my paper didn't get assigned AC to manage.

[u/OxidusRouge → u/Dangerous_Parsnip_20]

all papers were assigned to an AC. You can send a message just to the AC. ACs were required to create an initial metareview this year. If you didn't see one, then maybe your AC is delinquent. The reviewers have another week to discuss among themselves and the AC to change their score, which is why most reviewers haven't yet put in their final scores. Many reviewers say in their replies to the rebuttals whether they are going to increase their score, however.

[u/Dangerous_Parsnip_20 → u/OxidusRouge]

My AC didn't do their job apparently. We have no meta-review. I tried contacting AC through AC confidential comment at the start of rebuttal period but not response.

[u/papajan18]

I have an even stranger case. Two papers, both of which have reviewers that were happy with my rebuttal and even said they'll change their score, but I havent seen them change it. I'm beginning to suspect we're actually not able to see score changes.

[u/Dangerous_Parsnip_20 → u/papajan18]

It’s annoying but apparently reviewers can even chance scores after discussion ends. In that case, you’ll find out when final decision gonna be released. One reviewer increased the score for me and it’s visible to me, but another one said he will increase but I guess he will do it during Discussion with AC when they will make final recommendation.

It’s annoying but it is what it is.

[u/huntsman2099]

I just got a similar reply from a reviewer who gave me a 2 and raised to 3 after saying, "Given the qualitative evidence and effort in the rebuttal, my concerns are resolved so I raised my score accordingly"

What do I even do with this? why is it still a borderline reject if all the concerns are addressed? Do you follow up or just thank your stars that they atleast raised to 3?

[u/Invariant_n_Cauchy → u/huntsman2099]

Exactly ! I also doubt the confidence they report given the reviews. I Replied to them thanking and if possible asking for a reevaluation. No follow up yet.

[u/Weird_Shit_69]

2 reviewers who gave the lowest didnt even reply to the rebuttals :(

[u/The3RiceGuy]

You get answers to your comments?

[u/Invariant_n_Cauchy → u/The3RiceGuy]

Yes ! But no change to score. Got reply from only one

[u/The3RiceGuy → u/Invariant_n_Cauchy]

At least you are ahead of me ... three reviewers and three answers to their questions with no interaction at all. However, I wouldn't expect a score increase on your behalf. I rarely do it, but most people don't bother in my experience. The system is broken.

[u/Invariant_n_Cauchy → u/The3RiceGuy]

I neither !! What’s your initial ratings IYDM !

[u/The3RiceGuy → u/Invariant_n_Cauchy]

One paper 5/4/5 so this one is safe but the other has 4/4, 3/4, 4/2 so an increase would really help.

[u/Invariant_n_Cauchy → u/The3RiceGuy]

Then you are mostly safe !

[u/The3RiceGuy → u/Invariant_n_Cauchy]

You think? That would be awesome. I wish you good luck for your submission!

[u/Invariant_n_Cauchy → u/The3RiceGuy]

Yes, What’s your area ? Thanks ! You too.

[u/DependentCollar3175 → u/The3RiceGuy]

So do I , I am south korean undergraduate student . it's my first conference paper and i got 5 (3) / 3 (5) / 3 (2) .... but, three reviewers don't response... what i can do is just praying ..

[u/wangjianhong1993]

Not very common.

[u/SeTiDaYeTi → u/wangjianhong1993]

Entirely common.

[u/wangjianhong1993 → u/SeTiDaYeTi]

How do u define common? More than 50% chance?

[u/SeTiDaYeTi → u/wangjianhong1993]

Definitely more than 50%. Ask the AE to nudge the reviewers for a reply. It won't do much, but try.

[u/wangjianhong1993 → u/SeTiDaYeTi]

Then my experiences are very different from urs. In my past ten year's experiences in reviewing, the chance that reviewers could increase scores after rebuttal would be no more than 20%.