EuroVis 2023 Reviewer Guidelines

Reviews have a direct and important impact on the quality of a conference. They also help the community as a whole to improve the quality of its research. The aim of this guide is to ensure that the review process is fair and leads to the best possible selection of papers. All reviewers should make sure that they follow the basic principles outlined in this text.

Be Timely

The EuroVis reviewing process has a very tight schedule, and an important part of making the commitment to review is that you have agreed to carry out the work before the stated deadlines. All reviewers need to finish reviews on time, and program committee members need to bid and declare conflicts on time. We expect IPC members to do everything in their power to meet these deadlines. It is the IPC member’s responsibility to inform the papers co-chairs of any unexpected problems in meeting the deadlines as soon as possible. On-time reviewing and participation in the discussion phase is a core requirement of the high-quality EuroVis review process.

Writing Reviews and Recruiting External Reviewers

We expect reviewers to write the reviews themselves, to the best of their knowledge in the field and according to the guidelines set out in this document. Delegation of the review duties is not allowed, because IPC members are carefully selected and vetted, and should not be substituted by another individual without the chairs’ approval. The assigned reviewers are accountable for the quality of their reviews. IPC members, besides writing their own reviews, are responsible for recruiting qualified external reviewers. IPC members are responsible for checking the qualifications of external reviewers to the best of their knowledge, including documented research experience in the area of the submission, and check for any possible conflict of interest in advance. We emphasize that the PCS reviewer search functionality can be used to identify candidate reviewers, but the qualification still needs to be carefully verified by the IPC member. The research experience is typically demonstrated by one or more previous publications in the respective area in relevant conferences or journals, and would imply at least senior PhD student level. External reviewers must not be from the same research institution or former research group of the recruiting IPC member. The recruiting IPC member is responsible for securing one external reviewer and following up with the assigned reviewers, including acceptance of the invitation and delivering a high-quality review in time.

Protect Ideas

As a reviewer for EuroVis, you have the responsibility to protect the confidentiality of the ideas represented in the papers you review. Submissions are by their very nature not published documents. The work is considered new or proprietary by the authors; otherwise they would not have submitted it. Of course, their intent is to ultimately publish to the world, but most of the submitted papers will not appear in the proceedings of the conference. Thus, it is likely that the paper you have in your hands will be refined further and submitted to some other journal or conference, or to the same conference next year. Sometimes the work is still considered confidential by the author’s employers. These organizations do not consider sending a paper for review to constitute a public disclosure. Protection of the ideas in the papers you receive means

Do not show the paper, submission videos or any other supplemental material to anyone else, including colleagues or students, or to help with your review.
Do not use ideas from papers you review to develop new ones.
After the review process, destroy all copies of papers and supplemental material that are not returned to the senior reviewer and erase any implementations you have written to evaluate the ideas in the papers, as well as any results of those implementations.

Protect the Integrity of the Review Process

The quality of the EuroVis review process depends on reviewers and authors respecting its integrity. It is the duty of the reviewers to protect the integrity of the review process. This implies not disclosing any information obtained explicitly or implicitly during the review process, including but not limited to reviewer assignments, reviewer comments, discussion phase comments, author rebuttal comments, and so on to persons outside of the set of assigned reviewers and papers co-chairs. Reviewers will never directly contact authors or reveal their or other reviewer identities to the authors. In case of any question or unclarity regarding the integrity of the review process, reviewers should contact the papers co-chairs.

Avoid Conflicts of Interest

As a reviewer of a EuroVis paper, you have a certain power over the reviewing process. It is important for you to avoid any conflict of interest. Even though you would, of course, act impartially on any paper, there should be absolutely no question about the impartiality of review. Thus, if you are assigned a paper where your review would create a possible conflict of interest, you should return the review and as soon as possible inform the papers co-chairs. Conflicts of interest include (but are not limited to) situations in which

You are a co-author of the work.
You have a strong affiliation with the same institution as one of the authors. This includes (but is not limited to) your current employment as a professor, adjunct professor, visiting professor, or similar position, in the role of a consulting or advisory arrangement, previous employment with the institution within the last 12 months, being considered for employment at the institution, any role as an officer, governing board membership, or relevant committee, or the current enrollment as a student.
You have been directly involved in the work and will be receiving credit in some way. If you’re a member of the author’s thesis committee, and the paper is about his or her thesis work, then you are involved.
You suspect that others might see a conflict of interest in your involvement. For example, even though Microsoft Research in Seattle and Beijing are in some ways more distant than Berkeley and MIT, there is likely to be a perception that they are “both Microsoft” and folks from one should not review papers from the other.
You have collaborated with one of the authors in the past five years. Collaboration is usually defined as having written a paper, book or grant proposal together, although you should use your judgment beyond these specifics.
You were the MS or PhD advisor of one of the authors or the MS/PhD advisee of one of the authors. Funding agencies typically consider advisees to represent a lifetime conflict of interest.
You are related to one of the co-authors. This includes (but is not limited to) being the spouse, a child, sibling, or parent, as well as any affiliation or relationship of your spouse, of your minor child, of a relative living in your immediate household or of anyone who is legally your partner that you are aware of.
Other relationships, such as close personal friendship or significant animosity between you, that you think might tend to affect your judgment or be seen as doing so by a reasonable person familiar with the relationship.
The blind reviewing process for some of our conferences will help hide the authorship of many papers, and senior reviewers will try hard to avoid conflicts. But if you recognize the work or the author and feel it could present a conflict of interest, send the paper back to the senior reviewer as soon as possible so he or she can find someone else to review it. If you are in doubt about any conflict, you should discuss it with the papers co-chairs or the person that assigned the paper to you. You should never contact the authors directly.

Be Specific

The publishing of scholarly work is essential for the academic community specifically and the research community in general. Therefore careers and reputations hinge on publishing in the proceedings, academic tenure decisions are based on the proceedings, and patent infringement cases discuss whether something is considered novel enough to publish in the proceedings. Therefore, it is our duty to do a careful, objective and scholarly review. The emphasis of our reviews should be to help the authors on how to improve their work and therefore to improve the overall quality of the work in our research community. In general, it will not be helpful to anyone – neither the program chair nor the authors

to do a quick or superficial review, to say that the work is good or bad without giving clear reasons;
to state that the work has been done before without giving clear citations of previous work;
to complain about the structure of the paper without making suggestions on how to improve it;
to dismiss the evaluation method without being specific about the flaws.

A casual or flippant review of a paper that the author has seriously submitted is not appropriate and may be rescinded from the reviewing process. In the long run, casual reviewing is the most damaging type of attack on our conference. Note that it is a big mistake to take on too many reviewing duties, and then not back out early enough (e.g., as soon as you are invited) to allow recovery. If you cannot do a decent job, give the paper back and say so. It is your responsibility to do it early so that the senior reviewer has time to select another reviewer before the deadline.

Be Helpful and Constructive

Have an open mind, or at least reveal your biases. If you are a die-hard algorithm-driven researcher, and you are assigned a user study paper, and the call for papers welcomes both types of papers, don’t bash the paper simply for its methodology. It’s not fair and it benefits no one. Either assess the paper according to the appropriate user-study standards, or admit that you are not capable of doing so. At most, you might discuss why the algorithmic approach does not provide a suitable answer to the research question. But don’t force the author to ask a different question that can be answered using algorithmic methods. Many of us have been frustrated by reviews in which the judge basically told the researcher to do a different study. Work within the author’s goals.

Look for the Good in a Paper

No matter how much you dislike the paper, try to find the good aspects. Perhaps there is a different approach to a problem, a novel application, a promising evaluation methodology, or even a potential twist which could be studied further during the revision of a paper. Be helpful to the authors and point out the potential positive aspects. Let the authors know if you think their work would be better suited for a different conference, journal or venue. Be encouraging.

Be Tactful

Belittling or sarcastic comments may help display one’s wit, but they are unnecessary in the reviewing process. The most valuable comments in a review are those that help the authors understand the shortcomings of their work and how they might improve it. If you intensely dislike a paper, give it a low score. That makes a sufficient statement. While we will assure the anonymity of the reviewing process, read over your review and ask yourself whether you would be able to recite your critique in front of the authors. If the answer is ‘no’ you might want to work on some of the wording of the criticism. A scathing review is usually not only unhelpful to the authors and paper chairs, but it also tends to alienate typically new researchers that would like to enter into our community. Put out a helping hand.

In Summary

Adherence to ethics sometimes requires a more careful analysis of the work, as well as a well-thought out response. However, whereas doing so requires additional time and energy, this expense will lead to an improved culture in our community overall, by helping each other to produce better results and by an overall increase in the quality of our conferences and journals. That is what we are striving for in our visualization community and it is well worth the effort.

Download the reviewer guidelines.

References

[1] Tutorial on How to Evaluate and Communicate VIS Research, Contributions by B. Lee, P. Isenberg, J. Stasko, D. Weiskopf, R. Maciejewski, https://docs.google.com/presentation/d/1MH7ylrHFy3KlXAGJtDX2hQsyGkQN51PuXIVei-txUFI/

[2] How to review papers by N. Elmquist, https://sites.umiacs.umd.edu/elm/2015/12/19/how-to-review-hcivisualization-papers/ with further helpful links.

[3] Writing Technical Reviews by Greg Turk, https://web.archive.org/web/20180124233323/http:/www.siggraph.org/s2008/submissions/juried/papers/review_writing.php