Paper Review
The paper review phase has the following configuration steps:
- Configure the review:
The review can configured in Reviews > Configuration.
- DEADLINE: expected date for reviews to be submitted.
- BLOCK REVIEW, REVIEWERS CANNOT EDIT REVIEW: allows the blocking/unblocking of reviews.
- ANONYMOUS: controls if a TPC member/editor can view the identities of the other reviewers of a paper. By default, if the TPC chair is an author of a paper, the identities of the reviewers are anonymized.
- VISUALIZATION: controls if a TPC member/editor can view only his/her own reviews, the reviews he/she assigned to others, or all reviews. It is also possible to allow authors to view the reviews. This is important in the case the event has a rebuttal phase.
- ASSIGN AND CLAIM OF SUBMISSION: controls if the TPC member/editor can assign reviews to reviewers that are not TPC Members and if TPC member can claim submissions with status 'PENDING'.
- REVIEW CONFLICT RESOLUTION: based on the reviews, JEMS calculates the span of a paper. The span is the difference between the highest and the lowest score of a paper. A large span means there is a conflict. It is possible to define two span limits (A and B). Span values between span limits A and B means conflict. Span values greater than B means serious conflict.
- REVIEW PROCESS: defines the number of reviews required by paper and the number of reviews each TPC member should complete.
- REBUTTAL: allows the configuration of the rebuttal period.
- DISCUSSION MESSAGE: allows the configuration of the discussion period.
After done, click on "Save".
- Configure the review form:
The review form can be accessed in Settings > Review form.
Questions can be added, modified, or removed from the review form. Alternatively, you can import questions from other events. Questions can be defined using three different formats: checkbox, radio buttons, or text.
For checkbox and radio buttons you must define two parameters: a Global Weight and a Weight. The sum of all (global) weights should be either 1 (one) or 0 (zero). These weights are used to calculate the average and the global (weighted) average of the paper, which, in turn, can be used to rank the papers. The average and global average of a paper are calculated as follows:
- Average (Avg) = sum(average(i))/N, where i: {1..N}, and N is the number of reviews of the paper
- Global Average (GAvg) = sum(average(i)*weight(i))/sum(weight(i)), where i: {1..N}, and N is the number of reviews of the paper
The average and weight of the i'th review are given by:
- average(i) = sum(score(j)/max_score * weight_question(j) * 10, where j: {1..M}, and M is the number of questions that have a score in review i
- weight(i) = sum(score(j)/max_score * global_weight_question(j), where j: {1..M}, and M is the number of questions that have a score in review i
The variables weight_question(j) and global_weight_question(j) are defined in the review form for each question.
Example:
Suppose the review form has 3 questions configured as follows:
| Number | Question | Scores | Weight | Global Weight |
|---|---|---|---|---|
| 1 | ORIGINALITY | 1-5 | 0.5 | 0.0 |
| 2 | RECOMMENDATION | 1-5 | 0.5 | 0.0 |
| 3 | CONFIDENCE | 1-5 | 0.0 | 1.0 |
And the reviews are filled as follows:
| Questions | 1 | 2 | 3 |
|---|---|---|---|
| Reviewer 1 | 4 | 4 | 3 |
| Reviewer 2 | 5 | 5 | 5 |
| Reviewer 3 | 5 | 5 | 5 |
The average and weight of each review is calculated by:
average(1) = (4/5 * 0.5 + 4/5 * 0.5 + 3/5 * 0.0) * 10 = 8
average(2) = (5/5 * 0.5 + 5/5 * 0.5 + 5/5 * 0.0) * 10 = 10
average(3) = (5/5 * 0.5 + 5/5 * 0.5 + 5/5 * 0.0) * 10 = 10
weight(1) = (4/5 * 0.0 + 4/5 * 0.0 + 3/5 * 1.0) = 0.6
weight(2) = (5/5 * 0.0 + 5/5 * 0.0 + 5/5 * 1.0) = 1.0
weight(3) = (5/5 * 0.0 + 5/5 * 0.0 + 5/5 * 1.0) = 1.0
| Review | Average | Weight |
|---|---|---|
| 1 | 8 | 0.6 |
| 2 | 10 | 1.0 |
| 3 | 10 | 1.0 |
Therefore,
Avg = (8 + 10 + 10) / 3 = 9.33
GAvg = (8 * 0.6 + 10 * 1.0 + 10 * 1.0) / (0.6 + 1.0 + 1.0) = 9.53
In the example, Reviewer 1 gave the lowest score but this review has the lowest influence in the Global Average of the paper since the score for the question 3 (Confidence) is lower than the other two reviewers.
After editing the review form, click on "Save Form" to save the changes.
- Adjust the e-mail templates that are related to the review phase. Go to Settings > Email Templates. There are 11 templates that can be configured. Some of them are shown in the figure:
- Add or invite TPC members:
TPC members can be invited or added directly to the TPC list. To add a member to the TPC go to People > TPC Members and search for members that are already registered in JEMS. Optionally, a new user can be created and added to the TPC.
Alternatively, TPC members can be added to an invitation list. To do so, search for existing users using the corresponding search bar (or create a new user). An invitation list can also be inherited from another event.
After defining the invitation list, select the members that will receive the invitation e-mail and click on "Send" as shown in the figure.
- Assign reviews to the TPC members:
Reviews can be assigned to reviewers in Reviews -> Assignment Suggestions. You can select the track, a group of papers (or all papers), and a group of TPC members (or all members). You can also indicate high (or low) affinity values to increase (or decrease) the chances of a suggestion to be made based on the match between the reviewer's topics of interests and the topics defined in the submissions. To generate an assignment suggestion, click on "Generate Suggestion".
JEMS has its own assignment suggestion algorithm that considers some conditions, including:
- the reviewer (TPC member) had bidden/claimed the paper
- the paper is not yet assigned to the reviewer
- the reviewer is not one of the paper's authors
- the reviewer is not on the conflict list of any of the paper's authors
You have the option to refine the suggestions made by JEMS by manually selecting the reviewers for each paper. After making the changes click on "SAVE".
- Notify reviewers
Once reviews are assigned, reviewers have to be notified about them. To notify reviewers, go to Reviews > Notify Reviewers. JEMS lists reviewers that were assigned but not yet notified. Reviewers that assigned to review papers that are (co) authored by the TPC chair are shown as "Anonymous reviewer". Notification emails are generated using the templates previously defined in step 3. To notify the reviewers click on Notify reviewers. Each reviewer on the list will receive an email generated according to the template.
Reviewers can also be notified for reviews that are pending, i.e., unfinished but within the deadline, or for late reviews. Also, the reviewing process can be followed using the functions Review > Review status by submission and Review > Review status by user:
Next phase: Rebuttal and Discussion phase