Submission review/Maintenance

From Wikimania

This page is used to help assist with the maintenance of the Submission Review page. The Submission Review page is used to track submissions to Wikimania and provide transparent scoring of submission by members of the Programme Committee. The page is primarily maintained by members of the Programme Committee, but Conference organizers and Wikimedia Foundation staff may also conduct maintenance.

Initial setup

As the call for submissions is going out, the Submission review and Programme Committee pages should be setup.

Pages to import and update

These pages should be imported from the previous year's Wikimania wiki, and updated accordingly.

  • Submission review
    • Will need to be updated to reflect any changes to the number of tracks.
  • Submission Review/1
    • Previous year's entries should be removed—and a blank template example should be left at the top. A subpage should be setup for each individual track for this year's Wikimania. Here is an example of a blank default:
|-
| --
| --
| --
| <!-- Marc André Pelletier --> || <!--Phoebe Ayers  --> || <!-- Deror Avi--> || <!-- Guillaume Paumier --> || <!-- Ellie Young --> || <!-- Caroline Becker --> || <!-- Nicolas VIGNERON--> || <!-- Léa Lacroix --> || <!-- Kiril Simeonovski --> || <!-- Orsolya Virág --> || <!-- Dumisani Ndubane --> || <!-- Jean-Philippe Béland  --> || <!-- Final --> || <!-- Comments --> || <!-- Submitter attending? -->
  • Submission review/Disqualified
    • Previous year's entries should be removed—and a blank template example should be left at the top. Here is an example of a blank default:
|-
| --
| --
| --
| {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || <!-- Comments --> || <!-- attending? -->

Pages to import

These pages should be imported from the previous year's Wikimania wiki, and will most likely not need to be updated further.

MediaWiki pages to update

These pages will need to be updated to reflect changes specific to submission pages.

  • MediaWiki:Edittools
    • Categories for this year's tracks updated.
  • MediaWiki:Common.css
    • These additions made to support templates used:
/*******************
** Default style for PrettyTextBox and InformationBox class of templates
** ie, {{warning}}, {{info}}, etc
*******************/
.InformationBox {
	width: 80%;
	clear: both;
	margin: 0.5em auto;
	padding: 3px;
	border: 1px solid#999;
	background: #F8F8F8;
	color: #000;
}
 
.PrettyTextBox {
	padding: 0.2em;
	border: 1px solid#AAA;
	background: #F9F9F9;
	color: #000;
}
 
/* Submission review */
.review-0 { color: white; background-color: maroon;}
.review-1 { color: white; background-color: red;}
.review-2 { color: white; background-color: orangered;}
.review-3 { color: white; background-color: salmon;}
.review-4 { color: black; background-color: white;}
.review-5 { color: black; background-color: #cec;}
.review-6 { color: black; background-color: lightgreen;}
.review-7 { color: black; background-color: greenyellow;}
.review-8 { color: black; background-color: lawngreen;}
.review-9 { color: black; background-color: lime;}
.review-10 { color: black; background-color: limegreen;}
.review-m { color: ghostwhite; background-color: gray;}

Initial review of submissions

As submissions are coming in, the Programme Committee should not begin scoring them, as people may still edit their submissions. However, work should begin on doing initial review of submissions and entering them into the Submission review page once the submissions appear ready for review. Again, the review for scoring should not yet begin until the submission deadline passes.

The initial review of submissions should consider if submissions are complete or not. While submitters may still edit pages, the initial review is generally enough to determine where to categorize the submission or provide some initial feedback to the submitter to warn them if their submission may not qualify.

Tagging submissions that are not yet completed

If a submission is not yet complete, the submitter may still be working on it, but sometimes they are not aware that something is missing from their submission, or other problems which may disqualify their submission.

To notify a submitter of a problem, replace the {{Open submission}} template automatically placed in every submission with the appropriate template:

{{Blank submission}}

Template:Blank submission

{{Incomplete submission}}

Template:Incomplete submission

{{Off-topic submission}}

Template:Off-topic submission

{{User page submission}}

Template:User page submission

Processing submissions that could be considered complete

While submissions can be updated through the submission deadline and should not yet be scored, if a submission appears complete enough to be ready for review by the Programme Committee and would not otherwise be disqualified, these updates to the page should be made:

  1. Replace the {{Open submission}} template automatically placed in every submission with {{Under review}}.
  2. Update the appropriate Submission review subpage for the track the submission corresponds to with an entry for the submission and assign the submission with the next sequential number available in that track. Example entry: 
    |-
| 4931
| [[Submissions/Example submission|Example submission]]
| [[User:Example]]
| <!-- Marc André Pelletier --> || <!--Phoebe Ayers  --> || <!--Deror Avi  -->  || <!-- Guillaume Paumier --> || <!-- Ellie Young --> || <!-- Caroline Becker --> || <!-- Nicolas VIGNERON--> || <!-- Léa Lacroix --> || <!-- Kiril Simeonovski --> || <!-- Orsolya Virág --> || <!-- Dumisani Ndubane --> || <!-- Jean-Philippe Béland  --> || <!-- Final --> || <!-- Comments --> || <!-- Submitter attending? -->
  1. Under "Submission no." replace <!-- will be added by the Programme Committee--> with the submission number created in the previous step. Example entry: 
    : 4931

Scoring submissions

Possible review scores
0 Out of Scope (OS) (R0)
1 Strong Reject (SR) (R1)
2 Reject (R) (R2)
3 Weak Reject (WR) (R3)
4 Neutral (N) (R4)
5 Weak Accept (WA) (R5)
6 Weak Accept (R6)
7 Accept (A) (R7)
8 Accept (R8)
9 Strong Accept (SA) (R9)
10 Exceptional (R10)

Once the submission deadline has passed, members of the Programme Committee can begin scoring submissions by reviewing ones should have been added to the Submission review page, editing the appropriate Submission review subpage that corresponds with the submissions track, and replace their name with the rating based on a 1–10 scale (see chart to the right) with the template that corresponds to the score they wish to provide the submission. For example, to give a submission a 9, or "Strong Accept", rating replace your name (i.e. <!-- Your name -->) with {{R9}} or {{SA}}.

Abstaining due to a conflict of interest

If you cannot vote for a submission due to a conflict of interest, replace your name (i.e. <!-- Your name -->) with {{COI}}.

Processing late submissions

Any submissions posted after the deadline has passed should be marked as late by replacing the {{Open submission}} template automatically placed in every submission with {{Late submission}}.

{{Late submission}}

Template:Late submission

They should also be added to the Disqualified subpage. Here is an example entry on that page: 

|-
| {{DQ}}
| [[Submissions/Example late submission|Example late submission]]
| [[User:Example]]
| {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || <!-- Comments --> || <!-- attending? -->

Processing disqualified submissions

Once the submission deadline has passed, any submissions which are disqualified should be marked as late by replacing the {{Open submission}} template automatically placed in every submission or the {{Under review}} template placed by a Programme Committee member with {{Disqualified submission}}.

{{Disqualified submission}}

Template:Disqualified submission

They should also be added to the Disqualified subpage. Here is an example entry on that page: 

|-
| {{DQ}}
| [[Submissions/Example disqualified submission|Example disqualified submission]]
| [[User:Example]]
| {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || {{DQ}} || <!-- Comments --> || <!-- attending? -->

Processing submissions which are withdrawn

Once the submission deadline has passed, any submissions which are withdrawn by the submitter should be marked as withdrawn by replacing the {{Open submission}} template automatically placed in every submission or the {{Under review}} template placed by a Programme Committee member with {{Withdrawn submission}}.

{{Withdrawn submission}}

Template:Withdrawn submission

If a submission is withdrawn after the submission deadline has passed and scoring of the submission has begun, replace each Programme Committee vote slot with {{WD}}. Here is an example updated entry for a submission withdrawn by the authors: 

|-
| 3981
| [[Submissions/Example withdrawn submission|Example withdrawn submission]]
| [[User:Example]]
| {{WD}} || {{WD}} || {{WD}} || {{WD}} || {{WD}} || {{WD}} || {{WD}} || {{WD}} || {{WD}} || {{WD}} || {{WD}} || {{WD}} || {{WD}} || {{WD}} || <!-- Comments --> || <!-- attending? -->
Retrieved from "https://wikimania2017.wikimedia.org/w/index.php?title=Submission_review/Maintenance&oldid=20642"