S-value Revision
During Summer 2009, NAQT plans to revise the S-value formula it uses to rank teams based on their performance at Sectional Championship Tournaments (SCTs) and thereby determine the majority of the invitations for its Intercollegiate Championship Tournament (ICT).
The primary goal is to come up with a new system that is not vulnerable to unsportsmanlike manipulation and which can, therefore, be made public. A secondary goal is improving the rankings produced by the system.
NAQT hopes to work with members of the quiz bowl community during the summer to devise the new system; people interested in helping out (or in just throwing out an idea or two) should contact NAQT at naqt@naqt.com to offer their services. NAQT will probably establish a mailing list and/or public forum for the project once the depth of interest has been gauged.
As a starting point, here are the characteristics that NAQT is looking for in its new system:
Critical characteristics:
The system should result in an unambiguous ranking of all Division I SCT teams and all Division II SCT teams such that we could just invite the teams in the order listed with no further manipulation.
The system should not be susceptible to gaming by coaches, players, or hosts. That is, no non-obvious degenerate behavior should be able to significantly improve a team's S-value. "Degenerate behavior" is any behavior that intentionally reduces the chance of a team winning an individual game. "Non-obvious" behavior is defined in contrast to that which can be objectively detected by staff observation (e.g., a team telling the other the answer to its bonuses) or by analysis of the statistics. We may be willing to live with a system that can be manipulated by "obvious" degenerate behavior under the assumption that it would be detected and prohibited by tournament staff.
The system must be able to handle combined fields where Division I and Division II teams compete against each other due to fewer than four teams being present in one division. This implicitly includes accounting for the fact that some teams may not hear the set formally designated for their division.
The system must work well with any number of teams and any number of SCTs.
The system must work well when there is a wide disparity in average ability among SCTs.
The system should only use data collected from SCTs.
The system should allow for the calculation of S-values for community college teams after the completion of the Community College Championship Tournament (CCCT) and the easy addition of community college teams to the Division II invitation list.
The system should be able to be made public along with all relevant raw data and calculations for each year.
Valuable characteristics:
We would like to use the same system for Division I and Division II.
We would prefer to not require extensive collection of additional data by scorekeepers or hosts.
Numerically close S-values should imply similar team abilities.
The system should generally respect the order-of-finish at a given SCT; that is, if team A finished higher than team B at an SCT, the system should almost always rank team A higher.
Nice-to-have characteristics:
Mathematical simplicity
The system should make use of already existing, clearly understood statistical measures (like points-per-bonus).
S-values should be comparable from year to year.
The system should be as robust and fair as possible in the case where a team needs to be retroactively disqualified after invitations have been issued and (possibly) accepted.
We are specifically looking for a system to issue ICT invitations, not a system to seed teams for the ICT. We will definitely consider S-values when seeding the ICT, but we will consider lots of other information (roster changes, host teams, etc.) when making that decision.
Finally, NAQT wants to highlight what it expects will be the three most difficult parts of assembling this system:
The implicit problem that occurs when Team A finishes ahead of Team B at one SCT but has inferior statistics while Team C at another SCT has stats that are better than those of Team A but worse than those of Team B. We want to "almost always" invite Team A ahead of Team B, but it's hard to say where Team C should fall in the invitation order. This is a relatively common occurrence in SCT data; it's not rare for there to be several teams in Team C's position.
Truly eliminating opportunities for manipulating the system. For example, any system that uses tossup data would have to contend with the possibilty that a team would choose to forfeit its match against a strong team under the assumption that they were going to lose anyway and forfeiting would prevent their PPTH from being lowered. On the other hand, NAQT does not want to penalize teams that forfeit matches for valid reasons.
Handling combined divisions; how does 20 PPB on Division II questions compare to 15 PPB on Division I questions?
NAQT has presented an S-value revision proposal based on the feedback to this initiative.