Testing the eSTV program for the Scottish Local Government Elections

Abstract

This report deals with the testing of eSTV software from ERS, Electoral Reform Services. The purpose of the test is to determine the conformance of eSTV to the WIG-rule [1]. In order to determine conformance we have created a new implementation of the rule in the functional programming language Clean [4]. Since this language uses a different programming paradigm and the implementation is made completely independent from the software to be tested it is extremely unlikely that both implementations contain the same error. Testing is done by computing the election result for a given set of votes with both programs. The results of both programs are compared to the level of votes numbers after each selection and elimination step in the algorithm. This has shown to be a very sensitive test for very small variations in the algorithm used in the programs. These tests are executed for 808 standard test cases. These test cases are partly handcrafted or the vote distributions in real elections for similar voting rules. Each of these sets is computed with 4 different elimination orders. In addition the results of 5000 generated data sets and a number of hand crafted data sets covering border situations in the algorithm are compared. Both test sets have shown to be very effective to identify even the smallest changes in the algorithm to compute the result of the election. Testing of preliminary versions of eSTV showed some small inaccuracies in the program as well as some small space for different interpretation of the rules. The Scottish Executive provided the final interpretation of the rules (as detailed in section 7.1) and all test where repeated for eSTV version 2.0.16. In these final tests there was an exact match for each of the 4 × 808 + 5000 = 8232 test cases.