Brookings seminar on "Voting Technology: The Not-So-Simple Act of Casting a Ballot"
A couple of weeks ago, there was a lot of publicity around the new book "Voting Technology: The Not-So-Simple Act of Casting a Ballot" by Paul Herrnson et al (Brookings Institution Press). Much of the publicity was focused on the critiques in the book of the need for computer security, including the authors claims that the needs for security are much less important than the need for usability.
This morning, Brookings hosted a panel with four of the six authors of the book. I won't try to summarize their book, other than to say that it's well worth reading about their usability results, some of which are quite surprising. There are serious scientific problems with their work even as far as it goes, but that doesn't take away from the fact that this is one of the first studies with field trials of voting systems. One of the major limitations of their results is that in considering usability, they entirely ignored usability by disabled voters. I had hoped that they would address some in their field trials some of the issues that Noel Runyan's team identified in the California Top to Bottom Review Accessibility Study. However, as Paul Herrnson told me, their funding was less than requested, and this is one of the areas they cut, to the great dismay of Jim Dickson, a leading advocate for blind voters.
The biggest issues I have with this report are as follows:
(1) It states categorically that no elections have been corrupted due to intentional security breaches (i.e., no hacking), so therefore security isn't an issue. While I certainly don't know of any examples of successful security attacks on real elections, there are many cases where there have been accidental problems that have caused incorrect election results. The ironic thing is, of course, that we only know of the ones that did NOT take place on paperless DREs, since if there's no paper, there's nothing meaningful to recount. Although we can't prove incorrect election results on the DREs, I'd bet money that we've had them from accidental errors, if not intentional ones.
Besides, if anyone has successfully caused incorrect election results, one would hardly expect them to brag about it - just as old fashioned ballot box stuffing and switching was well known, but not advertised.
(2) Their primary focus is on whether voters get the votes selected correctly. This is important, but it misses the even more important factor of whether votes are recorded correctly. If the voting system (whether it's a computer, paper, punchcard, or something else) doesn't accurately record what the voter selected, it doesn't matter whether the voter was able to figure out how to use the system.
(3) Their secondary focus is on whether voters feel comfortable with the voting system, and are confident that it worked correctly. As was pointed out by Roy Saltman (author of "The History and Politics of Voting Technology: In Quest of Integrity and Public Confidence", an excellent book on voting systems), while it's important that the voter feel confident, it's less critical than whether the auditors can actually verify the results. He noted that we need to have systems that can be verified, even if that makes it slightly more difficult for voters to vote.
Incidentally, Roy told me his book will be available in paperback this summer for about half the price of the hardback, and Amazon is taking orders.
Norman Ornstein asked which of the technologies they investigated offer meaningful opportunities for recounts, especially given that "optical scan is what people perceive as the right answer". The authors didn’t answer directly, but noted that the real problem is which is the ballot of record in systems where there is more than one form (such as DREs with VVPAT). Herrnson strongly prefers that the electronic ballot fill that role - which of course defeats the purpose of having paper, and guarantees that we'll have more elections where we'll never know who was the real preference of the voters.
Among their recommendations which I agree with are the need for "pre-testing" of ballots to make sure they're not confusing to voters (ala the infamous Florida "butterfly ballot" of 2000 or the Florida 13th Congressional District election of 2006). While this won't help with security issues, it will address many of the problems that plague elections today. I also agree with them that there should be parallel testing, although there's no indication that they understand the limitations of that technology.
After the seminar, I suggested to Herrnson that asking voters about their comfort with the voting systems is like asking patients which of two medical procedures to diagnose a problem is better - trained experts (i.e., doctors) can be expected to answer that question, but the patient can only comment on the patient experience not the test accuracy. I suggested that offering as a choice an invasive procedure vs. a Star Trek-like "magic scanner", most patients would select the scanner as both more accurate and more desirable. He disagreed, saying that most patients would conclude that the invasive procedure is more accurate - if it's uncomfortable, it must be better. He might be right on that one...
Also after the seminar, when I asked Herrnson about his critiques in the press of security, he complained that it was taken out of context - he said he spoke to journalists for as much as an hour, and the part they chose to publicize was the security critique, which he claimed was hardly the focus of his work. I can appreciate that - anyone who talks to the press knows they'll take the "juicy" parts. But at the same time, I think it's unwise for a group of political scientists to be passing judgment on whether computer security is a real problem or not. As with the doctors above, let's leave that to the subject matter experts - the computer scientists.
I had at least a hundred more questions I wanted to ask, but given the constraints of the seminar that wasn't possible. Brookings videotaped the presentation; hopefully viewers will be able to judge for themselves the limits of this study.
[Updated 21 Mar to correct a number of typos and add a link to the video archive.]
This morning, Brookings hosted a panel with four of the six authors of the book. I won't try to summarize their book, other than to say that it's well worth reading about their usability results, some of which are quite surprising. There are serious scientific problems with their work even as far as it goes, but that doesn't take away from the fact that this is one of the first studies with field trials of voting systems. One of the major limitations of their results is that in considering usability, they entirely ignored usability by disabled voters. I had hoped that they would address some in their field trials some of the issues that Noel Runyan's team identified in the California Top to Bottom Review Accessibility Study. However, as Paul Herrnson told me, their funding was less than requested, and this is one of the areas they cut, to the great dismay of Jim Dickson, a leading advocate for blind voters.
The biggest issues I have with this report are as follows:
(1) It states categorically that no elections have been corrupted due to intentional security breaches (i.e., no hacking), so therefore security isn't an issue. While I certainly don't know of any examples of successful security attacks on real elections, there are many cases where there have been accidental problems that have caused incorrect election results. The ironic thing is, of course, that we only know of the ones that did NOT take place on paperless DREs, since if there's no paper, there's nothing meaningful to recount. Although we can't prove incorrect election results on the DREs, I'd bet money that we've had them from accidental errors, if not intentional ones.
Besides, if anyone has successfully caused incorrect election results, one would hardly expect them to brag about it - just as old fashioned ballot box stuffing and switching was well known, but not advertised.
(2) Their primary focus is on whether voters get the votes selected correctly. This is important, but it misses the even more important factor of whether votes are recorded correctly. If the voting system (whether it's a computer, paper, punchcard, or something else) doesn't accurately record what the voter selected, it doesn't matter whether the voter was able to figure out how to use the system.
(3) Their secondary focus is on whether voters feel comfortable with the voting system, and are confident that it worked correctly. As was pointed out by Roy Saltman (author of "The History and Politics of Voting Technology: In Quest of Integrity and Public Confidence", an excellent book on voting systems), while it's important that the voter feel confident, it's less critical than whether the auditors can actually verify the results. He noted that we need to have systems that can be verified, even if that makes it slightly more difficult for voters to vote.
Incidentally, Roy told me his book will be available in paperback this summer for about half the price of the hardback, and Amazon is taking orders.
Norman Ornstein asked which of the technologies they investigated offer meaningful opportunities for recounts, especially given that "optical scan is what people perceive as the right answer". The authors didn’t answer directly, but noted that the real problem is which is the ballot of record in systems where there is more than one form (such as DREs with VVPAT). Herrnson strongly prefers that the electronic ballot fill that role - which of course defeats the purpose of having paper, and guarantees that we'll have more elections where we'll never know who was the real preference of the voters.
Among their recommendations which I agree with are the need for "pre-testing" of ballots to make sure they're not confusing to voters (ala the infamous Florida "butterfly ballot" of 2000 or the Florida 13th Congressional District election of 2006). While this won't help with security issues, it will address many of the problems that plague elections today. I also agree with them that there should be parallel testing, although there's no indication that they understand the limitations of that technology.
After the seminar, I suggested to Herrnson that asking voters about their comfort with the voting systems is like asking patients which of two medical procedures to diagnose a problem is better - trained experts (i.e., doctors) can be expected to answer that question, but the patient can only comment on the patient experience not the test accuracy. I suggested that offering as a choice an invasive procedure vs. a Star Trek-like "magic scanner", most patients would select the scanner as both more accurate and more desirable. He disagreed, saying that most patients would conclude that the invasive procedure is more accurate - if it's uncomfortable, it must be better. He might be right on that one...
Also after the seminar, when I asked Herrnson about his critiques in the press of security, he complained that it was taken out of context - he said he spoke to journalists for as much as an hour, and the part they chose to publicize was the security critique, which he claimed was hardly the focus of his work. I can appreciate that - anyone who talks to the press knows they'll take the "juicy" parts. But at the same time, I think it's unwise for a group of political scientists to be passing judgment on whether computer security is a real problem or not. As with the doctors above, let's leave that to the subject matter experts - the computer scientists.
I had at least a hundred more questions I wanted to ask, but given the constraints of the seminar that wasn't possible. Brookings videotaped the presentation; hopefully viewers will be able to judge for themselves the limits of this study.
[Updated 21 Mar to correct a number of typos and add a link to the video archive.]
posted by Jeremy at
3:06 PM
2 Comments:
It's worth pointing out that the public opinion poll Herrnson et al did for MD's State Board of Elections (SBE) in late 2005, while showing that voters generally "like" voting on touch-screens, also showed that more than 2/3 of the voters they polled wanted a paper trail of their votes.
(Download the pdf at http://www.saveourvotes.org/reports/index.htm#13)
That sentiment has been corroborated in 2 other statewide polls since then by MD's most respected political polling firm, Gonzales Research. The first one showed that 56% of MD voters wanted to vote by opscan compared to 41% who wanted to stick with DREs (April 2006).
http://www.gonzalesresearch.com/Surveys/Maryland_Media_Poll_April_2006.htm
The second one, done last fall just before a special legislative session on the budget, showed that 64% of MD voters wanted the governor to fund the opscan bill passed last spring, with support as high as 74% in some demographic groups.
(Pdf at http://www.saveourvotes.org/reports/index.htm#24)
So the majority of MD voters consider unverifiable elections a problem. I don't know how this compares to other regions of the U.S., but this is the environment in which Dr. Herrnson & his colleagues are working.
The same study he did for the SBE included testing the usability of several verification methods for DREs. In the end he and his colleagues concluded that they couldn't actually recommend any of them, but thought some of them showed promise. (I haven't seen the book, so I don't know if this study is in there.) That study was done 2.5 yrs ago and I doubt that any of that "promise" has matured to the point of viability for any of the ideas they tested, not to mention the additional expense of adding a verification system to 20K DREs on top of our already astronomical voting system costs.
From Tom:
Is there a video archive involved? If so, please post link.
Tom
Post a Comment
Subscribe to Post Comments [Atom]
<< Home