7/27/2009 Ballot images available
Ballot images from the May 19, 2009 special election are
available from the Elections Department on DVD or they can be
downloaded in zipped files from the Election Administration Research Center at UC Berkeley.
I have also uploaded ballot images from the precinct where I vote. Here is the link
SOS report: Numerous 'deficiencies' in elections software
Thadeus Greenson/The Times-Standard
Posted: 03/05/2009 01:31:44 AM PST
The elections system software error that dropped almost 200 ballots
from Humboldt County's final November elections results is only one of
a number of problems with the system, according to a recently released
report from Secretary of State Debra Bowen's office.
In addition to the programming error, the report found three
“deficiencies” in the system's audit logs that place
incorrect data-entry time stamps in the logs, allow operators to delete
decks of ballots without a paper trail and even allow system operators
to, intentionally or inadvertently, erase the system's audit logs.
Discovery of any of the deficiencies could have warranted the system's failing of independent testing, according to the report.
Bowen has called a March 17 public hearing to consider withdrawing the
state's approval of the particular version of the Premier Elections
Solutions software in question, GEMS version 1.18.19. The integrity of
that antiquated version of the software, which is used in Humboldt to
tabulate the county's total vote counts, first came into question when
a discrepancy arose with the county's final vote tallies for the
November presidential election.
The software error reportedly first came to Bowen's attention after
Humboldt County Registrar of Voters Carolyn Crnich informed her office
that the first-of-its-kind Humboldt County Election Transparency
Project found that 197 vote-by-mail ballots, which had been scanned
through vote counting machines, mysteriously disappeared from the
final ballot tally as tabulated by the GEMS software.
The problem was traced to a programing error with the specific version
of the software used in Humboldt -- a programing error that sometimes
results in the first deck of ballots scanned through the vote counting
machine, known as “Deck 0,” vanishing without a trace from
the final results.
Chris Riggall, a spokesman for Premier Elections Solutions, said the
company became aware of the error in 2004 and issued “work
around” orders to its customers instructing them how to take
steps to avoid the problem.
In two other counties using the same software -- San Luis Obispo and
Santa Barbara -- elections officials included the “work
around” orders into their written Election Day procedures. In
Humboldt County, then Elections Manager Lindsey McWilliams said he
received the e-mail, but failed to pass the information along to his
boss at the time, or his successor, Kelly Sanders.
That point seems lost in Bowen's report, said Riggall, who was also
careful to say that Premier agrees the particular version of GEMS in
question should no longer be used.
”I think it's fair, from our perspective, to say that the report
editorializes in a number of ways that don't reflect the sort of
even-handedness that one would like to see,” Riggall said, adding
that he sees the software error simply as a “contributing
factor” to Humboldt County's inaccurate election results.
“To say it caused it and to solely attribute that discrepancy to
the software is, in our judgment, not accurate and not objective.”
Riggall contends that McWilliams obviously should have ensured his
successors knew of the “work around” procedures and that
elections staff should have realized earlier on that a discrepancy
existed between the number of physical ballots scanned and the vote
According to Bowen's report, Crnich's staff twice checked the number of
physical ballots against the vote totals -- once on the day the
vote-by-mail ballots were scanned and again on Election Day -- and that
the ballots in question didn't disappear until days later.
Bowen's report also highlights what it deems as three
“deficiencies” with the system's audit logs. First, the
report says that the logs fail to include important actions, including
when a deck of ballots is deleted from the tallies. While administering
the November election, Humboldt County elections officials deleted 26
decks of ballots from the tallies, according to the report. But, none
of the deletions were recorded on the audit logs.
This issue also existed in subsequent versions of GEMS software,
Riggall said, but has been resolved in the latest version, which has
not been certified for use in California.
Additionally, the report states that some of the actions that were
recorded in the audit logs were given inaccurate date stamps. For
example, the report states that a status report for a deck of ballots
entered into the system on Nov. 3 carried the date stamp Nov. 25.
Finally, the report found that the audit logs also contain a function
allowing elections administrators to irrevocably delete the log, either
intentionally or inadvertently. Riggall said this function was removed
from subsequent versions of the software, but the report notes it is
still present in the version of the software used in Humboldt, Santa
Barbara and San Luis Obispo counties.
In fact, the report states that one California elections official,
while trying to print an audit log to fulfill a public records request,
inadvertently and irrevocably deleted the log instead.
Crnich said she's pleased with Bowen's report.
”I think it's pretty thorough and accurate,” she said. “That is what happened here.”
With the state having called a May 19 special election, Crnich said her
office will have to conduct one more election on the system it now
knows to be flawed, but said the system will not be used in November.
The Humboldt County Board of Supervisors has already approved a
tentative plan for Crnich's office to switch voting equipment, but
Crnich said funding for the new equipment has to be secured.
While the discrepancy in Humboldt County's final November tallies did
not impact the outcome of any race or ballot measure, Bowen's report
cautions that it could have been much worse.
”The loss of votes could have been greater; it's magnitude was
limited only by the number of ballots the county elections official had
chosen to scan as part of Deck 0,” it states.
On the web: To see the Secretary of State's complete report, visit www.sos.ca.gov/elections/elections_vs_premier.htm
Thadeus Greenson can be reached at 441-0509 or email@example.com
State considers decertifying election software
Thadeus Greenson/The Times-Standard
Posted: 02/22/2009 01:27:29 AM PST
California Secretary of State Debra Bowen is considering withdrawing
the state's approval of the flawed Premier Elections Solutions software
that resulted in almost 200 ballots disappearing from Humboldt County's
final November election results.
Bowen's office announced that it will hold a public hearing March 17 at
her Sacramento office to discuss the findings of its investigation into
the problem, and to take public comment on the possible withdrawal of
the state's approval of the voting system. Meanwhile, the two other
counties in the state that use the system are left contemplating what
its decertification would mean, and some election advocates are
lobbying for some punitive action to be taken against Premier Elections
”The public hearing is typically the last step in the
process,” Secretary of State spokeswoman Nicole Winger said.
“It offers one final opportunity for input, for the vendor to
state their side of the story and for the public to give comments or
input as well. It's a chance for everyone to be on the same page and
see where things stand.”
The software error reportedly first came to Bowen's attention after
Humboldt County Registrar of Voters Carolyn Crnich informed her office
that the first-of-its-kind Humboldt County Election Transparency
Project had uncovered a discrepancy in the final vote tallies from the
county's November election.
The project found that 197 vote-by-mail ballots, which had
been scanned through vote counting machines, mysteriously disappeared
from the final ballot tally as tabulated by Premier Elections
Solutions' GEMS software.
The problem was traced to a programming error with the specific version
of the software used in Humboldt County -- GEMS version 1.18.19 -- a
programming error that sometimes results in the first deck of ballots
scanned through the vote counting machine vanishing without a trace
from the final results.
Chris Riggall, a spokesman for Premier Elections Solutions, said in a
previous interview with the Times-Standard that the company had known
of the programming error since 2004. Saying the certification process
is too lengthy and time consuming to have had the software
re-certified, Riggall said Premier instead issued “work
around” orders by e-mail to its customers instructing them how to
take steps to avoid the problem.
The terse, 40-word e-mail, a copy of which was obtained by the
Times-Standard, tells the vendor's customers that it is “very
important that you follow these instructions,” and urges them to
contact the Premier's representatives with any questions.
In two other California counties using the same software -- San Luis
Obispo and Santa Barbara -- elections officials included the
“work around” orders into their written Election Day
procedures. In Humboldt County, then Elections Manager Lindsey
McWilliams said he received the e-mail, but failed to pass the
information along to his boss at the time, Crnich, or his successor,
The transparency project that discovered the error passes every ballot
cast in an election through an optical scanner after it's been
officially counted. The ballot images are then placed online, along
with open-source software, created by volunteer Mitch Trachtenberg,
that allows viewers to sort the ballots by precinct and scrutinize the
vote as they see fit.
Trachtenberg and Crnich met last week with Bowen and members of her
staff to offer a demonstration of the project, and they said the
presentation went well.
Careful to say that Bowen has not yet endorsed the transparency
project, Winger said her office was certainly interested to hear about
the fledgling project.
”Secretary Bowen is always interested to hear the details of
efforts to make elections more transparent, auditable and
accurate,” Winger said. “She certainly is open to ideas
about how to accomplish that on a secure, statewide level.”
Reached Thursday, Riggall said Premier believes the version of its GEMS
software in question should be taken out of circulation in California.
”We completely agree that that version of GEMS should not
continue to be used,” he said. “We absolutely have no
problem and certainly would concur were the secretary to take that
action directing counties to use a later version.”
In Humboldt County, the Board of Supervisors has already approved a
plan for the county elections office to switch to Hart InterCivic
equipment. Crnich is currently working with Bowen's office to come up
with a plan to fund the switch, which will then come back before the
board for final approval.
But, the possible decertification of the version of GEMS currently used
in Santa Barbara and San Luis Obispo is raising some questions there
for elections officials, especially with the state calling a special
election in May. Officials in both counties said they are still looking
into how they would proceed if the approval of their voting systems
were to be withdrawn.
Meanwhile, some in the election watchdog community are pushing for
either the Secretary of State's Office or the federal Elections
Assistance Commission to pursue punitive actions against Premier, which
they say knowingly kept elections systems in place that had
unacceptable error rates.
Riggall said Premier has done nothing disingenuous, and notified its
customers immediately upon discovering the error in its vote counting
system, carefully instructing them how to “work around” the
”I don't see that there is anything -- absolutely nothing -- in
how we have handled this issue going back several years that has been
disingenuous,” Riggall said.
Trachtenberg, on the other hand, said he thinks it would be appropriate
for the Department of Justice or the Attorney General's Office to
investigate the matter.
”I think that would be totally appropriate,” Trachtenberg said. “They sold software that doesn't work.”
Thadeus Greenson can be reached at 441-0509 or firstname.lastname@example.org.
New error found in county election results
Thadeus Greenson/The Times-Standard
Posted: 01/10/2009 01:28:11 AM PST
Once again, the Humboldt County Election Transparency Project has
found an error in the county's final tallies from the November election.
Humboldt County Registrar of Voters Carolyn Crnich announced that
the first-of-its kind transparency project uncovered the fact that 57
vote-by-mail ballots from a Petrolia precinct were counted twice in the
results certified by the Board of Supervisors early last month. This
revelation comes about a month after the transparency project
discovered that almost 200 ballots from a Eureka precinct disappeared
from the final results due to a programming error in Premier Elections
Solutions software used to add up the county's votes.
The double-counted ballots, Crnich said, would not have changed the outcome of any of the election's races.
According to Crnich, it remains unclear whether the 57
double-counted ballots were the result of the same software error
uncovered last month, a completely different problem, or simply the
result of operator error.
Crnich also announced that an error had been discovered in the
transparency project itself. She said 63 duplicate images were placed
online, which she attributed to operator error and a simple
“hiccup” in getting a first-of-its-kind program up and
The Humboldt County Election Transparency Project takes every ballot
cast in an election and passes them through an optical scan machine
after they are officially counted. The ballot images are then placed
online and available for anyone to download.
Open-source software, created by volunteer Mitch Trachtenberg,
allows viewers to sort and count ballots to conduct recounts at their
Back in the beginning of December, the project unearthed a software
error in the version of Premier Elections Solutions' -- formerly known
as Diebold Election Systems -- software used in Humboldt and two other
California counties. The error sometimes results in the first deck of
ballots scanned through the vote counting machine, sometimes known as
the “zero deck,” to vanish without a trace from the final
election results when another deck is deliberately deleted by elections
According to a spokesman, Premier Elections Solutions had known
about the error since 2004, and had issued “work-around”
instructions via e-mail to its customers instructing them how to avoid
Crnich said Friday that there is some reason to believe the recent
problem is related to what is now know in some circles as the
“zero deck bug.”
According to Crnich, the 57 duplicate ballots were run through the
same scanner on the same day as the 197 disappeared ballots. In this
case, Crnich said it's possible that an election worker tried to delete
the deck of 57 ballots in order to re-scan them, feeling an error had
occurred, but the software bug instead left those ballots in the system
and deleted the zero deck.
But, it seems equally likely that the duplicate ballots were the
result of operator error -- that an election worker intended to delete
one of the batches and forgot or inadvertently scanned the ballots
”I'd guess operator error is the most likely
(explanation),” Trachtenberg said, “but, given what's
happened with Diebold in the past, I'd never rule (the possibility of a
software error) out.”
Crnich said she has passed all information about the new vote
discrepancy on to California Secretary of State Debra Bowen's office
Kate Folmar, a spokeswoman for Bowen's office, said earlier this
week that her office's “fact-finding” mission into the
“deck-zero bug” is ongoing.
Federal Elections Assistance Commission Chairwoman Rosemary
Rodriguez said in a previous interview with the Times-Standard that the
commission has asked for reports on the error from both Crnich and
Bowen, and will widely disseminate them when they become available.
Regardless of the cause of the new error, it now seems obvious that
both it and the previous one would have slipped through unnoticed if
not for the transparency project.
North Coast Congressman Mike Thompson, who penned a letter to the
federal Elections Assistance Commission making it aware of Humboldt's
voting issues, issued a statement last month voicing support for the
”Our community should be extremely proud of the effort by
local citizens and the county registrar, Carolyn Crnich, who put
together this ground-breaking pilot project,” Thompson said.
“The Humboldt County Election Transparency Project made sure
that, in this county, every vote really does count.”
Federal election commission eyes Humboldt
01:33:58 AM PST
There are more than 3,000 counties in the
United States, but the federal Elections Assistance Commission's eyes are
firmly fixed on Humboldt.
The commission chairwoman, Rosemary
Rodriguez, said she received a letter from North Coast Congressman Mike
Thompson, D-St. Helena, early this month alerting the commission that Humboldt
County had certified inaccurate elections results due to a programming error in
its version of Premier Elections Solutions software that led to almost 200
votes being dropped from the county's final results.
”(Thompson) definitely asked the EAC to
follow up on this information and to try to work to support California's
elections officials in their attempts to get to the bottom of this and resolve
this matter,” Rodriguez said.
But, unlike the Federal Aviation
Administration, Rodriguez said the EAC doesn't have the authority or capacity
to launch independent investigations. So, Rodriguez said her commission is
currently awaiting reports from Humboldt County Registrar of Voters Carolyn
Crnich and California Secretary of State Debra Bowen detailing the error and
how it was discovered.
The commission, Rodriguez said, will then
use the reports in two ways in an attempt to ensure these types of errors don't
pop up again: It will pass them along to the federal elections systems testing
laboratories so it can test for precisely these types of problems; and it will
widely disseminate the information so every election official using the same
type of software is aware of the problem and how to work around it.
Just days after the Humboldt County Board of
Supervisors certified the November election results, the first-of-its-kind
Humboldt County Election Transparency Project uncovered the fact that 197
vote-by-mail ballots, which had been scanned through vote counting machines,
were mysteriously deleted from the final ballot tally as tabulated by Premier
Elections Solutions' GEMS software.
The problem was traced to a programming
error with the specific version of the software used in Humboldt County -- a
programming error that sometimes results in the first deck of ballots scanned
through the vote counting machine vanishing without a trace from the final
Chris Riggall, a spokesman for Premier
Elections Solutions, said in a previous interview with the Times-Standard that
the company had known of the programming error since 2004. Saying the
certification process is too lengthy and time-consuming to have had the software
re-certified, Riggall said Premier instead simply issued “work-around” orders
to its customers instructing them how to take steps to avoid the problem.
In two other California counties using the
same software -- San Luis Obispo and Santa Barbara -- elections officials
included the “work around” orders into their written Election Day procedures.
In Humboldt County, then Elections Manager Lindsey McWilliams said he received
the e-mail, but failed to pass the information along to his boss at the time,
Crnich, or his successor, Kelly Sanders.
It seems like both the EAC's courses of
action outlined by Rodriguez, if taken years ago, could have prevented Humboldt
County's current headache.
The version of Premier's error-prone GEMS
vote-counting software used in Humboldt County's November election went through
Bowen's 2006 top-to-bottom review of California's elections systems, but the
review did not pick up the programming error, which Bowen's office has said
didn't come to its attention until the almost 200 Humboldt County votes
If Bowen's team had known to look for this
type of problem, it's possible the error would have turned up during the
testing process and that the software never would have been certified. Making
sure testing laboratories look specifically for this type of programming error
will make them much less likely to slip through cracks in the certification
process, Rodriguez said.
Rodriguez said the fact that the EAC will
disseminate the contents of the Humboldt County report to elections officials
from coast to coast will also prevent similar problems from occurring
Evidently, part of what hit Humboldt County
on Election Day was a breakdown in communication. Rodriguez said the EAC's new
certification process will require manufacturers to notify it of any problems
with their systems, so that the EAC can make sure that information gets to all
elections officials, from secretaries of states to local elections managers.
One of the issues the error in Humboldt
County has raised, Rodriguez acknowledged, is whether similar problems may have
occurred elsewhere but gone unreported, simply having been dealt with between
the vendor and the local elections officials where the problem occurred.
”I think that's an appropriate concern,”
Rodriguez said. “There's this very confidential relationship between counties
and vendors, and I think issues have been resolved as they've arisen with these
kinds of work-arounds, but who knew. The voters certainly didn't know, and
elections officials aren't going to necessarily want their voters to know there
are problems. So, it's resulted in this kind of unhealthy situation.”
Because of that, Rodriguez said she's proud
of Humboldt County, and especially of Crnich, for stepping forward and acknowledging
there was a problem.
”The fact that they're sharing this
information is a gift, I think, for other counties,” Rodriguez said. “We're all
learning together and Humboldt County is really going to help us in that
process... I really commend the county for engaging in this transparency
project, and Secretary Bowen certainly has established the appropriate climate
of scrutiny, and I just think there's lots of good that comes from this very
unfortunate situation. Votes aren't to be trifled with.”
Thadeus Greenson can be reached at 441-0509
Registrar of Voters considers dumping equipment
Thadeus Greenson/The Times-Standard
Posted: 12/22/2008 01:33:45 AM PST
Humboldt County Registrar of Voters Carolyn Crnich said this week she is
considering dropping the Premier Elections Solutions software that dropped
almost 200 votes from the county's final election results.
But, some say that Crnich's proposal that the county discard Premier's
optical ballot scanning machines and GEMS vote-counting software in favor of
those of another elections company, HART InterCivic, is simply trading one
flawed system for another.
Meanwhile, the California Secretary of State's Office continues to look into
the error in Premier's software that led to the mysteriously deleted votes --
an error that the company said it has known about since 2004.
Just days after the Humboldt County Board of Supervisors certified the
November election results, the first-of-its-kind Humboldt County Election
Transparency Project uncovered the fact that 197 vote-by-mail ballots, which
had been scanned through vote-counting machines, were deleted without a trace
from the final tally, at least as tabulated by Premier Elections Solutions GEMS
The missing votes would not have changed any of the election's outcomes,
according to Crnich.
The programing problem has been traced to an error with a specific version
of GEMS software that sometimes results in the first deck of ballots scanned
through the vote-counting machine to vanish without a trace from the final
The error was caught by the transparency project, which
passes every ballot cast in an election through an optical scanner after it's
officially counted. Those images are then placed online, along with open-source
software, created by Mitch Trachtenberg, that allows viewers to sort the
ballots by precinct and scrutinize the vote as they see fit.
The week after the error was found, a Premier spokesman said the company had
known of the software error for years, and had simply sent e-mails to its
customers instructing how to “work around” the problem.
The company, however, did not notify the California Secretary of State's
Office of the problem.
While elections offices in San Luis Obispo and Santa Barbara added the work-around
instructions to their written Election Day procedures, then-Humboldt County
Elections Manager Lindsey McWilliams did not, and failed to notify Crnich, his
direct supervisor, or Kelly Sanders, his successor, of the problem.
Kate Folmar, press secretary for Secretary of State Debra Bowen, said Bowen
believes that Premier should have notified her office when this software error
”Right now, the secretary of state is talking to the company, other
elections officials and other experts,” Folmar said. “Once Secretary Bowen is
done gathering all the facts, she definitely plans on notifying the (federal)
Elections Assistance Commission of her findings and the flaw in Premier's
software. When that's done, she will determine what, if any, other action can
be taken in California.”
Crnich said she has already been contacted by the commission, which has
requested a report from both her and Bowen's office on the problem.
North Coast Congressman Mike Thompson, D-St. Helena, said he has also
contacted some federal elections officials about the problem.
”The health of our democracy depends on the integrity of our elections
process,” Thompson said in a statement. “Our community should be extremely
proud of the effort by local citizens and the county registrar, Carolyn Crnich,
who put together this ground-breaking pilot project. The Humboldt County
Election Transparency Project made sure that, in this county, every vote really
”On a larger scale,” Thompson continued. “I am very concerned that this
software problem has existed for at least four years and the company has not
taken adequate steps to assure the integrity of the system. I have written to
federal elections officials to make sure they review the matter and take the
steps necessary to ensure this type of error doesn't occur again.”
While Crnich's announcement that she is proposing the county scrap Premier's
software and ballot-scanning equipment in exchange for that of Hart InterCivic
only came this week, she said it's not for the reasons many might think.
”This plan that is proposed pre-dates any of the problems that were found to
exist in this election,” Crnich said.
She said the move is simply intended to bring the county's whole system
under one vendor. As is, the county uses some equipment from Premier and some
from Hart InterCivic, which means the elections staff has to essentially
operate two different systems, according to Crnich.
Under Crnich's proposal, the county would purchase 80 new eScan machines and
new central vote-counting software for a total of approximately $600,000.
Crnich said the funds would not come from county coffers, but instead would
come from money set aside through the passage of Proposition 41 and the Help
America Vote Act.
Crnich's proposal is tentatively slated to come up for approval by the
Humboldt County Board of Supervisors at its Jan. 6 meeting, after which it
would go to Bowen's office for final authorization.
But some charge that Crnich's proposal simply swaps one untrustworthy system
for another, and others are calling for a more public process to determine what
the best course would be for the county's Elections Office.
”After Premier's GEMS system covertly deleted 197 ballots from the November
2008 election, my hope would be that Humboldt County could slowly and
deliberately evaluate the various election technologies available,” said
election volunteer Parke Bostrom in an e-mail to the Times-Standard. “To
rapidly abandon a previously trusted technology, only to quickly replace it
with another secret, proprietary counting system, without allowing the public
an opportunity to comment, seems to me the wrong course. Or, maybe, it is the
right course, but it is being taken at the wrong speed.”
For her part, Crnich said the public will have an opportunity to weigh in
when the matter comes before the supervisors.
Dave Berman, co-founder of the Voter Confidence Committee and an outspoken
critic of proprietary vote-counting technology, agreed with Bostrom, urging the
county to court more public opinion before deciding how to proceed.
”I think it's a matter of process,” Berman said. “If we're going to have a
meaningful public exploration of the issues, I don't think that process should
begin with the decision-maker already having made a decision.”
The Board of Supervisors will have the final county say on the matter and,
whatever is decided, many remain thankful that the county has the transparency
project in place as a safety net.
Though the project, in its first general election, has already proven
valuable, Folmar said it has yet to be endorsed by the Secretary of State's
”Secretary Bowen has a long history of supporting transparency in
government,” Folmar said. “She would love to see the day when California
elections can be run on fully developed open-source software once that
technology is available and fully tested. But, Secretary Bowen has not fully
endorsed any transparency project.”
Thadeus Greenson can be reached at 441-0509 or email@example.com.
november 2008 ballot images are online
by Mitch Trachtenberg, 12/15/2008
The Transparency Project's scanned images of Humboldt County's November
2008 ballots are now available online. Thank you to Streamguys,
especially Jonathan Speaker and Andy Jones, for helping us get the bits
out of our beautiful low-bandwidth region. And thank you to EARC for
offering the hosting space.
The UC Berkeley Election Administration Research Center
is hosting the images in collection form, as a series of approximately
one gig files and accompanying GPG signatures. The site is momentarily
password protected; if it is still password protected when you read
this, we've been given permission to give the case-sensitive
user/password combination of "download/Down2disc".
The ballots will also be hosted as individual images (and perhaps as collections as well) at Parke Bostrom's ETP site.
We expect the ballots to become available at other sites as well.
The latest version of Ballot Browser software is also available. This location will point to updates as they appear.
Our Votes are Too Important...
by Mitch Trachtenberg, 12/7/2008
Copyright 2008 by Mitch Trachtenberg, see end for Creative Commons license.
Humboldt County, California is fortunate to have a County Clerk and Registrar of
Voters, Carolyn Crnich, who is herself an election transparency advocate.
There were probably groups in just about every jurisdiction in the country who
would have liked to do an independent vote count, but in Humboldt County we were
in the pleasant situation of having Humboldt's registrar not only neutral, but
actively on our side. Through her election advisory committee and the
Humboldt County Election Transparency Project, Crnich was able to provide access
to the ballots -- something that in some other areas seems to require a lawsuit.
I met Crnich earlier this year, and remember excitedly telling people "she gets
it!" Years ago, I'd spent some time trying to convince the previous
elections manager of the dangers of DRE touch screens, but I could tell I was
getting nowhere. This time, the person in charge of the vote counting was
as indignant about secret counts and missing paper trails as I was!
Humboldt uses equipment produced by Diebold, a name which became so notorious in
election circles that the company has changed its name to Premier. Still,
our county hadn't switched to DRE touchscreens. We were still using paper
ballots and Diebold's optical scanners. Because Humboldt's system
preserves paper ballots, it seemed to me highly unlikely that there'd be any
discrepancies between our independent count and the count generated by
Diebold. "They wouldn't dare," is what I'd been thinking. Still, you
don't KNOW things are OK until you've done your independent count.
I figured that by demonstrating how paper ballots could be independently
counted, we'd be providing a useful argument for those who were trying to
prevent the touch screens from becoming more popular. I also thought we'd
be verifying the vendor counts, and putting the vendors on notice that
independent counts could be done, limiting any future impulse towards
Others in the community wanted complete manual counts. I've seen what
manual counts look like, and they have their own problems. The more people
that are in a room with the ballots, the more distractions. Every time
ballots are moved, that's a potential opportunity for misplacement or
worse. And the time commitment to get a solid manual count is
astonishing. I watched the start of the mandated 1% count, and was
impressed by the commitment of the citizens doing the counting. Panels of
five. One person picks up a ballot and calls out the name of the winning
candidate in the Presidential race. Person one hands the ballot to person
two, who looks at it and puts it in a stack. Persons three and four both
mark tally sheets with a pencil. Person five watches. After a bunch
of ballots go by, person one calls "tally" and persons three and four both read
their count. Repeat, seemingly forever. Then start again on the next
race. It works, but it's very slow; unrealistically slow for full timely
I agreed with Kevin Collins, the commercial fisherman who, together with Crnich,
Tom Pinto of the D.A.s office, Green Party Presidential Candidate David Cobb,
and Parke Bostrom, had initiated the transparency project: let's scan the
ballots so people can do recounts on their own. I'm a software developer,
and I felt it was important to provide open source software that could count the
votes on the scanned images. Otherwise, I thought, the scanned images
might sit there with nobody really bothering to count their votes. So I
wrote a program to count the votes.
To keep things as open as possible, I wrote the counting program completely
separate from the software we used to do the scanning. We used Debian
Linux, a completely open platform, to do the scanning. Linux uses an open
scanning protocol called SANE (short for Scanner Access Now Easy). SANE
didn't work with the serial number imprinter on our Fujitsu scanner, so I
contacted M. Allan Noah in Virginia -- in open source lingo he's the
author and "maintainer" of the SANE Fujitsu backend. He was incredible,
getting us some initial changes the same day and guiding us in getting him
enough information that, even though he doesn't have the very expensive scanner
model Humboldt had purchased, he was able to decipher the USB dialog that took
place between Windows and the scanner and reproduce it within SANE. I
wrote a very simple script to allow our volunteers to drive the SANE program and
get new serial numbers every time they started a scan batch.
We'd heard concerns that stray marks on the ballots might lead to allegations
that we were facilitating vote buying. Kevin Collins, along with many
others, has argued extensively that if someone wants to buy votes in California,
they'd do it the easy way: by taking advantage of vote-by-mail, offering money
to voters who let the high-bidder fill out their vote-by-mail ballots. In
any event, I wrote software to spot stray marks on the ballot images and "scrub"
the images clean. Then we all realized no one would want to see "scrubbed"
ballots, they'd want to see what we originally scanned. So with a bit of a
wink, wink, nudge, nudge from some high state officials, we figured we'd release
the ballot images and see if anyone complained. And, under California law,
any ballot that can be traced to an individual is automatically invalid.
I also wanted to try to replicate a manual count as closely as possible.
So I set up the counting program to display each image as it counted it,
optionally overlaying the ballot image with boxes around each vote oval, and a
red box and label around each vote oval that, in the opinion of the program, had
a vote contained within. I added a delay slider, so we could project each
image in a public hall for a certain number of seconds so that people could
count along with the program, or note their disagreements. Finally, I had
the program write out each vote to a file that could be imported into a
spreadsheet. This file shows the image number, information that can
identify the precinct, the contest, the candidate, and some debugging
information that can be used to locate ballots where the program's output should
Last Sunday, after the election department had wrapped up its certification
work, the last ballots were made available to our project for scanning.
Carolyn would unseal a bag of ballots and hand them to me, I'd feed the ballots into
the scanner, and Carolyn would reseal the bag. With the stack of bags
dwindling, Carolyn and I both began making nervous jokes about what would happen
if we didn't have as many ballots as the elections office had certified.
We looked at the bags and made estimates about how many remained.
When our numbers were approaching those of the official count, we breathed a
sigh of relief. And then I joked that we weren't out of the woods yet,
maybe the Project would have too many ballots. We both laughed. And
then that's what happened -- we had 433 images too many.
We were rapidly able to determine that one of the images was a single-sided
ender card that had been scanned by mistake... that left us with 432 extra
images, or 216 extra ballots. I think both Carolyn and I assumed at that
point that somehow we'd double-scanned a bag or two, though it was hard to
imagine how that had happened. Every ballot side that went through the
scanner got a serial number imprinted upon it, so that the scanned image could
be tracked back to the individual ballot side if questions arose. How do
you fail to notice that you're feeding the scanner a ballot that already has a
serial number... 216 times?
I started a run of the counting software. The software can be run
simultaneously on multiple computers, so that each computer only needs to run
part of the vote. I wasn't yet confident that the merge would go smoothly
and I didn't think this was a good time to experiment, so I just ran the run on
one machine... it took 12 hours. At the end, it was clear that precinct
1E-45's vote count for both Obama and McCain was substantially off from the
official results. We'd kept careful logs of the scanning process and went
through them and, sure enough, there was a whole bag of ballots from precinct
1E45 that we'd scanned for the Project, but hadn't made the official count.
Here, I believe, the elections office began trying to figure out what went
wrong. I think everyone in the office probably assumed that, if the
Transparency Project hadn't made a mistake, they must have. No one thought
we were both right. I wasn't there as the office went through their own
logs, but I'm sure it was tense... the election results had already been
certified before the Board of Supervisors. If I were Carolyn, I'd have
been thinking "no good deed goes unpunished."
But then it turned out that the elections office HAD run the "missing" bag
through the Diebold system, but the Diebold system had not included that bag in
the detailed, final, official canvass results that it spits out. Calls to
Diebold and the Secretary of State's office confirmed that we'd found a problem
that had been in the Diebold counting software for the last four years.
Diebold had told -- rather casually, it seems to me -- county election officials
about the problem, but they never notified the office of the California
Secretary of State. And Humboldt's previous elections manager never passed
along news of the problem to his successor. It looks like Diebold released
subsequent versions of their software without fixing the problem. It looks
like some of those versions may have been used in states like Maryland and (you
guessed it, didn't you) Florida. Hey, it's always Florida.
What about going forward? Well, we've demonstrated that independent counts
are practical, and we've demonstrated that, without independent counts,
unexpected and potentially undetectable errors can creep into official
results. I think it's time for every jurisdiction in the country to
eliminate use of DRE machines. I think we need to stop using secret,
proprietary software to do our vote counting.
Some registrars may say that independent counting will only work in small
counties like Humboldt. I disagree. I believe that the approach
we've used can be scaled up to any size. Larger counties will have more
ballots, but they'll also have more volunteers to scan, or more of a budget to
pay people to scan. If you've got a million registered voters, just split
your ballots into ten groups and buy ten scanners. If you've got five
million, just split your ballots into fifty groups.
I'm hopeful that some inspired group may develop faster and less expensive
ballot scanning hardware -- I have ideas about how to do it, but I'm all thumbs.
This was all made possible by a registrar, Carolyn Crnich, who was willing to
work with the concerned citizens in her community, rather than dismiss
them. I think Carolyn Crnich is now a national hero. And all
California citizens are very fortunate that our Secretary of State, Debra Bowen,
stood up to the companies that wanted to foist DRE touch screen voting on
us. Touch screens don't solve any problem -- their use to help those with
disabilities was, in my opinion, a phony argument from the start.
Touch screens must go. In counties and states that have switched, paper
ballots must return. Those with disabilities can be helped to vote with
systems that assist them in filling out paper ballots.
Our votes are too important to be counted by secret software running on
black-box vendor machines.
Our Votes Are Too Important... by Mitch Trachtenberg
is licensed under a
Attribution-No Derivative Works 3.0 United States License.
by Hank Sims, 12/11/2008
A couple of weeks ago I got a
chance to do my duty for democracy at the Humboldt County Elections
Office. I was working a scanner with Kevin Collins --
Ettersburg resident, commercial fisherman, old-school frizzy-bearded
Humboldt County visionary. Collins was keeping logs, I was operating
the computer, Humboldt County Clerk-Recorder Carolyn Crnich
and other elections employees lurked about, bringing us stacks of
ballots from the Nov. 4 election and spelling us during our lunch
This was the work of the Humboldt Transparency Project,
which had been going on for several weeks before I dropped in one
afternoon. The idea of the project, which, after a June test-run, got
going in earnest with this election, is to scan every ballot cast in
Humboldt County and make them freely available on the Web for all to
peruse. Let 100 recounts bloom. Collins dreamed up the idea a few years
ago, in the wake of the horrors of the 2000 Florida presidential
election. Crnich endorsed it and found funding to make it happen. Local
software developer Mitch Trachtenberg built the
infrastructure and wrote a program that would help DIY recounters
examine election results from the comfort of their own homes. Other
local volunteers, notably Parke Bostrom and Tom Pinto, chipped in.
As everyone knows by now, the Transparency Project drew instant
dividends. All that scanning was completed the weekend after
Thanksgiving, and it revealed that there was a problem. The
Transparency Project had scanned about 220 more ballots than had been
recorded by the official Diebold tabulating software
that the county has been using for ages. And it wasn't the ragtag
Humboldt volunteers that had fucked up. It was Diebold. And it was
"Premier," which is what Diebold renamed itself after its old name
became synonymous with incompetence and scandal.
Crnich tracked down the missing ballots after the discrepancy came
to light last week, and held a press conference. It turns out that
there is a very longstanding glitch -- at least four years old,
probably more -- in the version of Premier's Global Election Management
System (GEMS) tabulating software used by Humboldt County. To make a
long story short, it turns out that the system can unpredictably delete
the first batch of votes run through it -- the ominously named "Deck
Zero," in Diebold parlance. Absentee ballots, provisional ballots and
ballots from vote-by-mail precincts each have their "Deck Zero." (Votes
cast at a polling place on Election Day are tabulated seperately.)
During the normal course of operations, it turns out, Deck Zero can
disappear without a trace. And so it was this time with absentee
ballots from Eureka precinct 1E-45, which takes in Lundbar Hills, the
municipal golf course and parts of outer Henderson Center. They were
run through first among absentee ballots -- they became Deck Zero --
and they were dropped entirely from the official results certified by
the Humboldt County Board of Supervisors Thanksgiving week.
No one in the California Secretary of State's office -- the state
agency charged with conducting elections -- knew anything about the
Deck Zero problem. Last year the agency conducted a "top-to-bottom"
review of elections equipment used throughout the state, with an eye to
uncovering all potential problems. Deck Zero never came up.
Yet it turns out that Diebold/Premier has known about the Deck Zero
glitch since at least 2004, and in a scattershot way has advised users
of its equipment about how to cope with the problem. Wired News,
which has jumped all over this issue, got hold of an e-mail from that
year which offered election managers a workaround. It suggested that
people should run through a fake or empty batch of ballots first, and
then manually delete them. Make sure Deck Zero is a fake deck. And it
turns out election managers in other California counties have been
doing this little pas de deux for some time, as had Humboldt
County, previously. But the problem with Diebold's brilliant solution
is that -- as in Humboldt County -- people move around, the new person
doesn't get the four-year-old memo.
The word "hack" has many meanings in the world of software
development. Sometimes it refers to a stupid, ugly, cumbersome
workaround, employed as a lazy alternative to actually fixing the
problem. This was that kind of hack. It's equivalent to an automobile
company telling its unhappy customers that the thing should start if
you first whack the engine with the crowbar a couple of times. The fact
that Diebold/Premier let it stand for over four years, potentially
undermining the first principle of American democracy, is an absolute
outrage. These people should be shunned. Maybe indicted.
Hooray for the Humboldt Transparency Project -- true American heroes. For more, visit humtp.com.
A glitch that should never have been
people have called those who have long decried our nation's move toward
voting machines nuts or just sore losers, but here we are in Humboldt
County with our own example of what can happen when there's a glitch in
They were loud, and they were strident in
proclaiming that they didn't trust election technologies as much as
they trust the ability of actual human beings to count votes.
recent discovery, thanks to the Humboldt County Election Transparency
Project, of a discrepancy in election results due to flawed software
reveals that these activists were right to make noise, and right to
complain about a company that has been less than responsible in dealing
with the problem.
Humboldt County recently certified inaccurate
election results due to a glitch in the software from Premier Elections
There is a work-around to make sure this doesn't
happen, but somehow that information was not passed on when former
Elections Officer Lindsey McWilliams left his Humboldt County post for
a better-paying gig down south.
In an interview with the
Times-Standard, McWilliams said that somehow, in the two and a half
weeks of his transition out, that was a detail that got omitted.
This is a flimsy excuse for several reasons.
a work-around to make sure ballots and votes get accurately counted
should be at the absolute top of anyone's list of important
Secondly, this supposed work-around came from Premier in the form of a memorandum.
McWilliams' boss, it seems absolutely ridiculous that Humboldt County
Registrar of Voters Carolyn Crnich was not forwarded a copy of this
memo, or that the county's instructions for operating the machines were
not updated to include the information.
Thankfully, we have the
transparency project, and because we have citizens who pay close
attention, we were able to find out about this glitch and can now
rectify the results.
Crnich deserves much commendation for
developing the Transparency Project, as do all the community members
who have long been working on making it a reality. Your work has paid
Local elections office commended
Since 2004, Premier
Elections Solutions has known about the programming error in its
software that caused almost 200 ballots to be dropped from Humboldt
County's final November election tally, but that came as news to
California Secretary of State Debra Bowen's office.
Bowen is certainly concerned about Premier's carelessness with yet
another elections product and thinks it's distressing that the company
took virtually no action for years on this apparent defect,” Secretary
of State Press Secretary Kate Folmar wrote in an e-mail to the
Times-Standard. “Secretary Bowen is talking with the company, county
elections officials and others about how to prevent this problem from
ever happening again in California.”
Just days after the
Humboldt County Board of Supervisors certified the November election
results, the county election's first-of-its-kind Humboldt County
Election Transparency Project uncovered the fact that 197 vote-by-mail
ballots, which had been scanned through vote counting machines, were
mysteriously deleted from the final ballot tally as tabulated by
Premier Elections Solutions GEMS software.
The 197 missing
ballots would not have changed any of the election's outcomes,
according to Humboldt County Registrar of Voters Carolyn Crnich.
However, transparency project volunteer Mitch Trachtenberg
said, the unveiled problem in Premier's software underscores a major
”Our votes are too important to be counted by secret code running on proprietary machines,” he said in a statement.
problem was traced to a programming error with the specific version of
the software used in Humboldt County -- a programming error that
sometimes results in the first deck of ballots scanned through the vote
counting machine vanishing without a trace from the final results.
Elections Solutions spokesman Chris Riggall said the company became
aware of the problem in October 2004, and sent out e-mails to its
customers detailing how to “work around” the problem and schooled its
customers on the procedure at a variety of events.
other counties in California that use the same version of the software
-- Santa Barbara and San Luis Obispo -- were aware of the problem and
how to work around it, but Bowen's office was not, despite having
conducted a top-to-bottom review of all the state's voting equipment in
2006 due to concerns about the reliability of the state's elections
San Luis Obispo Assistant County Clerk Recorder Tommy
Gong said his office received the advisory from Premier, quickly worked
the work-around procedure into its election protocol, and hasn't had
any problems. Gong also said his county has a method for double
checking the results.
”Physically, we know how many ballots
we've counted through the machines and the final report should reflect
that number,” he said. “If not, there's an issue.”
Santa Barbara County Chief Deputy Registrar of Voters Billie Alvarez
said Premier notified her office of the problem some years back, and
she simply included the work-around steps into the department's written
Apparently, nothing like that happened in Humboldt County.
McWilliams was the elections manager for Humboldt County when the
programming error was discovered by Premier. He has since moved on to
become the assistant registrar of voters for Solano County.
said he did receive the e-mail from Premier identifying the problem in
its software and how to work around it. But he never included the
work-around steps into the county's detailed written elections
procedures nor apparently did he inform Crnich, then his boss, of the
issue, even as he left the Election's Office last year to head to
”There really wasn't any schooling anyone to
take over (my position) because I had all of two and a half weeks of
transition between (Solano County) and (Humboldt County), and there
wasn't an election in process, so there was not a complete transfer of
knowledge,” McWilliams said.
Back at Premier, Riggall said that
part of his company's job is to communicate well with its customers,
specifically surrounding the operations of its systems and potential
Asked why when McWilliams left, the company didn't
make sure to inform his replacement of the software problem, Riggall
said he didn't know.
”I certainly would accept the fact that our job, as always, is
to try to make sure we're communicating well,” he said.
as to why the company didn't do more to ensure the software problem
didn't affect any elections -- either by recalling the software
entirely and correcting the problem or, at least, issuing new
operations manuals with the work-around procedure included -- Riggall
said federal and state certification processes, which can take years,
made that impractical.
”It's one of the real obstacles in our
business that when we identify an issue, getting that enhancement into
the field so that issue can be corrected is a very lengthy, laborious,
expensive and time-consuming process,” Riggall said. “When you're not
able to do that, you have to rely more on work-arounds and the guidance
of your customers.”
Since 2004, Riggall said the company has
also revised its “product advisory notice” process so that the
Secretary of State's Office is made aware of any software or equipment
issues as soon as counties are.
But all this just seems to beg
the questions that some have been asking for years: Are our elections
too important to be subject to the same cost-benefit analysis of other
private businesses? Should there be fail-safe ways to ensure elections
are conducted properly, regardless of the turnover in a given elections
office? Should there be a level of transparency to the extent that
everyone knows how vote-counting software works? And, should every
election be audited to ensure that every vote is counted?
has answered the last question, at least locally, through the creation
of the Humboldt County Election Transparency Project. On the surface,
the project is pretty simple: Every ballot cast in an election is
passed through an optical scanner after being officially counted and
the images are then placed online and made available for download.
software, created by Trachtenberg, then allows viewers to sort the
ballots by precinct and conduct recounts or scrutinize the vote as they
In this, the first election in which the project was
fully up and running, it uncovered a software glitch that otherwise
would have gone completely unnoticed and potentially could have changed
the election's outcome.
The Secretary of State's Office
certainly thinks Crnich deserves a proverbial pat on the back for
catching the discrepancy, and consequently notifying Bowen's office of
the problem in Premier's software.
”Carolyn Crnich deserves
kudos for her dedication to election integrity and for alerting
Secretary Bowen immediately when Crnich uncovered the problem,” Folmar
Thadeus Greenson can be reached at 441-0509 or firstname.lastname@example.org.
Software glitch yields inaccurate Election results
first of its kind Humboldt Election Transparency Project has uncovered
a glitch in the county election's software that resulted in almost 200
ballots not being counted and the county certifying inaccurate election
The 197 uncounted ballots would not have changed the
outcome of any of the election's races, according to Humboldt County
Registrar of Voters Carolyn Crnich.
Crnich said the company
that provides the county's election software, Premier Elections
Solutions (formerly known as Diebold Election Systems, Inc.), seems to
have known about the glitch at least since 2004.
Crnich said a
discrepancy in vote counts came to her attention after the election was
officially certified by the Humboldt County Board of Supervisors, while
she and volunteers were preparing ballot images for the transparency
The basic idea behind the first-of-its-kind
transparency project is fairly simple: every ballot cast in an election
is passed through an optical scanner after being officially counted and
the images are then placed online and available for download.
created by volunteer Mitch Trachtenberg, then allows viewers to sort
the ballots by precinct or race to conduct recounts at their pleasure.
after the election was officially certified Monday, Crnich said she got
an e-mail from Trachtenberg saying something was amiss.
”(Eureka's) Precinct 1E-45 seemed out of kilter,” she said. “The count just wasn't adding up.”
After double checking all
of the precinct's logs and ballots, Crnich said she discovered a deck
of 197 vote-by-mail ballots for the precinct that had been run through
the ballot counting optical scanner, but did not seem to appear in the
final vote tallies.
After exchanging several calls with Premier
Elections Solutions, Crnich said she was told that the software begins
counting decks of ballots at zero, and that sometimes when a deck is
deleted from the machine due to normal complications, the software also
deletes the Deck Zero, which in this case was the vote-by-mail ballots
from Precinct 1E-45.
Crnich said she then called the Secretary of State's Office.
”They were very interested and actually offered great congratulations on this project,” Crnich said.
said she later learned from the Secretary of State's Office that two
other California counties, Santa Barbara and San Luis Obispo, use the
same version of GEMS elections software (version 1.18.19), as well as
several entire states, including Maryland.
The Secretary of
State's Office was not immediately available for comment by deadline
and a late call to Premier Elections Solutions was not answered.
said it appears that Santa Barbara and San Luis Obispo counties had
been informed of the software glitch, and were told how to work around
it to avoid having any effect on the election counts.
Secretary of State's Office, however, had not been notified of the
problem despite having conducted a top-to-bottom review of the state's
elections systems in 2006, according to Crnich.
part of all this, said Trachtenberg, is that the issue would have never
been uncovered without the transparency project.
”Has this happened in other counties or other states?” he asked. “How can we know?”
also said she was informed by the Secretary of State's Office that this
version of Premier Elections Solutions GEMS software was in use in the
highly contested 2000 Florida election before the problem surfaced.
the glitch also seems to lend credence to groups of people across the
country who, for years, have criticized placing the nation's elections
in the hands of private companies that dispense vote counting machines
that operate with secrete, proprietary codes that, in many cases, leave
no paper trail.
Kevin Collins, who volunteers with the
transparency project and is one of its charter members, said this never
would have been uncovered without Crnich's dedication to transparent
”She deserves a huge amount of credit for devising a
system for doing something in Humboldt County that isn't being done
anywhere else, and that's auditing 100 percent of the ballots,” Collins
The uncovered glitch means little for Humboldt County's
election, as it won't change the outcome of any races and, consequently
won't even require a re-certification of the election's results, but it
has implications that could reverberate throughout the world of
”You just can't trust a secret program to count this
stuff because programers make mistakes,” Trachtenberg said. “People
have been complaining about secret machine counts and the companies
have said these folks are nuts. But, the first time (the transparency
project) is done in a general election, it comes up with a problem -- a
problem (Premier Elections Solutions) has known about for four years.”
COMMENTARY ON PREMIER ELECTION SYSTEM'S GEMS CENTRAL TABULATOR NOT COUNTING 197 BALLOTS IN HUMBOLDT COUNTY
My name is Parke Bostrom. I am a volunteer working on the Humboldt
Transparency Project (ETP). (See
for more information on the ETP.) I
do not work for the County of Humboldt and I am not directly involved
in the primary counting of the vote in Humboldt County. Much of the
information below is second or third-hand and may need to be
corrected, updated or clarified as more reliable information emerges.
On Sunday November 30th the ETP volunteers (with assistance from
Humboldt County elections
office employees) finished the ETP scanning
of the ballots from November's election. We finished several days
prior to the deadline for the Elections Office to certify the results.
According to the ETP's initial (and largely unanalyzed) results, we
scanned 216 more ballots than the county's primary count system
(Premier's GEMS). As the ETP is still a new, developing "beta-ish"
project, it seemed most likely to me that the extra ballots were due
to some error in the execution of the ETP. I expected us to be able
to identify the cause of the error, correct our results, and improve
our procedures in future elections.
On (I believe) Tuesday December 2nd, County Registrar of Voters
Carolyn Crnich certified the election results provided by GEMS. On
Wednesday December 3rd, ETP volunteers working together with Crnich
and the elections office identified the cause of the largest part of
the 216 ballot discrepancy. The GEMS totals (which had just been
certified) failed to include 197 ballots from "Deck 0". The ETP had
also scanned these ballots, so they showed up in our totals. The ETP
therefore identified a serious error in the vote count generated by
Some background information on the counting process helps explain the
nature of "Deck 0". Deck 0 contains some (but not all) of the
absentee ballots from a single precinct (in this case, precinct
1E-45). Absentee ballots are sent out about a month before the
election, and are returned to the election office continuously over
the entire month leading up to the election. At multiple points in
time, whenever the number of returned ballots is sufficiently large,
all the queued incoming ballots that have been received thus far are
sorted by precinct into "decks" of ballots. A deck contains ballots
from only one precinct, but as this sorting happens at multiple points
in time, the ballots from a single precinct are almost always spread
across three or four decks.
Several days before the election (I believe the Saturday before), the
Elections Office began feeding the decks of ballots into the GEMS
central tabulator. The Elections Office is not allowed to print out a
report of the vote totals until the polls close on election night, but
the computerized counting process can begin several days before
election night. The reason for starting in advance is that machine
counting of thousands of ballots, while faster than hand counting, is
nonetheless a time consuming process
Deck 0 was the first deck of ballots counted by GEMS. It is called 0
(instead of 1), because it is common practice for computer programmers
to start counting at 0.
Upon discovering that deck 0 was not included in the certified
election results, Crnich contacted Premier. I did not observe that
conversation, but from what I understand, Premier claims the deck 0
results are sometimes automatically deleted by the system when a
subsequent deck is intentionally deleted and rescanned. It is
expected that at least several later decks will be deleted and
rescanned as part of normal procedures. The reason for this is that
there is no way to amend a deck's results, so if you need to amend a
decks results, you have to rescan the entire deck. However, when you
intentionally delete a given deck's results (prior to rescanning), you
expect that the other decks results will remain unchanged.
Premier claims (or so I hear tell) that Deck 0 is sometimes "randomly"
deleted because different programmers worked on GEMS at different
points in time, and some programmers start counting at 0, while other
programmers start counting at 1. (Given my own experience as a
programmer, it may be the case that deck 0's results are not really
deleted, but were instead "skipped" or not included in the updated
report. This is not a terribly happy distinction, however, because
the updated report is still wrong, indeed just as wrong as if deck 0's
results had in fact been deleted.)
The ETP had 216 extra ballots. 197 of those were from deck 0. That
means that the ETP now has 19 extra ballots to account for. 19 extra
ballots could be due to a variety of causes, including: GEMS double
feeding ballots (counting 2 ballots as 1), and various operator
errors/anomalies on the part of the ETP volunteers scanning the
ballots. In the June election the ETP counted 2 more ballots than
GEMS. 216 extra ballots are truly alarming. 19 extra ballots (out of
60,000+) are worth investigating and accounting for.
After additional investigation, we learned the following information:
The precinct voted ballots were fed into GEMS on election night, as is
standard procedure. From November 5th until November 23rd, no
additional results were fed into GEMS. During this period, the
elections office was busy auditing the rosters containing voters'
signatures, preparing the remaining vote-by-mail ballots for scanning,
and reviewing and preparing provisional ballots
Scanning resumed on November 23rd. Prior to scanning, the elections
office printed out a GEMS report. The report contained the same
totals from the final report on election night, as it should. The
remaining ballots were all fed into GEMS during November 23rd through
When the dropped ballots were discovered, Premier requested copies of
the GEMS database in order to try to determine what went wrong.
Premier claims that deck 0 was deleted sometime during the processing
of decks 131 through 135.
Crnich remembers that during the scanning of deck 132, the GEMS
program was restarted (indeed, the whole computer hosting GEMS was
rebooted) *after* deck 132 had been "opened" but *before* any ballots
had been scanned into deck 132. The reason for the reboot was that
the ballots for deck 132 were of a different type (vote-by-mail,
provisional, mail-ballot-precinct, and precinct voted ballots all
being distinct ballot types). Standard procedure is to reboot the
computer when switching ballot types. (Requiring reboots seems to me
like a kludge/workaround designed to cover up bugs and design flaws in
GEMS.) Our best guess is that restarting GEMS while deck 132 was open
causes GEMS to silently delete deck 0. At no point did GEMS give any
indication that anything was going wrong.
There are other anomalies in the GEMS audit log. There are actually
three "deck 0"'s: deck 0 of vote by mail, deck 0 of provisional
, and deck 0 of mail-ballot-precincts. Deck 0 of 1E-45
(containing the 197 dropped ballots) does not show up in the audit log
and those ballots are not included in the final report. The other 2
deck 0's also do not appear in the audit log, but apparently their
votes are included in the final report. This means the audit log is
not truly a "log" in the classical computer program sense, but is
rather a "re-imagining" of what GEMS would like the audit log to be,
based on whatever information GEMS happens to remember at the end of
the vote counting process.
The version of GEMS the county is using is 1.18.19
. This version is
also used by Santa Barbara
and San Luis Obisbo counties in California
It is also used by the entire state of Maryland. Premier has an
internal memo dating from 2004 describing the deck 0 problem. The
workaround to the problem is to manually create and delete all the
deck 0's prior to scanning any ballots. Santa Barbara and San Luis
Obisbo counties are aware of the workaround and have done it in all
recent elections. However, the Humboldt elections office has been
unable to find any official Premier documentation or addenda
describing the workaround. Apparently, the workaround was
communicated from Premier to counties via word of mouth, and the
knowledge of the problem and the workaround may have departed from
due to employee turnover. Additionally, in 4 years
Premier has not fixed the underlying problem.
The California Secretary of State
knew neither of the problem, nor of
the workaround. The problem passed certification prior to the 2006
top-to-bottom review, made it through the 2006 top-to-bottom review
undetected and existed in the version of GEMS that was re-certified
after the 2006 top-to-bottom review.
1. The ETP can effectively discover and identify errors in the
primary count, errors that otherwise would not have been detected.
2. The deck-0 problem may be affecting other jurisdiction using GEMS
version 1.18.19, or, for all we know, any version of GEMS whatsoever.
3. Even though this is the second election for the ETP, in many ways
it is a first. In the June election, we used Microsoft Windows
to control the scanner. Between June and November, with the
help of a third party open source developer, we gained the capability
to control the scanner using the open source Linux operating system.
This allows us to publish all the source code used to control the
scanner, thereby greatly increasing the transparency of the technology
we are using. Additionally, in the future I hope to automate the
process of discovering and identifying discrepancies between ETP
results and primary count results. In a few more elections, the ETP
should be able to identify errors in the primary count *before*
certification of the election results, rather than *after*. So the
ETP will only get better with time.
4. To the best of my knowledge, GEMS does not have the capability to
export machine readable per precinct/per deck results in an easily
analyzable format. If GEMS had the capability to export machine
readable information, it would be a lot easier to audit GEMS (even
without the ETP). AS it stands, it is very difficult to notice that
decks or precincts have been dropped from the final report due to this
serious deficiency in the design of GEMS. (If GEMS does have such a
data export capability, no one has been able to tell me how to use it
- not even Premier's election support specialists.)