"Shocking" election omission: VOTER-VERIFICATION CAPACITY NEVER TESTED
By Katy Human, Denver Post Staff Writer, November 16, 2006

Denver election officials never tested the capacity of the troubled computer systems they used to verify voter registrations on Election Day - an omission one computer expert called "shocking" and others said seemed shortsighted.

Elsewhere, county technicians or contractors load-tested communication lines and central servers to make sure they never neared capacity, so election workers could process voters quickly and keep lines moving.

Denver officials, however, never monitored election computer performance, even during the Aug. 8 primary, when Denver first used vote centers and revised software, Denver Election Commission spokesman Alton Dillard said Wednesday.

"The type of capacity report you request would have had to be set up before November 7," Dillard wrote in response to a request for the information. "If you know of any way capacity can be retroactively determined, please let me know."

"That's shocking," responded computer science professor David Wagner at the University of California, Berkeley.

"If they didn't know in advance the capacity of their own system, how did they know it would be able to handle Election Day demand?" he asked. "This is Engineering 101."

Denver's computer problems created hours-long lines at many of the city's 55 voting centers. Other technical issues and ballot printing errors have delayed a final vote tally.

Officials Wednesday completed counting absentee ballots, meaning only provisional ballots remained. Under state law, those must be counted by Tuesday.

Over the weekend, the commission's technology director, Anthony Rainey, was put on administrative leave. And Mayor John Hickenlooper has created an investigative task force that Wednesday began a probe of the fiasco.

Rainey on Wednesday issued a brief statement saying, "There are two sides to every story." However, he declined further comment "until a thorough investigation is complete."

In Larimer County, technical staff monitored their entire voter-registration network every second of Election Day and before, said John Lee, senior systems administrator. Systems never exceeded 20 percent capacity, he said.

During system stress tests, officials knocked out servers one by one to make sure an outage wouldn't crash the whole system, said Lee's colleague Thad Pawlikowski.

"We take it all the way down to one server," Pawlikowski said, "and we've not had it fail."

The main bottleneck involved electronic poll books, used to access voter-registration data on Denver's three central servers.

Larimer County used six servers, each of which has faster processors and more memory than Denver's three.

Larimer has about 152,000 registered voters to Denver's 271,000.

But those figures don't necessarily mean Denver's system was inadequate, said Larimer's Lee. Hardware needs depend on software, networking and other details, he said.

Denver's software - made by Sequoia Voting Systems - is proprietary, and city and Sequoia officials declined to discuss its structure.

Lee and Wagner, who saw the system operate or heard the descriptions of election judges, said it seemed poorly designed and would put intense demands on servers.

To verify a voter's registration, election workers typed names and birthdates into an Internet browser window and worked through at least three screens. Workers were supposed to manually close windows between voters, but many apparently didn't.

Open pages consumed computing resources, eventually clogging the system, Denver officials said.

"If I were in the IT department, I would have said to Sequoia, 'No, no, no, no, no!"' Lee said. "You need to make a decay or timeout feature so these pages do close and free up resources."

Berkeley's Wagner said: "It's crazy."

Sequoia officials have said they did not include a timeout feature in their software because Denver officials never requested it.

Software and hardware were not Denver's only problems.

Election officials ordered only 4,799 provisional ballots - ballots that could be cast by someone without proper ID or whose registration was in doubt. That's enough ballots for about 1.8 percent of registered voters.

Ultimately, many vote centers in Denver ran out of those ballots, and judges were reduced to letting some voters use sample ballots as provisionals.

Much-smaller Morgan County, with about 14,000 voters, had enough provisional ballots for about 7 percent, said election deputy Vickie Wiederspan.

And Douglas County ordered 130,000 provisional ballots for its 154,000 registered voters, said County Clerk Carole Murray.

"The only real benefit that can come out of a debacle like that is to make sure it doesn't happen again," Hickenlooper said Wednesday as he and City Council President Michael Hancock convened the task force reviewing the Denver Election Commission.