Sunday, December 10, 2017

Auditing Update

This is an update to the previous post based on a memo issued by the Tribunal Supremo Electoral (TSE)  (not Audisis) on the crash of the TSE servers on November 29.  This update is a result of finding a technical appendix on another copy of the TSE memo.

First, lets make it clear the memo comes from the TSE, not Audisis as the press coverage implied.

The appendix describes the vote counting system as a group of clustered servers.  Scanned acta images are transmitted from the INFOP warehouse to a cluster of two receivers at the TSE.  Transmission is via JSON, a markup language, over a secure network connection.  From there, the JSON markup is converted to a database record, transitioning through an application software cluster of two servers, and a database cluster of two servers.  We learn for the first time what we had guessed; that the database software is MS SQL.  The database cluster talks to a single  database of 600 Gbytes on a SAN system which has 12 TBytes of free storage.  The transcription of the actas takes place on a series of desktops connected to the application software level over a network.  We know that there are at least 60 computers doing transcription today, and probably more.  The Technical diagram does not show the web dissemination of data a part of this process.

The appendix says that Dell AMC sized the original database at 600 GBytes based on recommendations of the vendor of the transcription software.

The tecnical appendix has confused and undoubtedly not correct set of times for events, and contradicts the main report of what happened with respect to the database.  We will use the times reported in the appendix here, but compare with the times reported in our previous blog post, which came from the body of the report.

Both the appendix and the body of the report agree that the servers ran out of space and went down at 9:47 am on November 29.  They shut down the transcription system and enlarged the database to 1.8 TBytes.  They repeat that it took 3 hours 20 minutes to enlarge the database and bring the transcription system back up, coming back on line about 1:10 pm the same day.  When they restarted transcription, the MS SQL database was unstable, needing to be restarted every 10 minutes.  No explanation of why it kept crashing or why they thought taking the next steps would resolve it.

At 5:30 pm they took down Node 1 of the database cluster, leaving Node 2 to do all database access. This supposedly restored some stability to the Transcription service.  At the time they took Node 1 off the cluster, they decided to install a new database instance on it, in case Node 2 started crashing.  This involved reinstalling MS SQL on Node 1, which they renamed SQL4, which they then gave a 6 TByte database.  They also configured another server, SQL5, as a mirror of SQL4, with a 2 TByte database.  They took a snapshot of the database on Node 2 at 9:47 pm? and restored it on to SQL4 and SQL5, finishing about 1:10 am? (the appendix says PM but that's not possible unless it took into the next afternoon, long after they'd restarted the transcription process).  The transcription server was then reconfigured to use SQL4 and Node 2 and its database were permanently retired. SQL5 collected a mirror of the database but otherwise was not part of the transcription process.

Audisis reportedly then audited what the TSE had done.  The TSE claims what they did was completely transparent.  This description of a reformat of the server and installation of a new, larger database on the SAN matches with what the Alianza reported when they stated that the TSE had formatted the server.  They did.

Just as a point of normal procedure, it should have been Audisis that released a report on the changes to the system, not the TSE, and it should have been Audisis in its own words reporting on the integrity, or lack thereof, of the systems after the change.  Instead the TSE chose to put words in Audisis's mouth.  That they managed to nearly fill up a 12 TByte SAN with databases is remarkable.

Saturday, December 9, 2017

Auditing a Failed Election

Auditoría Integral y Seguridad de Sistemas de Información (Audisis),the Colombian firm hired by the TSE on November 13 to audit the election results for the November 26th election, has released an unsatisfactory memo detailing what caused the TSE computers to fail on November 29,  and what modifications were made to the system to fix it.  In the process it raises more questions than it answers.

The Tribunal Supremo Electoral (TSE) announced it had reached a contract with Audisis to audit the election results of the November 26th election on November 13th.  David Matamoros Batson informed everyone at the time that they were lucky to have found such a qualified firm for only $700,000.  Matamoros stated that Audisis and the TSE had agreed on 7 points to be audited:
(1) Security of the actas, that the image scanned in the voting center is the same as the image received by the TSE to count.
(2) verify the software used by the TSE for counting and publicizing the tallies is what is being used.
(3) network security
(4) verify the functioning of the vote tallying software for all the elections being run on November 26.
(5) security of the database
(6) make sure transmitted actas are shared with the political parties
(7) evaluate the suitability of the technology being used to publicize the results.

So with the above in mind, Audisis released, through the TSE, a "report" of what happened when the system went down on November 29th.  According to Audisis, the system went down at 9:42 AM because it had filled up its database.  This, by the way, contradicts President Juan Orlando Hernandez who claims it never crashed or became unavailable, just slow. 

What the graphic in the report shows is two server instances running with a 12 Terabyte SAN storage network, but only a 600 Gigabyte database allocated, and apparently shared between the two servers, which are clustered for high availability.  It then took them 3.2 hours to expand the database to 1.8 Terabytes, bring up the servers, and perform a minimal data audit.  The servers were up for production again at 1:08 pm and they began adding actas again at 1:10 pm.

 They continued to observe problems with database performance and decided to bring the system down again at 6 pm the same day.  They increased the database size again; this time to 6 Terabytes.  It took them 5 hours 30 minutes to reconfigure the system to use the additional capacity.  They also added a 3rd server, this one configured with a 1.8 Terabyte database, to receive replicated data from the original database as a check of system integrity.  The system returned to production around 11:30 pm that evening, almost 9 hours after it was halted.

The first thing you do when you design a database is design the table structure, then make a good faith estimate of how much storage space the data will need.  You always give a healthy overestimate because you have to remove the database from use to increase its size.  When you're processing election data, you don't want that to happen.  With the kinds of data we are dealing with here, you should be able to make very precise estimates of how much storage space should be needed. Yet somehow they failed.

I can't fathom how they filled up a 600 Gigabyte database, even with all the acta images from Presidential, Congressional, and Municipal elections and a complete voter roll stored in the database, I estimated it would only take about 34 Gigabytes of database storage to process the results of the election.  I, after all, stored the complete results of the 2013 election in a database on my laptop without it taking up even 20 Gigabytes.  Even with every conceivable kind of transaction logging turned on, I'd be hard pressed to design a database requiring 100 Gigabytes.  What were they doing?

Replication put simply is the ability to have two or more databases stay synchronized to provide greater availability.  If they are in geographically different locations they can also be used for disaster recovery.  Since the databases need to remain identical, normally they would need to be of the same size.  So why would you replicate data from a 6 Terabyte database to a 1.8 Terabyte database?  Doesn't that mean you didn't need the 6 Terabyte database size?  Or even the 1.8 Terabyte database size?  Unclear from the report released by the TSE is whether the third server added already had a cloned database, so that only newly added data needed to be replicated, or whether the database was empty and needed to replicate all the existing data across a network.  While replication is designed by database vendors to minimize its impact on servers, it still has a measurable impact.

In many ways the report released by the TSE, supposedly compiled by Audisis, raises more questions than it answers.  It provides an excuse for why the TSE systems went down for almost 10 hours, but the reason doesn't seem credible. 

Monday, December 4, 2017

Analyses support doubts about vote and OAS disclaims results while police refuse to stop protests

Our headline may seem to juxtapose three unrelated things. But we think they have to be seen together. And actually, there's one more thing... and it's a doozy.

To recap where the process stands: the TSE resumed counting vote tallies without representatives of the other parties. At the end, it says its count shows the incumbent president with a lead of around 55,000 votes. According to the TSE, the next step is for the parties to register challenges and petitions, within 10 days, and then within 20 days the TSE will certify an outcome to the election.

(In reporting this story, the Voice of America slipped up, describing Juan Orlando Hernández as "U.S.-backed"; the US doesn't normally take sides in a foreign election.)

No one who has watched the situation unfold can be completely satisfied that the vote count has been transparent or without problems. Unexplained prolonged delays in posting numbers, computer crashes that received different explanations at different times, and above all, the weird behavior of the numbers before and after the more than day-long delay, have Hondurans and international observers alike worried.

The OAS actually went so far in its preliminary report to conclude that "the tight margin of the results, and the irregularities, errors and systemic problems that have surrounded this election do not allow the Mission to have certainty about the results".

In the body of the report, the OAS expresses concerns about the vote counting process, noting some ballot boxes arrived open, missing documents, or without security. They also write that after initially counting ballots as they arrived, at some point the TSE "altered the order" to use "criteria that were not explained". So the vote counting switched from non-selective, to selective-- but we don't know what criteria were used to select votes to count.

The OAS concluded that the only route out would be for the main candidates to negotiate an agreement to review the 1000+ poll tallies that were scrutinized for inconsistencies, as well as recount the 5000+ tallies counted after the initial phase of vote counting, when the trend changed, as well as do a complete recount of three departments (Lempira, La Paz, and Intibuca), rural states that had exaggeratedly high reported voter turnout. That is a complete endorsement of the position of the Alianza.

Independently, The Economist, which previously published an article about a tape they received apparently showing training of National Party operatives in ways to cheat, undertook a statistical analysis that gives support to Alianza complaints that the change in voting trends after the break in counting is statistically improbable.

And that brings us to today's amazing development: the police across Honduras, including the US supported militarized policing units, standing down and returning to their bases, refusing to follow the orders they received to stop protests. Under the state of exception declared by Juan Orlando Hernández, free circulation in the country was limited, a night time curfew was declared, and the armed forces and police were directed to remove protesters. What followed was violence, including deadly violence.

Announcing their stand-down, the national police spokesman said "“We want peace, and we will not follow government orders – we’re tired of this". 

When a sitting president who has concentrated power loses the ability to command the police, it is a signal of loss of control over the forces necessary to maintain dominance. Even if the TSE were to declare him the winner, it is not clear how governable Honduras would be for a president who took advantage of a somewhat ambiguous court ruling to seek a deeply unpopular second term in office.

After the 2013 election, when Hernández received only 37% of the vote, the three parties that split the majority of the presidential votes did not cooperate as a concerted opposition. This time around, two of those parties entered an alliance and ran an agreed on presidential candidate.  This time, the Liberal Party candidate who trailed in the polls has been vocal in saying his review of the poll tallies says the Alianza won, and has supported their calls for a recount, even a full 100% recount if needed.

And here's the extra bit: according to a Honduran lawyer, whose twitter profile says she is a Liberal Party member, election law actually demands a recount of some votes already.

This isn't because of the uncertainties about counting the poll tallies that are already being debated.

It's because the margin between candidates is less than the number of null votes. Null votes are those marked as invalid at the polling place, and thus not included in the totals on the poll tallies from which the central electoral authorities work.

The law appears to require reviewing the null votes from the original ballots, if there are more of them than the margin between candidates. With around 55,000 votes officially between the two candidates, the number of votes marked null at the polling places is 135,000.

The TSE is unlikely to do any of this. Unfortunately, we doubt Hernández will risk the victory he went so far to gain and agree to the kind of recount and scrutiny of the counting process that is being called for by the Alianza, the Liberal Party-- and the OAS.

Until the army stands down and returns to its barracks. Unlikely, yes. But stranger things seem to be happening...

Saturday, December 2, 2017

Government Curfew and Suspension of Right to Move

Juan Orlando Hernandez declared a curfew and suspended some constitutional rights of Hondurans yesterday in response to protests against the way his party is stealing the election.

The first part of Hernandez's decree declares a 6 pm to 6 am curfew, and removes for 10 days the constitutional right of the people to freely move around the country.  It exempts those working on the election (TSE employees, National and International Observers, Party Officials), cargo transport, Police, Military, and Diplomatic officials.  Notice that the press is not free to move around or cover what's happening.

The second part of the decree directs the military to take whatever measures it needs to maintain order, either separately or in conjunction with the police.  It orders them to arrest and detain anyone who is out during the curfew, to read them their rights, and to maintain a list of those arrested at every military post.

All those arrested will be held until the Prosecutor's office can charge them with a crime.

The military is to clear all the roads, bridges, and public places seized by protesters. UPDATE: order from President this morning also calls for removing protesters from private property. This makes any protest a target for repression; goes beyond the official order.

All government parties are to assist the military as requested.

Thursday, November 30, 2017

Controversy about votes being "monitored"

 UPDATED to add Santa Barbara and Cortes to list of departments where votes held for "monitoreo" favored the Alianza; and to note that the TSE has said it will count all votes, something we are seeing happen as we go through our list of tallies in monitoreo.

Today. reports from Honduras indicated that the Tribunal Supremo Electoral was holding about 300,000 uncounted votes in a state of "monitoreo". They offered no explanation why. What was expressed was that these votes would not be counted before a winner was named in the now extremely tight contest between Juan Orlando Hernández and Salvador Nasralla.

This alarmed international observers, who observed-- quite rightly-- that this many uncounted votes could well swing such a tight election. Both the OAS and the European Union publicly called on the TSE to count all the votes before designating a victor.

Remarkably, this call was echoed by COHEP, the Honduran council of private enterprise.

The EU in particular urged the TSE to take the time to count the votes, so that every vote was recognized, rather than hurry to end the election prematurely. UPDATE: as of 10 PM Tegucigalpa time, the TSE has said it will count all votes; and we are seeing the status change as we go through our list.

We decided to review the vote tallies that are being held for greater scrutiny-- or monitoreo-- ourselves, to see if the suspicion many have, that this includes a preponderance of pro-Alianza voting, was upheld.

It may take us a few hours. So far, though, in the Departments of Atlantida, Colon, Cortes, Santa Barbara, Valle and Yoro, the total of votes on these uncounted tallies for the Alianza is higher than the total for the National Party. (We are suspending this project at 1 AM Tegucigalpa time, as the TSE continues to update some of these. We will spot check other departments tomorrow...)

There may be reasons these tallies require extra scrutiny. There are check sum features built into the tallies, so errors in transcription or uncertainties about numerals can be resolved in many cases. In others, the question would be if, for example, over-writing on one line should result in ignoring the votes for other candidates.

So far, though, it is clear that this vote pool reserved from counting would contribute to shifting the margin back in the other direction.

Election update: four days since polling ended

Yesterday the Tribunal Supremo Electoral said it would make an announcement of final vote count at 3 AM local time Thursday.

One assumes that was a projection based on the pace of counting, not (just) a way to try to avoid having people awake and paying attention. We did not set an alarm, which is just as well, since nothing was announced at 3 AM.

In part, that may be due to an as-yet incompletely explained event that affected the computer equipment Wednesday evening. This took the entire TSE system down with about 82% of the votes counted-- just after the vote had swung slightly to favor Hernández.

The explanation offered by David Matamoros, head of the electoral tribunal, was that this was a computer breakdown, due to the high volume of data being too much for the system used, requiring additional servers to come online. Continuing a pattern of uncertainty and confusion stemming from the Tribunal, another tribunal member, Marco Ramiro Lobo, was quoted as saying the system had been "hacked".

Regardless of the actual cause, the break in the technology came at an unfortunate point in the process. Moments before, an agreement (since repudiated) was released, brokered by the OAS, in which the two candidates agreed to accept the numbers that the TSE was supposed to be reporting in the early morning.

At 8 AM Thursday, Tegucigalpa time, the count is still stalled at just under 89%.

The vote count posted favors Juan Orlando Hernández by 23,000, out of a total of 2.92 million votes-- less than 1% difference.

Due to the procedures used by the electoral tribunal, it is impossible to be certain which polling places have yet to be tabulated. Where the 11% of votes still outstanding comes from is critical, because of the sharp differences in vote preference from region to region.

For example, in the Department of Cortes, where Salvador Nasralla has won 56% of the 404,000 votes counted, we can compare to the 2013 results, which showed a total of 516,000 voters. The possibility of there being more than 100,000 votes still uncounted from this region could be enough to shift the totals, if the current 56%/32% split of vote there continued, as that would be a 24,000 vote advantage for Nasralla.

This won't be settled until every vote has been counted. As the slow process drips on, Honduran citizens continue to have their trust in democratic institutions eroded.

And it appears that the almost inevitable round of repression of protest has also begun, with twitter reporting (and photos confirming) the militarized police or military tear-gassing protesters assembled outside the location of the counting in Tegucigalpa last evening.

It could be easy to lose sight of one clear lesson in this election: even if the incumbent president somehow holds on for a second term, against the popular rejection of presidential re-election seen in pre-election opinion surveys, the opposition campaign mobilized a far larger group of voters than international observers expected.

They maintained the level of support seen in the 2013 election, when it was split between the component Partido Anti-corrupción and LIBRE parties that make up the present Alianza, thus allowing Hernández to win with only 37% of the 2013 vote.

Whether denied office this year or not, the Alianza should be a political force to reckon with over the next four years, representing as many Honduran voters as the Partido Nacional, inheriting the role long played by the now diminished Partido Liberal as the counter to that political force.

Wednesday, November 29, 2017

Rodolfo Pastor Fasquelle on the election

¿Como? ¿Quienes les ganamos, a quienes, porque y para que? 
How? Who won, over whom, why and what for?
 To my skeptical friends:

The two parties of opposition have, since yesterday, counted all the poll tallies and the Alianza wins with the same 5% margin that was published by the Tribunal Supremo Electoral on Monday. Interesting that the victory could be so overwhelming when the polls by the Liberal and National parties, of private business and some by the Embassy continued foretelling the defeat of the Alianza up to a couple of weeks ago, that it would not manage to harvest more than 25%. The press in the USA followed, giving the win to JOH up to the very day of the elections. In Northern Europe, the same. On the one hand, important sectors there would prefer the status quo, to foretell his triumph was a way to support him. On the other hand, people were afraid to say it, and hid their determination. The rest is obvious.

Certainly, it is an extreme case. In the electoral canton in which I am selected to vote and to supervise, in Las Carmenes, in San Pedro, Luis Zelaya didn't even get 10%. The Alianza won every polling place over JOH, almost 2:1, 60% to 30%, despite the efforts of the mancha brava to pull in votes first, and later to insert twisted votes, and in the end, to impede people from continuing to vote, by prematurely closing the gates. But even in the neighborhoods of the bourgeousie, in San Pedro Sula, the Alianza won the majority of polling places. And the margin of advantage for Salvador on the North Coast and the west-- with the most urbanized population-- will keep safe the final result. We can say the triumph of the Alianza, thanks to Salvador, and God help him. We lost the mayors. There is no local leadership of the Alianza!

This stubbornness in hanging on to executive power is also interesting, which is not only personal to the despot. The first news indicated that los cachos (the conservatives) would hold many municipalities and-- thanks to the atavistic slate-- an important part of the Legislature. This instinctive tenacity of the mancha brava and total lack of responsibility is a dangerous bet, because JOH couldn't govern with 60% of public opinion against him and could lose everything. This could empowered Nasralla in his position to call for a strike and a Constitutional convention now, and it would be over. They could disappear. As happened with the Partido Liberal.

Clearly, there is not on this side less will to defend the conquest, and there are more of us. Who won? Everyone! Beyond Honduras-- that might finally truly reconcile itself, beginning with the recognition of that necessity-- Nasralla won of course, and the recomposed opposition, when LIBRE became the largest party with Mel at its head. Those who asked for the vote and those who, without being candidates, worked to attract them won, as -- in the end-- definitively, we won the voters. Some electoral tribunal officials [Marco Ramiro Lobo] will gain in general recognition, although the performance of the TSE in the end was calamitous. Never again! At the moment, the militia is gaining credibility!

We are defeating their greater resources and their better organization with our grip and enthusiasm. We are winning by overcoming indolence, indifference, negativity, the inertia of apathy, both our own and that of the voter. Overcoming in some cases the fear derived from the pre-electoral violence or the impression that the triumph of fraud was inevitable. Attaining the vote, with our own means, at times also with some sacrifice. Voting massively with a level of participation that hasn't been seen since the 1980s. We cusucos, armadillos, were too much for the fraud. The voters favored us with a majority very much higher than the official figure.

The notable failure of the eternal traps is also interesting. That, at least in the urban ambit, these don't work anymore. Could fraud influence the count of the totality of the vote for congress members, that the cachos, better organized, count on their own and in their manner? Could be. But the principal problem there is that there is no binding theme like re-election, and we voters of the Alianza are more critical and rebellious than the cachurecos. Not that there wasn't, it's clear there was, cheating. It is tiring to list them. Over in the lands of Fito Irías, famous for his gadgets, various activists were detained filling ballot boxes, afterward sent via helicopter to the site in Tegucigalpa, where at the last moment the mancha brava tried to rob ballot boxes. Here in El Ocotillo there were detected and detained bad citizens who were creating el trencito, a little train: one entered the polling station with hidden marked votes, deposited those that he carried and pulled out another group of sealed ballots, for the next and charged... they fished. Everywhere there appeared marked as deceased the people that someone supposed were ours. In El Carmen there was a a voter who was asked for his vote and ID, to photograph his vote to go out and sell it. Angry young men hurried by the well-paid activists who walked from midday on with money and some cases of falsified ID cards paying people to vote with them in the polling places where they controlled the oversight, which was discovered when-- later-- the actual owners of the identities arrived to vote, finding that someone had already voted for them, living a fingerprint "por no saber firmar" (not knowing how to write).

It wasn't defining. Now what before, on various occasions was just PR of the system was-- really-- a civic festival. There was a complicity among a great diversity of classes, of ages, of affiliations. All people, determined to "remove JOH". Great happiness. And after, equal worry, widely shared. (The manipulation by technicians of the TSE is another role. Almost incredible.)

Over whom did we win? Over JOH the dictator, because he undertook to make it something personal, and his gang, the deceitful cahirecos and their mancha brava, the imposters who see visions, the Cardinal, who was making an electoral intervention until the last minute, denouncing la intolerable injerencia extranjera (the intolerable foreign interference), we won over the bosses Flores and Micheletti, who have never lost everything, and the corrupt journalists and the media companies that misinform and mistake ratings for agreement, over the foreign lobbyists that have come to do so much damage, and also, J.J. Rendón and Arcadia lost.

Why did we win? It wasn't just that Salvador attracted voters or that the people identified with our proposals. Nor was it only a vote against re-election. We won because the people got fed up with JOH. With the omnipresence and almost omnipotence of JOH. They punished him as vain and prepotent, as abusive and cynical, and they will return to do it when it's needed. They voted to restore the agreed upon order and clean up the government. Those of us connected wish to preserve a space for our liberty, menaced by the dismantling of the state of law and sufficient guarantees.

La gente lee de otro modo,  pero cree en su voto.
The people read in another way, but they believe in their vote.

[our translation]