EOS Misled the Public on the Number of Transactions it Can Handle Says New Study

“The performance metrics observed when factors such as real world network latency and packet loss are implemented indicate that TPS in EOS approaches levels comparable to Ethereum.”

So says a new study that claims to have completed the first independent benchmark testing of the EOS software.

While previously eos blockchain explorers have shown the network can handle more than 1,200 transactions a second, the study says it can actually handle only 250 transactions a second even under unrealistic conditions of no latency. They say:

“High levels of TPS are sustained at 0 ms round trip latency. When 50 ms of RTT is added into the system, performance immediately falls below 50 TPS indicating that latency has a significant effect on throughput. Once RTT is increased to 100 ms, performance drops even further.”

Round trip time (RTT) is the length of time it takes for a signal to be sent plus the length of time it takes for an acknowledgement of that signal to be received. That’s obviously not possible at zero seconds under real world conditions, with the authors so stating:

“The research results prove the inaccuracies in performance claims and concluded that the foundation of the EOS system is built on a flawed model.”

The study, funded by the Bounties Network which is led by Brent Xu of ConsenSys, replicated the eos network to analyze its performance during two months of testing. The conclusions it reached are fairly damning. They say, for example:

“There is no formal verification protocol that occurs to validate transactions beyond the fact that that block producers are including them within transactions.

The configuration of this system introduces further suspicion as mentioned in our performance testing on the fate of orphaned transactions that are not successfully processed by the network.

In Ethereum, orphaned and stale transactions are apparent for the network to see in the available mempool, while economics are used to determine whether miners or validators choose to process those transactions.

In EOS, there is no true mechanism to determine the fate of unprocessed transactions. One theory as mentioned in our performance section is that transactions are selectively chosen for the purpose of maximizing transaction throughput in the system while bypassing difficult transactions that would cause network congestion. Picking transactions in this manner does not alleviate network congestion but rather ignores it.

What this means is that unprocessed transactions are kept in the computer heap while only easy transactions are successfully processed. Unprocessed transactions are essentially left behind as there is no access to the multi- index table that houses these transactions.

This evidence further calls into question the design of EOS and how it functions as a software that does not successfully optimize the consensus around processing the transactions that are sent to the network.”

Orphaned blocks occur when two blocks are found at or near the same time. One block has to be discarded as otherwise you’re printing money out of thin air. What happens to orphaned blocks in eos, however, is apparently not very clear.

Nor is it very clear whether this is a blockchain at all. The study says it isn’t. It’s just a database. They say:

“EOS is not a blockchain, rather a distributed homogeneous database management system, a clear distinction in that their transactions are not cryptographically validated.”

Describing what’s occurring in the above featured image, they say: “The duplicate transaction errors execute every time the block producer with faulty code processes transactions. After the duplicate transaction errors, the non faulty block producers continue to successfully process transactions.

One important point to make here is that only the block producer that is running the modified source code is able to notice the duplicate transaction errors.

Nothing in the system allows other block producers to audit the logs of the faulty block producer because the system relies on social trust that each block producer is correctly running the right software and there is no formal verification of that software.”

Eos thus runs on social “consensus” according to the study, which further says eos is not Byzantine Fault Tolerant and suffers from consensus failures. They conclude:

“The transaction throughput in the system does not exceed 250 TPS even in optimal settings with 0 ms of latency and 0% packet loss.

During tests with real world conditions of 50 ms of round trip latency and 0.01% packet loss, performance dropped below 50 TPS putting the system in close proximity to the performance that exists in Ethereum.

Consensus in EOS does not exhibit Byzantine Fault Tolerance. The use of 21 block producers in the system is arbitrary as the producers are processing transactions in a round robin manner, with no algorithmic logic when block producers behave nefariously.

Various forms of consensus failure were shown in the network testing. Consensus is entirely dependent upon the voting mechanism which is based on social consensus, not algorithmic BFT consensus.”

They say the eos set-up is not good enough even for a sidechain due to the black box nature of what block producers are doing. The authors so stating:

“EOS can potentially act as a side chain appended to other more foundationally secure networks, though the system would need to be rebuilt in order to address the problems detailed in this report.”

The tests were commissioned by ConsenSys, so one can argue there might have been some bias, but the authors state the raw data will be shared with:

“Cosmos, Ethereum Community Fund, Amis, Maker DAO, IMToken, PlatON, Status, Loom Network, Coinfund, 1KX, Transference Fund, Web 3 Foundation, Grid+, MixLabs, Ledger Capital, Enterprise Ethereum Alliance, Google, Microsoft, ConsenSys and Bo Shen – Founding Partner of Fenbushi Capital & Cofounder of Bitshares, along with academic institutions such as Duke, USC, and MIT.”

They “have committed to contributing resources to develop comprehensive reports based on Whiteblock’s raw data,” the authors say, with Google apparently having an interest in participating.

A number of problems with eos have previously been revealed, but the findings that even block producers themselves do not abide by some algorithmic logic, as in protocol rules, goes far beyond what could have previously been imagined.

Eos, in effect, is worse than Ripple. There at least there is some sort of algorithm, there are some sort of rules, albeit a consensus algorithm that requires a high level of centralization and a greater degree of trust than for proper public blockchains.

In eos, however, there appears to be no rules that have to be followed. Transactions are not cryptographically validated. Block Producers have to manually kick out misbehaving individuals, yet a bored kid could probably manually overwhelm them, let alone a kid with simple tools.

Suggesting that a word which is often incorrectly thrown around in this space, is perhaps well deserved in this case because eos appears to be a complete facade.