EOS Stops Functioning, Network Down

1

EOS is down. The network was frozen around 11AM London time due to an unclear issue. “We are aware of the issue. We will provide a full report shortly,” Rick Schlesinger from EOS New York says. In a statement they say:

“AT 9:56 UTC the EOS Mainnet paused… At 10:57 UTC The choice was made that all Standby Block Producers would stop their nodes and backup any information that could helpful in diagnosing the problem.

At 11:02 UTC, a method to unpause the chain was formulated and is currently underway. Normal EOS chain functions should be available within 3 to 6 hours from the time of the publication.”

The chain has stopped at block 1,028,053. A blockchain explorer shows supernodes are out of sync. Schlesinger says:

“We are currently diagnosing the EOS mainnet and will be updating everyone with a full diagnosis shortly. DDOS was NOT the cause.”

They are not ready to say at this stage what exactly is the cause, Schlesinger says:

“We are running tests to confirm the issue. Better to check thrice and validate with all BPs before saying. We’d like to provide the most accurate information so it can be audited by the community.”

13:27 London Time: While some block explorers show the chain has frozen, new blocks are apparently being produced. Andrei Nadejde says:

“Block explorer uses what node? Probably one of the ones who stopped. This is interesting. It’s sort of a fork. If some nodes are still producing, and the coalition of bps stoped and will restart at the block they selected, then basically we will have two chains running with the same ID.

If the ones still going on now don’t fall into line, I wonder what that will do. Right before they starting crapping out:

1027925 ddos gu2tembqgage {“memo”:”ddos eos! short eos”}
1027915 ddos gu2tembqgage {“memo”:”ddos eos! short eos”}
1027914 ddos gu2tembqgage {“memo”:”come on! lets DDOS for free and short eos!”}
1027905 ddos gu2tembqgage {“memo”:”ddos eos! short eos”}
1027895 ddos gu2tembqgage {“memo”:”ddos eos! short eos”}
1027894 ddos gu2tembqgage {“memo”:”come on! lets DDOS for free and short eos!”}
1027885 ddos gu2tembqgage {“memo”:”ddos eos! short eos”}”

If new blocks are being produced then all are urged to not transact. Exchanges should stop deposits and withdrawals. There might be a fork. If transacting wait for a stupendously high number of confirmations. Better to not transact and to not accept transactions until the situation clears.

13:39 There might be a reorg, that is a reversal of history. Supernodes have stopped at block 1,027,927, but the chain has managed to get up to 1,028,053. That’s a difference of some 1,000 blocks.

How exactly they will address this, however, remains unclear.

14:06 “Block one seems to have identified the issue. We need to confirm this and roll out to everyone.”

What the issue is exactly has not been revealed publicly, but someone from SonataSystems, which seems to be working on EOS, says:

“Feels like a racing condition in the code. I wouldn’t characterize it as an edge case, given this issue paused the mainnet within just a few days of full operation.

If so, these are the worst kind of bugs to resolve– because they aren’t detectable in simple unit tests, usually requiring a sophisticated integration test between multiple nodes; even if such integration tests were written they wouldn’t be really effective unless such tests were executed in a realistic testnet configured as in production with similar load levels (server and network) and similar transaction traffic.

If this realistic “acceptance” test environment doesn’t exist, it should. Block.one should maintain it. They’ve got the $$$$, and this chain manages billions….”

However as stated we do not yet know what exactly caused this pause of the blockchain just around one day after it passed the threshold vote.

It appears there was a new code release and there may have been protocol incompatibilities, with an apparent coder stating:

“As a result of the code review, there are very few code changes in 1.03 and 1.04. The compatibility of the minor version update does not seem to be tested and seems to be a problem.

We also frequently use vulnerable functions such as memcpy (). Secure coding guide is not applied.

Is this the main net?”

But we do not quite know at this point what caused this “pause” of the blockchain.

14:19 “The following steps will be taken:
1. A nodeos fix will be released
2. The block producer nodes will be upgraded
3. Blocks will be replayed until last irreversible block
4. Block producers will sync to each other
5. Chain will be opened up for mainnet”

14:24 “Chain will be upgraded to version 1.0.5. No data on the blockchain will be lost.”

We’re not sure how they’ll manage that. There are 1,000 blocks unaccounted by seemingly only one Block Producer.

14:38 Looks like this was the issue. A consensus bug was found around 17 hours ago and it is now fixed.

“Root cause was due to how deferred transactions were handled. Fix is being released shortly by BlockOne,” Eosys says.

14:51 EOS’s price has not really responded much so far, falling “only” 2%.

Trading is still open, but the network is inoperational. When exactly it will be back up is unclear at this stage, but hopefully in a few hours.

The fix is now merged and supernodes can build, but v 1.05 has not yet been released. Why this took 17 hours to be fixed is unclear at this stage, with the focus understandably being on addressing the very urgent matter of the blockchain being paused.

15:13 The chain might start get up and running again shortly once all supernodes upgrade and one node enables stale production:

15:30 Unclear what’s goin on. The network is sort of up but it doesn’t look like it is operating as normal at this stage.

15:34 The chain is now kind of up and running around five hours after it went down. We’d recommend caution in the next few hours and we’d recommend you still do not transact and do not accept transactions until some more time has passed and EOS is running without problems.

We might close the live update at this point however as the issue appears to have been resolved now.

Copyrights Trustnodes.com

 

Leave a Reply

1 Comment on "EOS Stops Functioning, Network Down"

100000
  Subscribe  
newest oldest most voted
Notify of

What a flaming garbageball. How anyone would trust this network to transact is beyond me. This post about scaling was right, ETH, EOS, BCH are all shambling wrecks that will implode under scaling pressure — https://hackernoon.com/the-ethereum-blockchain-size-has-exceeded-1tb-and-yes-its-an-issue-2b650b5f4f62

Coin security and scaling are hard, and just because someone has new jargon to explain their flashy network, doesn’t mean its secure, or worth using.