Duplicate accounts issue

Incident Report for Hummingbot Miner

Postmortem

As a brief post-mortem, we introduced a database fix to invalidate API keys of users who used keys linked to the same Binance account in an attempt to earn multiple rewards for the same order. While they were unable to do so because our database enforces uniqueness of orderIDs, it also had the unfortunate side effect of stalling our database operations.

Unfortunately the fix was problematic and affected many more users than it should have. We've fixed the fix, restored the invalidated API keys, and replayed our Binance historical order data stream.

During the outage, users who kept their bots running have continued to earn rewards.

We apologize again for the technical issues and strive to improve the system!

Posted Mar 27, 2020 - 12:09 PDT

Resolved

On March 20, the Rewards Engine began rejecting orders because multiple miners were trying to submit orders from the same Binance account.

Investigating
As of 3am PST, the Hummingbot Miner app has been experiencing issues again - we are triaging currently and will provide an update when we identify the cause.

We apologize for these teething pains and will strive to get it back up as soon as possible.

Identified:
We have identified the component in our pipeline that failed and will roll out a fix.

Resolved:
As of 5pm PST, we have brought Miner and Rewards Engine back online. We replayed the process of rewards from our data fetcher afterwards, so rewards have been updated.

Posted Mar 20, 2020 - 17:00 PDT