Deletion of API keys

Incident Report for Hummingbot Miner

Postmortem

This issue is now resolved. Hummingbot Miners is now back to normal, and your API keys have now been restored.

As a brief post-mortem, we introduced a database fix to invalidate API keys of users who used keys linked to the same Binance account in an attempt to earn multiple rewards for the same order. While they were unable to do so because our database enforces uniqueness of orderIDs, it also had the unfortunate side effect of stalling our database operations.

Unfortunately the fix was problematic and affected many more users than it should have. We've fixed the fix, restored the invalidated API keys, and replayed our Binance historical order data stream.

During the outage, users who kept their bots running have continued to earn rewards.

We apologize again for the technical issues and strive to improve the system!

Posted Mar 26, 2020 - 16:35 PDT

Resolved

This issue is now resolved, and Hummingbot Miners is now back to normal.

Posted Mar 22, 2020 - 00:00 PDT

Monitoring

We have fixed the bug and re-deployed the change to the Rewards Engine that prevents cheaters. In addition, we have replayed data from our Exchange Data Fetcher, and have allocated rewards to users who had their keys invalidated. We will monitor to ensure that it doesn't affect honest users.

Posted Mar 21, 2020 - 19:00 PDT

Identified

We identified that a change in the Rewards Engine intended to prevent users from adding duplicate exchange accounts had inadvertently invalidated keys of many honest users.

Posted Mar 21, 2020 - 10:00 PDT

Investigating

Some users have reported that their API keys were being automated deleted in Hummingbot Miner.

Posted Mar 21, 2020 - 01:00 PDT