This incident has been resolved. The new Rewards Engine is running smoothly and we have restored the Miner app.
During this outage, we have replayed the Rewards Engine from the Binance Data Fetcher (except for the 1-hour missing gap that initiated this issue), and have allocated liquidity rewards to bots running during that time.
Posted Mar 27, 2020 - 22:37 PDT
Monitoring
We have deployed the Rewards Engine fix to our production environment and will monitor its behavior before putting up back the Miner app.
Posted Mar 27, 2020 - 19:44 PDT
Update
We have deployed the Rewards Engine fix to our staging environment and will confirm that it works before deploying to production.
Posted Mar 27, 2020 - 18:26 PDT
Identified
We are putting up the maintenance page in order to deploy a fix to the back end system which should resolve the current issue with Hummingbot Miners.
Posted Mar 27, 2020 - 15:56 PDT
Investigating
At approx 2:00am PST last night, our Binance data fetcher went down; after one hour, it self-resolved and is now back online. During this 1 hour gap, we were not able to collect trades or orders. This gap locked up our Rewards Engine, which is responsible to processing the data and allocating rewards. Since the Rewards Engine component is down, Hummingbot Miner is not displaying any data.
Our Binance data fetcher is currently running, so any bots running during this time should get credit for its orders. We will replay the data and allocate rewards when the Rewards Rngine is back online.
We are investigating why this issue occurred. We were planning to deploy a partial fix to the Rewards Engine today and will need to see that that fix addresses this issue or if more work is needed.
Posted Mar 27, 2020 - 15:55 PDT
This incident affected: Data fetchers (Binance Data Fetcher), APIs (Miner API), and Reward Aggregators (Binance Rewards Engine).