Rewards engine stoppage during deployment

Incident Report for Hummingbot Miner

Resolved

This incident has been resolved.
Posted May 09, 2020 - 04:02 UTC

Update

We have reverted the patch and restored the Rewards Engine.
Posted May 08, 2020 - 16:42 UTC

Update

We're still working on this issue. We rolled out a fix but it exhibited unusual behavior during monitoring so we are currently investigating.
Posted May 08, 2020 - 06:02 UTC

Update

We are continuing to work on a fix for this issue.
Posted May 08, 2020 - 01:31 UTC

Identified

During a scheduled deployment this afternoon, we encountered an issue that caused the Rewards Engine component to exceed the Binance rate limit. We have identified the fix and are currently waiting for the rate limit to resolve before deploying the Rewards Engine.

Our Binance data fetcher is unaffected and running bots should keep earning rewards during this outage.
Posted May 08, 2020 - 01:31 UTC
This incident affected: Data fetchers (Binance Data Fetcher), APIs (Miner API), and Reward Aggregators (Binance Rewards Engine).