Resolved -
This incident has been resolved.
Dec 4, 19:17 EST
Update -
The recovery of transaction and position data for backtests from August 28th to present is underway, however we expect it to take most of the night.
Paper trading algorithms are being prepared for tomorrow, and we expect that they will process as expected as of market open tomorrow.
Dec 4, 19:17 EST
Monitoring -
Backtesting performance for new and currently-running backtests should now be back to normal.
As part of remediation, we removed position and transaction information from backtests start since August 28th, 2019. We believe we can restore this data and are working on doing so now.
The performance problems also resulted in some paper trading algorithms not finishing correctly today. We will be re-processing those algorithms as we can.
Dec 4, 17:05 EST
Identified -
We have identified the source of the performance problem and are proceeding with an emergency remediation. We expect that the timeout issues will get worse for a brief period.
Dec 4, 16:39 EST
Update -
We continue to investigate the source of backtesting performance issues. We are evaluating a handful of options for improving the situation temporarily.
Dec 4, 16:15 EST
Investigating -
We are investigating performance and timeout issues in our community backtesting platform.
Dec 4, 15:08 EST
