Community Backtesting Performance Issues
Incident Report for Quantopian
Resolved
This incident has been resolved.
Posted Dec 04, 2019 - 19:17 EST
Update
The recovery of transaction and position data for backtests from August 28th to present is underway, however we expect it to take most of the night.

Paper trading algorithms are being prepared for tomorrow, and we expect that they will process as expected as of market open tomorrow.
Posted Dec 04, 2019 - 19:17 EST
Monitoring
Backtesting performance for new and currently-running backtests should now be back to normal.

As part of remediation, we removed position and transaction information from backtests start since August 28th, 2019. We believe we can restore this data and are working on doing so now.

The performance problems also resulted in some paper trading algorithms not finishing correctly today. We will be re-processing those algorithms as we can.
Posted Dec 04, 2019 - 17:05 EST
Identified
We have identified the source of the performance problem and are proceeding with an emergency remediation. We expect that the timeout issues will get worse for a brief period.
Posted Dec 04, 2019 - 16:39 EST
Update
We continue to investigate the source of backtesting performance issues. We are evaluating a handful of options for improving the situation temporarily.
Posted Dec 04, 2019 - 16:15 EST
Investigating
We are investigating performance and timeout issues in our community backtesting platform.
Posted Dec 04, 2019 - 15:08 EST
This incident affected: Backtesting.