Shortly before the close of trading on September 15, one of our members reported to us that he was seeing invalid pricing data for a stock in live trading.
We immediately assigned a team of engineers to investigate. They determined that the root cause was human error while recovering from bad data provided by one of our vendors on September 9 (http://status.quantopian.com/incidents/q21n3s726m40
), coupled with a bug in our data ingestion code. The human error and bug caused us to improperly roll back the bad data before re-ingesting the corrected data from our vendor.
We are in the process of correcting the bad data, and we expect to be finished in time for live trading on September 16.
The impact of this incident is as follows:
1) Between September 12 and today, "history()" and pipeline returned incorrect data in backtesting, live trading, and the research environment for dates prior to September 12 for the following list of stocks: BATS, DG, DHF, DMF, DSM, EAD, EE, EOD, ERC, ERH, FICO, HPQ, HRB, IIM, IQI, LEO, MER_PRK, MER_PRP, NEE_PRC, NNA, OIA, PSA, PSA_PRA, PSA_PRC, PSA_PRD, PSA_PRS, PSA_PRT, PSA_PRU, PSA_PRV, PSA_PRW, PSA_PRX, PSA_PRY, PSA_PRZ, PSB, PSB_PRS, PSB_PRT, PSB_PRU, PSB_PRV, SCE_PRF, SCE_PRG, SCE_PRH, SCE_PRJ, SGA, VKI.
2) In addition, a large number of NYSE stocks had incorrect historical pricing data for the day of September 9 in the research environment.
Our experiences recovering from the bad data provided by our vendor on September 9 revealed a number of issues with our ability to recover from such bad data. We have carefully documented all of these issues, and we have already addressed more than half of them. We intend to fully address all of them in the near future, so that the next time we receive bad data from our vendor, we will be able to recover from it reliably, quickly, and correctly.
Sep 15, 20:15 EDT