A series of compounding events on the US region Platform led to network throttling of a network file system which severely degraded file extractions on the Platform.
At approximately 10 AM PDT on June 11, Flatfile experienced several conflating events that resulted in network bandwidth on our network file systems being overwhelmed and throttled by our cloud service provider. The event resulted from a spike in usage on the Platform that led to autoscaling events which dramatically increased read operations against the network file stores, eventually leading to exhaustion of throughput.
This event led to a severe degradation in file extraction performance on the Platform for all customers. The initial response to the incident involved increasing capacity which compounded the problem. At approximately 14:20 PDT, the engineering team provisioned a large increase in throughput which ended the throttling and allowed file extraction jobs to proceed as expected.
The incident severely degraded all file extractions and led to increased latency on jobs as the queue became backlogged.
The throttling of throughput on the network file system was the cause of the incident; this was exacerbated by automated scaling operations that attempted to increase queue capacity.
Flatfile engineering provisioned increased throughput on the file system to resolve the incident at 14:20 PDT.
Please be assured that this incident did not compromise the security or integrity of your data. Our commitment to data protection remains a top priority.