Caching Round-trip Discrepancy

Incident Report for Flatfile

Postmortem

Introduction

A recent update introduced a change to how sheet counts are processed. As part of this update, we began including a timestamp (countsComputedAt) to indicate when the counts were calculated. Shortly after being deployed to production, several endpoints (such as get workbook, update records, patch job) started returning 400 errors with the message recordCount.updated_at.getTime is not a function.

Incident Details

Date Reported: April 24, 2025

What Went Wrong:

Data from workbooks is cached for performance. When this cached data is retrieved, it goes through a process that converts date objects into plain text (specifically, ISO8601-formatted strings). This conversion removes the special capabilities of a date object—such as calculating the time—causing our system to break when it tried to use those capabilities.

This error did not appear in our testing or development environments because caching is disabled there, so the date information remained intact and behaved as expected.

Why It Wasn’t Caught Earlier:

Development and test environments differ from production: Since caching is turned off during development and testing, we didn’t see the issue where the date value was converted into a plain string.

Incomplete type checking: The part of our system responsible for managing cached data didn’t reflect that the cached date values had changed from their original form, so the issue wasn’t flagged by our automated checks.

Next Steps:

We’re updating our development environment to more closely mirror production so that we can catch this type of issue earlier in the future. Additionally, we’re improving our type handling and tests around cached data to ensure these transformations are properly accounted for going forward.

Resolution & Fix

  1. Production was rolled back while the issue was investigated.
  2. The workbook presenter logic was updated to handle the different types resulting from caching behavior.
  3. The local and test environments were updated to perform type conversion consistently with production, even when caching is disabled.
Posted Apr 24, 2025 - 12:09 PDT

Resolved

A recent update introduced a change to how sheet counts are processed. As part of this update, we began including a timestamp (countsComputedAt) to indicate when the counts were calculated. Shortly after being deployed to production, several endpoints (such as get workbook, update records, patch job) started returning 400 errors with the message recordCount.updated_at.getTime is not a function.
Posted Apr 24, 2025 - 10:00 PDT