Post-Incident Report: Elevated API and Login Errors - August 20, 2026

Status: Resolved

Incident window: August 20, 2026, 06:31 - 08:11 PDT (13:31 - 15:11 UTC)

Affected: ShipHawk API and dashboard requests in shared production environments, plus the login service. Impact was partial rather than a complete outage: approximately 25% of overall API traffic on shiphawk.com failed during its affected window; failure rates within affected environments ranged from approximately 37% to 48%, and approximately 41% of login-service requests failed.

Not affected: In-Cart rating (and all /api/v4/rates requests), background processing (all scheduled jobs, write backs, webhooks, async label generation, tracking and carrier communications ran normally), data integrity.

Summary

On the morning of August 20, an operating-system critical security update published by Ubuntu - and applied automatically by our standard patching process - contained a defect in the web server component (nginx) that sits in front of the ShipHawk application. The affected package was published on August 19 as USN-8563-3. Ubuntu confirmed that this update introduced a regression and published USN-8563-4 the same day, reverting the problematic change pending further investigation.

While the faulty version was running, the proxy layer corrupted the URL of many incoming requests before handing them to the application. The application could not match the corrupted URLs to any known endpoint and answered 404 Not Found. The failures were immediate, clean rejections: no request was partially processed, routed to the wrong account, or lost after acceptance.

Our servers do not all download and install operating-system security updates at the same moment; update checks and installation windows are staggered across hosts. As a result, some servers downloaded the faulty nginx build before Ubuntu published the corrected package, while others checked later and downloaded the corrected build directly. Only the servers that had already downloaded the faulty package became affected when their scheduled installation ran. This is why the issue appeared intermittent: otherwise-identical requests could fail or succeed depending on which server handled them.

Even on servers running the faulty nginx package, only a subset of requests failed. The regression affected specific nginx routing rules rather than the entire proxy configuration, so many URL patterns continued to work normally on an affected server.

The incident was fully resolved by 08:11 PDT after every affected server was upgraded to the corrected package and verified healthy. No customer action was or is required.

What was affected

The numbers below count only failures caused by this incident. Ordinary 404 responses (lookups of records that genuinely don't exist, invalid URLs, bot traffic) were identified by their distinct response signature and excluded.

‌

What this looked like in practice:

• API integrations received HTTP 404 responses for valid requests. Because failures were immediate and stateless, client retries could succeed when they landed on an unaffected server.
• Dashboard and login pages failed to load or sign in intermittently.
• Failures depended on the exact URL: some request types passed through unaffected even on faulty servers, adding to the intermittent appearance.

What was NOT affected

• In-cart rating requests. All rating requests from the web portal, e-commerce platforms, ERP platforms and regular API requests to /api/v4/rates were working as usual.
• Background jobs were not affected at all. All asynchronous processing - scheduled jobs, write backs, inventory sync, webhook deliveries, document and label generation, carrier and ERP communications - runs behind the proxy layer and continued normally throughout the incident. No queued work was lost or delayed.
• Data integrity. No data was lost, altered, or corrupted. Requests either completed normally or were rejected outright.
• Security and tenancy. No request was routed to another account, and no security boundary was crossed. The corruption occurred after all access controls were applied. The underlying Ubuntu update was a preventive security patch; the vulnerability it addressed was not exploited on our systems.

Timeline (all times PDT, August 20, 2026)

• Aug 19 (daytime) - Ubuntu publishes a security update for nginx; a defect is reported, and Ubuntu publishes a corrected package the same day. The corrected version propagates to public update mirrors overnight.
• Aug 19, 18:24 - 23:09 - The nightly update checks on the later-affected servers download the day's nginx update. At these moments, the faulty build is still the newest available on the mirrors. This step only downloads the package; installation happens during the next morning's patch window.
• Aug 20, 04:12 - 05:05 - Another group of servers runs its nightly update check after the corrected build has reached the mirrors. These servers download the fixed version and remain healthy throughout the incident.
• ~06:00 - A routine, unrelated application configuration update is applied for upcoming releases. It has no effect on currently released functionality and plays no role in the incident, but because it is the only known change that morning, it becomes the first suspect once errors appear.
• 06:00 - The nightly automated patching window begins rolling nginx updates across environments. Some servers already have the corrected package downloaded, while others have the faulty package.
• 06:31 - 06:34 - INCIDENT START. As the rolling automated patch window progresses, the previously downloaded faulty nginx package is installed on multiple web and login servers across shared production environments. Because patch schedules are staggered, not all servers update at once, and some servers remain healthy. The first customer-facing failed requests begin at 06:31.
• 06:35 - Automated external monitoring alerts on elevated errors. Investigation begins immediately.
• 06:36 - 07:15 - Engineers first investigate the ~06:00 configuration update, the only known application-level change with closely matching timing. It is ruled out, and attention turns to the web/proxy layer.
• 06:52 - The rolling patch window continues and the faulty package is activated on additional servers. Impact increases as more affected servers restart onto the faulty nginx version, while servers that downloaded Ubuntu's corrected package remain healthy.
• 06:55 - A remaining web server updates using Ubuntu's corrected package and stays healthy throughout, continuing to serve its share of traffic correctly.
• 07:18 - 07:19 - Affected web servers are restarted as a mitigation attempt. This has no effect because the faulty nginx package remains installed.
• 07:20 - 07:55 - Suspect servers are removed from load-balancer rotation. Symptoms persist because the login service and application environments are independently affected, which materially widens the search.
• 07:41 - The URL-corruption pattern is identified in application logs.
• 07:45 - 08:00 - Per-server testing isolates the faulty servers. The only difference from healthy servers is the nginx package version. The faulty build is matched to Ubuntu's published regression notice and corrected package.
• 08:02 - 08:11 - The corrected package is installed across all affected servers. Error rates return to normal immediately on each server as it restarts onto the fixed version. The final affected environment returns to normal at 08:11 - INCIDENT FULLY RESOLVED.
• 08:11+ - Full verification is completed: every server is individually tested, and API, dashboard, login, and production environments are confirmed healthy.

Why resolution took ~95 minutes from alert

Detection was fast, but three factors slowed diagnosis. First, a routine configuration change earlier that morning was the only known change in the environment and had to be ruled out - automated OS patching does not appear in any application-level change log. Second, the failure was intermittent by nature: unaffected servers continued serving normally, and even affected servers successfully handled request types whose routing rules were not impacted. Third, removing the suspect servers from rotation did not stop the errors - because other tiers were independently affected - which initially pointed the investigation away from those servers.

What we are changing

1. Stage operating-system security patches before production. Automated OS- and nginx-level security updates, including critical patches, will first be installed on non-production servers. Automated application-level validation will exercise representative API, dashboard, and login paths against the updated servers before the same package versions are allowed to roll into production. Production rollout will begin only after those checks pass.

2. Faster version-level diagnosis. Our incident runbooks now include immediate comparison of package versions and restart history across servers whenever identically-configured servers behave differently.