NetSuite Integration Incident Response: A Runbook for When Your WooCommerce Sync Goes Dark

Every NetSuite-WooCommerce integration eventually goes dark — a NetSuite maintenance window overruns, an API token expires, a webhook endpoint starts silently 500ing. The difference between a 20-minute blip nobody notices and a multi-day mess with duplicate orders and furious customers is almost never the underlying cause. It’s whether you had a runbook, or whether everyone improvised, as covered in more depth in the NetSuite–WooCommerce integration guide.

Step One: Stop the Bleeding, Not Root Cause

The instinct when a sync goes dark is to start debugging immediately. Resist it. The first move in any integration incident is to protect data integrity while the store keeps taking orders — root cause analysis comes after the immediate risk is contained.

Why NetSuite Syncs Actually Go Dark

“The sync is down” has a small set of actual causes, and only half of them are visible before they happen. Knowing which one you’re looking at changes the first move.

Verdict: certificate expiry and maintenance windows are calendar problems — a reminder 30 days out fixes both. Webhook failures and concurrency exhaustion are monitoring problems — catching them before a customer does is what proper observability is for, not what a runbook alone can cover.

Assign the Four Roles Before You Need Them

Without these roles assigned ahead of time, the person who happens to notice the outage first ends up doing all four jobs badly instead of one job well — usually the technical diagnosis, while customer communication and duplicate-prevention get forgotten until it’s too late. Google’s own SRE incident doctrine, which this role split is adapted from, states the reason plainly: “it’s important to make sure that everybody involved in the incident knows their role and doesn’t stray onto someone else’s turf.” A technical responder who also decides customer messaging is doing that — straying onto the incident lead’s turf while the actual fix waits.

The 15-Minute Customer Decision

Within 15 minutes of confirming an outage, someone needs to decide: does this warrant a customer-facing notice (a banner, an email, a status page update), or is it invisible enough to customers that no communication is needed? Waiting until the outage resolves to make this call means you either send a “sorry for the inconvenience” email about a problem that’s already fixed and confusing, or — worse — customers discover the problem themselves via a support ticket, which erodes trust faster than the outage itself did.

The Full Incident Timeline, End to End

The sections above cover each decision point on its own. Laid out on one timeline, the sequence is: detect and queue, assign roles in parallel, make the 15-minute customer call, wait for reconnection, replay throttled rather than all at once, then dedupe before calling it resolved. The diagram below is the whole runbook as one sequence — the prose sections give the reasoning behind each step, the diagram gives the order.

Replay the Queue Without Triggering a Second Outage

The queue-don’t-drop architecture solves the first outage. It creates a second failure mode if the replay itself ignores NetSuite’s concurrency governance: draining a few hours of queued writes as fast as the client can fire them is exactly the burst pattern that exhausts the account’s concurrent-request pool and produces a self-inflicted second incident, right as the team is declaring victory on the first one.

Verdict: the formula is base limit plus 10 per SuiteCloud Plus license, and it covers REST web services and RESTlet requests as one shared account-wide pool — a replay script that doesn’t know its own tier’s number is guessing, and guessing wrong during a replay burst just starts a second outage.

NetSuite’s REST web services expose a governanceLimits operation that returns accountConcurrencyLimit and accountUnallocatedConcurrencyLimit for the account, so a replay job can check real headroom instead of hard-coding a guess. The catch: the operation must be called by an administrator-privileged token, which sits at odds with running your sync service account on the least-privilege role netsuite.md-style OAuth 2.0 M2M doctrine recommends. The practical resolution most integrations land on: hard-code the throttle to the account’s known tier limit (from Setup > Integration > Integration Governance) for the production replay job, and reserve a live governanceLimits call for an admin-run diagnostic script, not the automated replay path itself.

The Replay Queue’s Real Job Is Idempotent Writes

A replay queue that fires each queued write exactly once, in order, with nothing else changing, is the easy case. Production outages rarely cooperate: a create call can time out after NetSuite has already committed the record server-side, leaving the client with an unknown outcome — not a clean failure it can safely retry blind.

The fix is at the write, not the replay: put the source platform’s order ID in NetSuite’s externalId field on every Sales Order create, before the first outage, not as a fix after one. A retried create for an order that already landed then errors on the uniqueness constraint instead of silently duplicating it — the retry gets caught mechanically, instead of relying on a human to notice. After reconnection, the data integrity checker’s dedup scan is a SuiteQL query over recently created Sales Orders grouped by externalId (falling back to PO number where externalId wasn’t populated), flagging any value that appears more than once. This is the same dedup pattern that prevents duplicate orders in normal operation — an outage just makes it load-bearing instead of a nice-to-have.

After Reconnection: Check Duplicates and Your Webhook Before Anything Else

The moment NetSuite connectivity is restored and the replay queue starts draining, the data integrity checker’s job is to verify no order was processed twice — once during a partial reconnection attempt, and again during the full replay, per the externalId scan above. This check happens before anyone declares the incident resolved. A duplicate order or duplicate invoice discovered a week later, during a customer complaint about a second charge, is a far worse outcome than an extra 20 minutes of verification immediately after reconnection.

There’s a second check the same role should run, and it’s easy to miss because it’s silent: WooCommerce automatically disables a webhook after more than five consecutive delivery failures, where a failure is any response that isn’t a 2xx, 301, or 302. If the outage was on the WooCommerce-facing side — the sync endpoint down, timing out, or 500ing while NetSuite kept sending updates — the webhook may have auto-disabled itself partway through the incident. Fixing the underlying cause does not re-enable it; that’s a manual step. A team that declares the incident resolved without checking webhook status can spend the next several days quietly missing order updates, with no error anywhere to flag it.

Write the Postmortem While It’s Fresh

If any part of this — the queue-don’t-drop architecture, the four roles, the communication templates, the concurrency-aware replay — doesn’t exist yet, that’s the actual action item from this post. But even a fully-built runbook isn’t the last step: Google’s SRE guidance is direct about why the incident record matters after the fix ships — “retain this documentation for postmortem analysis.” Reconstruct the timeline while it’s still accurate: when the outage was first detected, when each of the four roles activated, when the 15-minute customer call was made, when the queue was declared fully drained, when the dedup scan and webhook check completed.

Keep the postmortem blameless in the literal sense: the write-up documents which process step was missing or slow, not who was slow to notice. “The customer-comms decision took 40 minutes because the incident lead role wasn’t assigned yet” is a process finding with an owner and a fix. “Someone should have caught this sooner” is not — it produces no action item and just makes the next incident’s team more likely to hide problems instead of surfacing them. Every postmortem ends with concrete, owned, dated action items, or it wasn’t worth writing.

The Pre-Incident Checklist

Everything above, condensed into what should already be true before the next outage. Most of it should be verified in a sandbox first, per the staging checklist, not discovered live in production.

• Confirm the queue-don’t-drop-don’t-block architecture in code, not just in a design document
• Name the four incident roles by person, not by “whoever’s online”
• Pre-write customer-notice templates for both the “invisible” and “visible” outage cases
• Verify externalId (or an equivalent) is written on every order create, tested against a real duplicate-retry, not just reviewed in code
• Hard-code the replay throttle to your account’s known concurrency tier from Integration Governance
• Put the OAuth 2.0 M2M certificate or token expiry date on a calendar with a 30-day-ahead reminder
• Alert on webhook delivery failures before WooCommerce’s own five-failure auto-disable fires, and check webhook status explicitly during every reconnection
• Schedule the dedup scan to run automatically after any detected reconnection, not as a manual step someone has to remember

Running this list against a live integration rather than a hypothetical one is what the ecommerce sync audit service is for — a structured review of queue behavior, dedupe coverage, and replay throttling against your actual account tier and traffic, before the next outage finds the gaps for you.

Sources & Further Reading

1. Oracle NetSuite — Concurrency Governance Limits Based on Service Tiers and SuiteCloud Plus LicensesConfirms the base-plus-10-per-license formula and worked examples by service tier.
2. Oracle NetSuite — GovernanceLimits OperationDefines the REST operation that returns live concurrency headroom, and its administrator-token requirement.
3. WooCommerce.com — Webhooks DocumentationSource for the five-consecutive-failure auto-disable behavior and its non-2xx/301/302 failure definition.
4. Google SRE Book: Managing IncidentsGoogle SRE — incident role clarity and postmortem-documentation guidance this runbook is adapted from.
5. Oracle NetSuite DocumentationGeneral reference for API availability, rate limiting, and maintenance windows.

Frequently asked questions

Related guides

• Multi-Currency NetSuite + WooCommerce: The Three Problems Every Integration Hits
• Observability for NetSuite Integrations: What to Monitor, What to Alert On, and What to Ignore
• WooCommerce Order Routing with NetSuite: The Column That Turns Off Auto-Assignment