Some Customers cannot sync Sales Orders / Item Fulfillments from NetSuite

Incident Report for ShipHawk

Postmortem

Incident summary

ShipHawk NetSuite SuiteApp users ran into an issue with item fulfillments and order syncing between NetSuite and ShipHawk. When NetSuite users saved an Item Shipment record, the error is shown: TypeError: ItemFulfillment.find is not a function

This was first reported at 9:12 pm PST on Wednesday 11/17 and affected all customers that were using ShipHawk bundle with version >=2021.6.0.

The issue was caused by a change made by NetSuite in the processing of NApiVersion 2.1 scripts, which ShipHawk bundles 2021.6.0+ are using, and the incident lasted until NetSuite reverted the change at 11:30 am PST Sunday 11/21.

Leadup

On 11/17, NetSuite changed the way how they process scripts with NApiVersion 2.1 without notice, to fix a known and unrelated defect (NetSuite defect #647251). When this happened, ShipHawk SuiteApp bundles 2021.6.x and higher could no longer sync orders or item fulfillments between NetSuite and ShipHawk.

Fault

ShipHawk bundle could not load dependencies correctly; therefore, it was not able to call static functions required to work properly and caused the code to raise an exception TypeError: ItemFulfillment.find is not a function [at Object.afterSubmit (/SuiteBundles/Bundle 161164/ShipHawk (2)/event_scripts/shiphawk-update-fulfillment-event-script.js:55:35)].

Impact

Order and Item Fulfillments could not sync between NetSuite and ShipHawk.

This incident affected all NetSuite customers who using ShipHawk bundle - 2021.6.x and 2021.7.x.

A total of 12 urgent cases were submitted to ShipHawk during the impact window.

Detection

The incident was reported first time at 9:12 pm PST, Wednesday 11/17.

More reports were submitted starting from 4:21 am PST, Thursday 11/18.

Response

During this incident, ShipHawk customer success and engineering teams worked around the clock to keep impacted customers informed, identify the root cause and search for workarounds.

ShipHawk and NetSuite engineering resources worked to identify the issues and work towards a resolution. NetSuite discovered two defects (defect #651122 and #651305) which they ultimately resolved. ShipHawk identified both near-term and long-term options to mitigate this in the future, both of which would have materially delayed resolution. As such, ShipHawk Engineering decided the best path was to collaborate with NetSuite as they reverted the changes introduced on 11/17 because this was determined to be the fastest way to get joint customers operational.

Recovery

Case #4491650 was submitted to NetSuite Support, and as a result, NetSuite created 2 defects that were escalated to U2 Critical priority:

Defect 651122SuiteScript > RESTLet Script > TypeError: Class constructor CounterEntry cannot be invoked without 'new'

Defect 651305 SuiteScript > RESTLet Script > TypeError: Class constructor CounterEntry cannot be invoked without 'new'.

NetSuite ultimately reverted their changes in the processing of NApiVersion=2.1 scripts. After clearing cached files, impacted customers were able to sync orders and item fulfillments between ShipHawk and NetSuite.

Timeline

All times are PST.

Wednesday 11/17

21:00 - NetSuite introduced a change to the NApiVersion scripts processor in order to fix defect #647251
21:12 - the first time the issue that Orders and Item Fulfillments are not syncing was reported by ShipHawk customers

Thursday 11/18

4:21 - multiple customers started reporting the same issue
5:00 - the issue was verified by ShipHawk CS team and passed to the Engineering team
6:21 - the incident notification was posted to the ShipHawk Status Page
6:23 - ShipHawk Engineering identified that the issue is happening only for the customers who use the latest bundle versions and that the issue is related to changes in how NetSuite processes NApiVersion=2.x scripts
6:23 - case #4491650 was submitted to NetSuite Support
10:23 - NetSuite team notified that they reverted changes but some of it is still stuck in the server cache – there is a chance the issue might self resolve if the partner’s cache is flushed
13:02 - ShipHawk Engineering team prepared a new bundle version 2021.7.1 which intend to reset cached files
13:30 - bundle 2021.7.1 was successfully tested and then pushed to some of the customer accounts where the fix was confirmed
20:40 - the same issue was reported again

Friday 11/19

7:58 NetSuite team notified us about a critical defect 651122: SuiteScript > RESTLet Script > TypeError: Class constructor CounterEntry cannot be invoked without 'new'
15:17 Defect 651122 was reported as fixed and deployed to all server
22:30 ShipHawk team verified that the fix doesn't work even after a cache refreshed
22:31 NetSuite Support Case #4491650 re-opened
15:23 A new critical defect 651305 was created in NetSuite: SuiteScript > RESTLet Script > TypeError: Class constructor CounterEntry cannot be invoked without 'new'

Saturday 11/20

23:56 NetSuite pushed fixes for ShipHawk testing accounts

Sunday 11/21

10:30 NetSuite team confirmed that the fix is pushed to all accounts
11:02 ShipHawk prepared the new bundle version 2021.7.2 which intend to reset cached files
11:09 ShipHawk team verified the fix working and CS team helped customers to install the new bundle

Root cause identification:

The customers were not able to sync orders and Item Fulfillments from NetSuite ShipHawk
Because NetSuite changed how they process scripts with NApiVersion 2.1
Because it was deployed by NetSuite without notice to find and resolve such issues without impacting customers
Because some ShipHawk scripts with Public scope return classes when they should use Same Account scope instead
Because ShipHawk does not have an alternative order and/or item fulfillments sync process for customers using the SuiteApp

Root cause

The instability we saw in customer accounts was introduced because some of our SuiteApp scripts with Public scope return classes. After the incident was resolved, NetSuite advised us that this is only supported for Same Account scope. Had we used this alternative scope, it may have mitigated the issue.

Lessons learned

NetSuite may deploy changes to SuiteApp Developer tools without notice
NetSuite recommends we change scope of scripts in the bundle from Public to Same Account
ShipHawk needs to investigate alternative integration methods
ShipHawk needs to explore manual workarounds in the event an integration encounters an unplanned breaking change
ShipHawk needs to explore alternative syncing strategies to further mitigate risk

Corrective actions

We will prioritize effort to change scope of scripts in the bundle from Public to Same Account per NetSuite’s recommendation
We will explore redundant and alternative syncing strategies to reduce reliance on changes made by integration partners

Posted Nov 24, 2021 - 08:55 PST

Resolved

This incident is resolved.

If you were impacted by this issue, update to ShipHawk bundle latest version 2022.7.2 to refresh cached files in NetSuite.

Once you have done so, you can update the saved search "Recent Un-synced Orders" to import any un-synced orders from NetSuite into ShipHawk. To do this in NetSuite:

1. Enter "Recent Un-synced Orders" in the Search bar at the top of the page.

2. Click on the search result returned.

3. Click "Edit this search".

4. In the Criteria tab, click on the row labeled "Date Created is after yesterday".

5. Click the icon that appears next to "Date Created" that prompts a new window to open.

6. Update the criteria so that the Date Created criteria will look for orders back to the first date you were impacted.

7. Click "Save" to update the search.

8. Wait 15-25 minutes for new orders to be imported.

9. Confirm missing orders are imported into ShipHawk.

Please contact support@shiphawk.com if you have additional questions or concerns.

A post-mortem will be provided and accessible on this status page within the next 3-5 business days.

Posted Nov 21, 2021 - 14:59 PST

Monitoring

NetSuite has released their fix for Defect 651305: SuiteScript > RESTLet Script > TypeError: Class constructor CounterEntry cannot be invoked with ‘new’ which is impacting ShipHawk and other SDN Developers.

In order to renew the cache, we deployed a new version of ShipHawk bundle - 2021.7.2

We recommend updating the ShipHawk bundle to the latest version 2022.7.2, and expanding the saved search criteria so that un-synced orders (if any) will be imported in ShipHawk from NetSuite.

Posted Nov 21, 2021 - 07:47 PST

Update

There are no current updates at this time.

Our engineering team continues to investigate, monitor, and also work toward a workaround that is not dependent on NetSuite's fix.

As previously noted, NetSuite has identified another defect which they have also deemed U2, a critical defect.

Defect 651305: SuiteScript > RESTLet Script > TypeError: Class constructor CounterEntry cannot be invoked with ‘new’ which is impacting ShipHawk and other SDN Developers.

NetSuite is working diligently to address this critical issue.

Additionally, ShipHawk engineering continues to investigate possible workarounds that do not depend on NetSuite’s resolution of these critical defects.

We will continue to share information until this is fully resolved.

Posted Nov 20, 2021 - 17:34 PST

Update

We continue to investigate, monitor, and also work toward a workaround that is not dependent on NetSuite.

As previously noted, NetSuite has identified another defect which they have also deemed U2, a critical defect.

Defect 651305: SuiteScript > RESTLet Script > TypeError: Class constructor CounterEntry cannot be invoked with ‘new’ which is impacting ShipHawk and other SDN Developers.

NetSuite is working diligently to address this critical issue.

Additionally, ShipHawk engineering continues to investigate possible workarounds that do not depend on NetSuite’s resolution of these critical defects.

We will continue to share information until this is fully resolved.

Posted Nov 20, 2021 - 09:33 PST

Update

We continue to triage this escalation with NetSuite.

NetSuite has identified another new defect 651305: SuiteScript > RESTLet Script > TypeError: Class constructor CounterEntry cannot be invoked with ‘new’ which is impacting ShipHawk and other SDN Developers.

The urgency level of this defect has also been escalated to U2, identified as a critical defect. NetSuite’s management and engineering teams are currently investigating and working on a resolution, with a range for solution within 1-2 days.

ShipHawk’s engineering team deems this urgent and as our greatest priority. We continue to closely work with NetSuite on a resolution and will continue to update you as we know more.

Posted Nov 20, 2021 - 07:06 PST

Update

We continue to monitor the issue and are working diligently with NetSuite technical support to resolve this issue as quickly as possible. We will continue to provide updates until this issue is fully resolved.

Posted Nov 19, 2021 - 16:50 PST

Identified

We continue to triage this escalation with NetSuite.

NetSuite has acknowledged Defect 651122: SuiteScript > RESTLetScript > TypeError: Class constructor CounterEntry cannot be invoked without 'new' which is impacting ShipHawk and other SDN Developers.

NetSuite has escalated the issue to "U2" and understands this is a critical defect with U2 priority, NetSuite's target resolution period is 1-2 days. That said, the resources we are working with are telling us they are confident it will be resolved today.

Within the next week, we will provide you with a full post-mortem, including our plans to further mitigate unplanned changes made by NetSuite and/or dependencies between ShipHawk and NetSuite.

If you are experiencing this issue, you can directly reach out to NS and reference NS support case #4491650, Defect 651122

Posted Nov 19, 2021 - 10:01 PST

Update

Posted Nov 19, 2021 - 09:59 PST

Update

Posted Nov 19, 2021 - 09:41 PST

Update

We are continuing to investigate this issue. We have escalated to NetSuite and are working with them for resolution. Our next update will be within 60 minutes.

Posted Nov 19, 2021 - 08:12 PST

Update

We are continuing to investigate this issue. We have escalated to NetSuite and are working with them for resolution. Our next update will be within 30 minutes.

Posted Nov 19, 2021 - 07:34 PST

Investigating

NetSuite is experiencing an issue impacting multiple partners and current NetSuite users. We are working with their team on an immediate resolution. Our next update will be within 30 minutes.

Posted Nov 19, 2021 - 07:04 PST

Identified

NetSuite has rolled back a change they made last night that caused issues for SuiteApp Developers who use NApiVersion 2.x. According to NetSuite technical support, their changes are reverted now, but some files in NetSuite may still be cached.

If you are still experiencing issues, it may be due to file caching.

We will continue to discuss this issue with NetSuite to learn what mitigations can be put in place to avoid an issue like this in the future and will provide our findings in a post-mortem.

Posted Nov 18, 2021 - 14:15 PST

Update

At this time, our engineering team is still investigating this issue.

Posted Nov 18, 2021 - 11:45 PST

Investigating

NetSuite users are facing issues with item fulfillments and order sync in ShipHawk. When NetSuite users save an Item Shipment record, the error is shown:

TypeError: ItemFulfillment.find is not a function [at Object.afterSubmit (/SuiteBundles/Bundle 161164/ShipHawk (2)/event_scripts/shiphawk-update-fulfillment-event-script.js:55:35)]

This was first reported at 9:12 PM PDT. We are currently investigating and have reached out to NetSuite support. No changes that could cause the issue were made by ShipHawk.

Posted Nov 18, 2021 - 06:21 PST

This incident affected: System Connectors (Oracle NetSuite SuiteApp) and ShipHawk Application.