ESET SSL protection produces an invalid certificate chain for NodeJS apps

[Guest Cohee]

First of all, I'm not an ESET user or customer. I'm a maintainer of an open-source NodeJS application called SillyTavern.

Yesterday (April 17, 2024) around 16:00 UTC I started receiving reports from users that they couldn't connect to secure hosts through the app, and a GitHub issue was raised. The source of the issue comes from the fact that the users have an incomplete certificate chain on their system, and unlike web browsers that use TLS AIA extension or use their own CA bundle, Node does not automatically complete the certificate chain, which results in an OpenSSL error like the follows (full error log in a linked issue): 

Quote

FetchError: request to https://api.cohere.ai/v1/models failed, reason: unable to verify the first certificate

Accessing the affected hosts (an example list below) through a web browser does not result in an error due to the reasons I stated above:

• https://api.cohere.ai/
• https://api.novelai.net
• https://openrouter.ai/

After some investigations with the author of GitHub issues, it was found that it only affects users who are using ESET antivirus products. There are more than 5 affected users so far who reported this either on GitHub or in the Discord community.

• ESET NOD32 17.1.9.0 on Windows 10 (unknown winver)
• ESET Smart Security 17.0.16.0 on Windows 10 22H2

It is important to clarify, that it is not a bug of the specific application as it affects all outgoing connections from other apps that do not implement the TLS AIA extension, here's a minimal reproducible example of a script that results in the same error for the affected users: https://gist.github.com/Cohee1207/10774c05392e1f15f0f5ceac2559e106

The users were also asked to run the following command to connect to an affected host directly by using the OpenSSL client, which resulted in a failed SSL handshake (Verification error: unable to get local issuer certificate): 

Quote

openssl s_client -connect api.novelai.net:443

The full output can be found here: https://gist.github.com/Cohee1207/0cf589be525a64e928e3e7260232f8d4

Also, the users confirmed that running Windows in Safe mode, removing ESET products from their system, or adding Node to a list of exceptions for SSL protection seemed to have fixed their issue.

I'm looking forward to your reply.

[Marcos 5,080]

Please raise a support ticket so that the issue can be reproduced and logs relayed to the developers for perusal.

[Guest Upset ESET Customer]

I have confirmed the same behavior. As a customer of ESET, all of our dev machines were having "unable to verify the first certificate" with nodejs. Removing ESET SSL protection fixed the issue. This only occurred this morning.

[Guest Same problem here]

We need a fix for this.

[Marcos 5,080]

Please raise a support ticket so that the issue can be reproduced and logs relayed to the developers for perusal.

[itman 1,662]

11 hours ago, Guest Cohee said:

Accessing the affected hosts (an example list below) through a web browser does not result in an error due to the reasons I stated above:

• https://api.cohere.ai/
• https://api.novelai.net
• https://openrouter.ai/

Only URL I could access w/o issue using latest ver. of Firefox was for openrouter.ai. The other two URLs resulted in;

[Guest guest7]

You need to access HTTPS via Node/NPM which already typically fails validation whenever an intermediate cert is not provided. Modern browsers rarely surface chain issues.

It's very easy to replicate using any node JS application, application error responses happen post SSL negotiation so don't matter.

It's a shame this forum exists yet a useful post with all relevant info is ignored until a support ticket is raised. This issue caused a huge headache, and OP posting this was a huge help in confirming it wasn't an isolated issue.

 

 

 

[Guest Julien Bras]

It's easily reproductible here with this as mentioned by the OP:

 https://gist.github.com/Cohee1207/10774c05392e1f15f0f5ceac2559e106

It causing headaches here in my company. I have already open a case (#YWH-412-64908 on https://customers.eset-nod32.fr )but no answer yet.

Thanks for the action on that.

[Guest kevin morizur]

I confirm the need for a fix we have more than 3000 machines running with ESET NOD32 17.1.9.0 on Windows 10 and that are facing similar issue when accessing https service from nodejs application.

Eset is adding its own certificate in the chain which is not valid for nodejs.

We can see that ESET certifcate has been updated on april 17th.

An "easy" fix if you have only few machines is to deactivate ssl option for configuration.

We already raised ticket yesterday morning without any answer...

[Marcos 5,080]

The issue has been reported to developers.

P_EESW-11455

[Guest Siemer]

Many hours of work lost thanks to this bug. For reference, here are a few common error messages:

npm ERR! code UNABLE_TO_VERIFY_LEAF_SIGNATURE
npm ERR! request to https://registry.npmjs.org/... failed, reason: unable to verify the first certificate

Error downloading and extracting template package: Error: npm exited with non-zero code: 1

In old messages an often recommended solution is to run 

npm config set strict-ssl false

which just disables SSL all together and is obviously a bad idea.

The current solution is to disable HTTPS traffic scanning in the advanced settings. This should be fixed asap, many developers depend on node.

[Marcos 5,080]

It is not actually a bug, we merely started to scan nodejs communication in order to detect nodejs malware payload.

An easy solution is to tell nodejs to use the system CA cert.. store by running:

node --use-openssl-ca

https://stackoverflow.com/questions/29283040/how-to-add-custom-certificate-authority-ca-to-nodejs

Alternatively (less safe) you can add nodejs executable to the list of applications excluded from TLS filtering.

[Guest Rob Lang @MCO]

Quote

It's not actually a bug

No, it's much worse. You willfully pushed out a change without warning that brings down line of business systems. That's negligence. I get the sense that you don't fully understand how serious this is.

I have raised a ticket and will be pursuing.

[Guest Roman]

Hi Eset, I've been a customer of yours for several years and I'm having serious problems with this new "feature". I can't develop anything and my work is stuck. Your suggestion does not work

node --use-openssl-ca .\dist\apps\api\main.js

I get an error:

TypeError: fetch failed
    at node:internal/deps/undici/undici:12500:13
    for process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  [Cause] Failure: Could not verify the first certificate
      on TLSSocket.onConnectSecure (node:_tls_wrap:1674:34)
      on TLSSocket.emit (node:events:519:28)
      on TLSSocket.emit (node:domain:488:12)
      at TLSSocket._finishInit (node:_tls_wrap:1085:8)
      at ssl.onhandshakeone (node:_tls_wrap:871:12)
      on TLSWrap.callbackTrampoline (node:internal/async_hooks:130:17) { code
    code: 'UNABLE_TO_VERIFY_LEAF_SIGNATURE' }
  }
}

Is it possible to disable this behaviour or do I need to uninstall Eset?

[Guest Rob Lang]

  

Quote

An easy solution is to tell nodejs to use the system CA cert.. store by running:

node --use-openssl-ca

This solution not possible when deploying to a server using GitHub Action Runner because GitHub Actions does not let you specify command line arguments for security reaons.

Furthermore, adding in TLS exception is unwise because the location of the node exe can change. Right now, my server using GitHub Action Runner and ESET Server Security 11 has the following installations of node:

C:\actions-runner\externals.2.315.0\node16\bin\node.exe
C:\actions-runner\externals.2.315.0\node20\bin\node.exe
C:\actions-runner\externals.2.314.1\node16\bin\node.exe
C:\actions-runner\externals.2.314.1\node20\bin\node.exe

In a few weeks when the GHA runner is updated, there will be a new folder and the TLS exception won't include it and my deployment will fail.

I need to add root cause anaylsis into our ISO27001 Information Management Systems. What was the root cause of your release without warning that directly affects business line systems? Can we expect a public blog article explaining the change, how it was signed off and what mitigations will be performed in the future? What assurance can you give that you won't release another security update without warning that might grossly affect a business line system?

I've raised this on the UK support ticket (CASE_00747921) but have been told that "Head Office" is the right place and that is here.

[itman 1,662]

40 minutes ago, Guest Rob Lang said:

I've raised this on the UK support ticket (CASE_00747921) but have been told that "Head Office" is the right place and that is here.

No. The Eset forum is not the proper place to pursue this matter. Eset headquarters info shown below;

[Marcos 5,080]

It is also possible to create exclusions from filtering by IP addresses and certificates as a workaround.

[Guest Josh]

This is also affecting all of our builds for our test and production systems. 
 

 

[Guest Pachomar]

Lost a full day of work thanks to this "feature". As a paying costumer, I hope you address this issue ASAP

[Guest Tim]

Any update on this? Is there a fix for this being worked on? Unable to continue development work without disabling HTTPS, which is obviously an incredibly bad idea!

[Guest Anon]

I have formatted my computer and have been looking for the problem for 3 days. And the source of the problem is ESET, I need this fixed as soon as possible.

[Guest Siemer]

No need to disable https, just disable this incredibly shortsighted feature in the advanced settings.

Marcos, are you serious? It's intended behavior to BREAK NODE (and some functionality of React Native Expo) and it's up to us to switch to system CA cert. store?

I hope you are doing everything you can to push an update that reverts this nonsense, or I will be pushing for a different security provider that actually respects their clients. 

[Guest _JV]

Thanks to this thread I was able to provide a workaround to my customers who opened a support ticket. I'm personally an ESET customer and I'm disappointed that an update that breaks all Node applications got rolled out, was identified as faulty here, and several days later, the update has not been rolled back. This is a high severity incident and should be handled as such.

[itman 1,662]

3 hours ago, Guest Siemer said:

No need to disable https, just disable this incredibly shortsighted feature in the advanced settings.

Are you referring to disabling Eset Browser Script Scanner?

[Guest Siemer]

This setting:

(image from this SO thread: https://stackoverflow.com/questions/78343601/node-unable-to-verify-the-first-certificate)