GZIP decompression not working correctly on web content

[gmr-prog 0]

We are a vendor of web-based software with multiple clients that use the ESET anti-virus package as a corporate standard. These clients have reported issues viewing our content in Chrome or Edge browsers. Sometimes a page will show up blank, and other times a string of nonsense characters is displayed on the screen. The developer tools revealed the presence of "ERR_INCOMPLETE_CHUNKED_ENCODING" errors when retrieving certain files, intermittently but frequently.

 

We have narrowed down the problem to a conflict between our software and ESET. We do not experience these symptoms when running on machines not equipped with ESET, and we do not experience them when running on a machine with ESET for which specifically the "HTTP(S) traffic scanning" setting has been disabled (just turning off protection doesn't seem to make an impact).

 

What appears to be happening is that we use GZIP compression on files retrieved from our server, and ESET decompresses them to scan them and delivers them to the browser uncompressed with the appropriate headers altered. However, something appears to be going wrong with the decompression, and the results being delivered to the browser are not always correct. In fact, for testing purposes we disabled compression on our server, and the software was delivered correctly with ESET running.

Below is a sampling of the response headers that the browser gets when ESET is not running vs. the response headers that the browser gets when ESET is running and the issue occurs. (Monitoring at the server side, the response headers were the same in both cases, other than timestamps.) Is there something we can add to prevent ESET from having these issues? Alternatively, is there a setting that would allow our clients to bypass that scan just for our software? We tried experimenting with the "List of addresses excluded from content scan" setting, but that didn't seem to help. If it matters, our in-house testing has been with ESET Smart Security Premium.

Response headers without ESET:

Accept-Ranges: bytes
Cache-Control: max-age=604800
Connection: keep-alive
Content-Encoding: gzip
Content-Range: bytes 0-30485/30486
Content-Type: text/html
Date: [timestamp]
Etag: [text-string]
Expires: [timestamp]
Last-Modified: [timestamp]
Referrer-Policy: strict-origin-when-cross-origin
Strict-Transport-Security: max-age=31536000; includeSubDomains
Transfer-Encoding: chunked
Vary: origin, accept-encoding
X-Content-Type-Options: nosniff
X-Robots-Tag: noindex, nofollow, nosnippet, noarchive, notranslate, noimageindex, nocache, noodp
X-Xss-Protection: 1; mode=block

Response headers with ESET:

Accept-Ranges: bytes
Cache-Control: max-age=604800
Connection: keep-alive
Content-Range: bytes 0-30485/30486
Content-Type: text/html
Date: [timestamp]
Etag: [text-string]
Expires: [timestamp]
Last-Modified: [timestamp]
Referrer-Policy: strict-origin-when-cross-origin
Strict-Transport-Security: max-age=31536000; includeSubDomains
Transfer-Encoding: chunked
Vary: origin, accept-encoding
X-Content-Encoding-Over-Network: gzip
X-Content-Type-Options: nosniff
X-Robots-Tag: noindex, nofollow, nosnippet, noarchive, notranslate, noimageindex, nocache, noodp
X-Xss-Protection: 1; mode=block

[Marcos 4,839]

Please carry on as follows:

1. Enable advanced logging under Help and support -> Technical support
2. Reproduce the issue
3. Stop logging
4. Collect logs with ESET Log Collector and upload the generated archive here.

If it's possible to reproduce the issue by ourselves, please provide step-by-step instructions as well.

[gmr-prog 0]

I generated a log file. Because there is potentially confidential info in there (and this is a public forum), and also because it is over 500 MB, is there an alternate way to submit it directly to you?

[itman 1,594]

Appears this a known issue: https://github.com/kucharzyk/spring-angular2-starter/issues/35 .

1 hour ago, gmr-prog said:

Alternatively, is there a setting that would allow our clients to bypass that scan just for our software?

Have you tried excluding your app from Eset SSL/TLS protocol scanning by setting Scan action to "Ignore"?

Note that this does put your Eset customers at risk in the event your app may contain malware.

[gmr-prog 0]

39 minutes ago, itman said:

Appears this a known issue: https://github.com/kucharzyk/spring-angular2-starter/issues/35 .

Yes, we found that Github page and the Stack Overflow page linked in it, among other pages, as part of our research. These pages pointed us in the direction of investigating ESET.

39 minutes ago, itman said:

Have you tried excluding your app from Eset SSL/TLS protocol scanning by setting Scan action to "Ignore"?

 

Note that this does put your Eset customers at risk in the event your app may contain malware.

Our software is browser-based; it wouldn't be feasible to ask our clients to exclude their web browsers from their security scans.

[Marcos 4,839]

You can upload the generated archive to a safe location and drop me a private message with a download link.

[itman 1,594]

2 hours ago, gmr-prog said:

Our software is browser-based; it wouldn't be feasible to ask our clients to exclude their web browsers from their security scans.

In the same Eset GUI section, you can exclude SSL/TLS scanning based on the certificate associated with the domain/URL of your web based app.

[gmr-prog 0]

1 hour ago, Marcos said:

You can upload the generated archive to a safe location and drop me a private message with a download link.

PM sent. Thanks!

[gmr-prog 0]

17 hours ago, itman said:

In the same Eset GUI section, you can exclude SSL/TLS scanning based on the certificate associated with the domain/URL of your web based app.

I tried that, and with that rule in place, our content does indeed work as expected.

While we will still work towards a solution that will resolve the problem completely, this is a useful option to have in case it becomes necessary. Thanks for the information!

[itman 1,594]

Here's a thread in the forum dating to 2016 on the GZip issue: https://forum.eset.com/topic/7429-does-eset-un-compress-gzip-encoded-html-pages-based-on-the-user-agent/#comment-40050 . It was never resolved. As such, I have doubts Eset will find a solution anytime in the near future.

[Marcos 4,839]

I assume that the application doesn't expect the scanned content to be gzipped but it can be already unpacked. Or is that a wrong assumption? If gzip is expected then the only solution will be creating an exception.

[gmr-prog 0]

2 minutes ago, Marcos said:

I assume that the application doesn't expect the scanned content to be gzipped but it can be already unpacked. Or is that a wrong assumption? If gzip is expected then the only solution will be creating an exception.

It is ok if the content is already unpacked by the time it reaches the browser. Our issue is that it appears the content is not always being unpacked correctly, resulting in "ERR_INCOMPLETE_CHUNKED_ENCODING" errors reported by the browser.

[itman 1,594]

10 minutes ago, Marcos said:

I assume that the application doesn't expect the scanned content to be gzipped but it can be already unpacked. Or is that a wrong assumption? If gzip is expected then the only solution will be creating an exception.

Reviewing the above posted response headers, it appears Eset is using GZip to compress the data again after analysis and that is where the issue lies. With Eset out of the picture, this header does not appear;

X-Content-Encoding-Over-Network: gzip

[gmr-prog 0]

4 minutes ago, itman said:

Reviewing the above posted response headers, it appears Eset is using GZip to compress the data again after analysis and that is where the issue lies. With Eset out of the picture, this header does not appear;

X-Content-Encoding-Over-Network: gzip

If ESET was re-compressing, I would expect the 

Content-Encoding: gzip

header to remain in the response, but it doesn't. Because it doesn't, if ESET is re-compressing, then the browser would be lost every time, trying to interpret a compressed file as uncompressed.

I can't find any reference to "X-Content-Encoding-Over-Network" online other than in conjunction with ESET. I suspect it may be a proprietary header that indicates "this was compressed with GZIP when it reached ESET, but ESET decompressed it and had to remove the Content-Encoding header, so this is the record that that happened".

[itman 1,594]

39 minutes ago, gmr-prog said:

I can't find any reference to "X-Content-Encoding-Over-Network" online other than in conjunction with ESET.

Mozilla defines it to be:

Quote

The Content-Encoding representation header lists any encodings that have been applied to the representation (message payload), and in what order. This lets the recipient know how to decode the representation in order to obtain the original payload format. Content encoding is mainly used to compress the message data without losing information about the origin media type.

Note that the original media/content type is specified in the Content-Type header, and that the Content-Encoding applies to the representation, or "coded form", of the data. If the original media is encoded in some way (e.g. a zip file) then this information would not be included in the Content-Encoding header.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Encoding

This also brings up the question of if this issues manifest when using the Firefox browser?

Edited September 15 by itman

[itman 1,594]

As far as X-Content-Encoding-Over-Network: gzip references, I have seen postings about a bug in Apache: https://forums.cpanel.net/threads/mod_deflate-check-tuning-and-test.683257/

[gmr-prog 0]

1 hour ago, itman said:

This also brings up the question of if this issues manifest when using the Firefox browser?

I have tested this, and yes, it does.

[itman 1,594]

I also checked a few web sites using GZip in Firefox that Eset is perform SSL/TLS protocol scanning. They all show the Content-Encoding: gzip header.

The mystery is why Content-Encoding-Over-Network: gzip header is being used on your web based app.

[Marcos 4,839]

We'll need a send and request captured in a HAR file. In Firefox developer tools, enable Persist logs, send a request, wait for a response and then select "Save all as HAR":

[gmr-prog 0]

Sent via PM. Thanks!