Is apache proxy only proxying or caching as well?

[carmik 0]

Hello,

I had this question a long time ago, while I was trying to optimize our ERA 6.5 setup. At that time, I installed the (ESET provided) Apache proxy on my ERA VA and presumed that it actually cached information. My question is: does it actually cache or does it merely proxy traffic (ie to ease communication of clients to ESET servers)? If it's the latter, was it a design decision?

If it did, I'd expect to see HTTP responses different than 200 in the access log. IIRC, on my squid proxy I'd expect to see 301 responses (not modified).

Edited March 19, 2019 by carmik

[MichalJ 434]

It´s meant to be also caching the data for both installations (installers) and updates (modules) 

[MartinK 378]

5 hours ago, carmik said:

If it did, I'd expect to see HTTP responses different than 200 in the access log. IIRC, on my squid proxy I'd expect to see 301 responses (not modified).

I think response 301 would be visible in case clients were using HTTP headers like "If-Match" or "If-Modified-Since" which is not case. Also be aware there are resources (especially update modules for endpoints) are configured in way that re-validation is required. This means that even when file is cached by proxy, HEAD call is made to ESET servers just to verify there has been no change in file. Also endpoint will be performing HEAD request with "no-cache" headers to verify authentication requirements -> those requests are automatically forwarded to ESET infrastructure.

[carmik 0]

I hoped I would be able to get some indication of cache hits to acknowledge that there is a cache and it does operate properly. Running a du -sk on /var/cache/httpd/proxy shows that it's around 1Gb.

A couple of questions:

1) How is cache maintained? Are old/stale entries purged in order to avoid cache reaching an absurdly high size? There's a CacheMaxExpire directive set to a week (IIRC), but I'm not sure whether stale entries are automatically purged to make space for newer ones.

2) CacheDefaultExpire is set to an hour. If my memory serves correctly, this just means that an entry with no expiration info will become sstale after one hour. But that does not mean that it will be purged also. Immediate purging would effectively destroy any decent cache hit ratio. Is that correct? That is, do large downloads still stay?

3) If it is not automatically maintained, should I do something?

4) I've noticed that max file size for caching (CacheMaxFileSize) 200000000, or 200Mb? Since a lot of installation packages are over that amount, can you please consider increasing this amount in future ESET Apache setups?

5) Finally, I've had a pre-ESMC (ERA) installation, with a remote point on which your Apache was installed as a proxy. ESMC seems to bring HTTP2 and other goodies. After upgrading to it, I did not change the remote apache installation though. Should I change something there?

Apologies for the long number of questions.

[MartinK 378]

12 hours ago, carmik said:

1) How is cache maintained? Are old/stale entries purged in order to avoid cache reaching an absurdly high size? There's a CacheMaxExpire directive set to a week (IIRC), but I'm not sure whether stale entries are automatically purged to make space for newer ones.

We are using standard apache utility named htcacheclean. In latest appliances it is started as a system service, regularly and it is using default configuration for now, which means cache should not be larger than 10GB and should not contain more than 12000 files after cleanup execution.

12 hours ago, carmik said:

2) CacheDefaultExpire is set to an hour. If my memory serves correctly, this just means that an entry with no expiration info will become sstale after one hour. But that does not mean that it will be purged also. Immediate purging would effectively destroy any decent cache hit ratio. Is that correct? That is, do large downloads still stay?

It is a precaution to not cache something unexpected for too long. ESET services that are expected to provide cached content are using specific values, ranging from 10sec to days, depending on type of provided data.

12 hours ago, carmik said:

4) I've noticed that max file size for caching (CacheMaxFileSize) 200000000, or 200Mb? Since a lot of installation packages are over that amount, can you please consider increasing this amount in future ESET Apache setups?

We do plan to increase this value in the future but I do not recall any ESET installer to be larger than 200MB, could you provide more details?

12 hours ago, carmik said:

5) Finally, I've had a pre-ESMC (ERA) installation, with a remote point on which your Apache was installed as a proxy. ESMC seems to bring HTTP2 and other goodies. After upgrading to it, I did not change the remote apache installation though. Should I change something there?

ESMC is using TLS+HTTP for Agent-to-Serve communication, which can be forwarded through proxy. In order for this to work, only change is that target hostname and target port has to be explicitly enabled in proxy configuration. Also changes are required for EDTD to work properly in case you are using it (it is extra paid service so you would know).

 

In case you are interrested in diagnostic of caching performance, I would recommend to extend configuration with logging of cache results. For this purpose, official apache documentation recommend somethinkg like this:

CustomLog "cached-requests.log" common env=cache-hit
CustomLog "uncached-requests.log" common env=cache-miss
CustomLog "revalidated-requests.log" common env=cache-revalidate
CustomLog "invalidated-requests.log" common env=cache-invalidate

Please see relevant apache documentation for more details and also be aware that most ESET resources will be in "revalidated" category, not in "cached" as most of the requires are configured in a way that revalidation of data is required (i.e. fast check will be made whether cached data has not changed in the meantime).

[carmik 0]

11 minutes ago, MartinK said:

We are using standard apache utility named htcacheclean. In latest appliances it is started as a system service, regularly and it is using default configuration for now, which means cache should not be larger than 10GB and should not contain more than 12000 files after cleanup execution.

First let me thank you for a very informative post!

I've used htcacheclean in the past. Is it always running as a daemon or regularly as a cron job on the VA? If it is the former, running ps aux |grep htcacheclean  did not produce any results.

It is a precaution to not cache something unexpected for too long. ESET services that are expected to provide cached content are using specific values, ranging from 10sec to days, depending on type of provided data.

Presumed so, thanks.

We do plan to increase this value in the future but I do not recall any ESET installer to be larger than 200MB, could you provide more details?

IIRC, I saw *msi downloads over HTTP around 220-240Mb in size. Not sure though.

ESMC is using TLS+HTTP for Agent-to-Serve communication, which can be forwarded through proxy.

A clarification please. In Agent-to-Server above, by "Server" I presume you are not referring to ESMC itself, but to the ESET server infrastructure for updates etc

In case you are interrested in diagnostic of caching performance, I would recommend to extend configuration with logging of cache results. For this purpose, official apache documentation recommend somethinkg like this:

CustomLog "cached-requests.log" common env=cache-hit
CustomLog "uncached-requests.log" common env=cache-miss
CustomLog "revalidated-requests.log" common env=cache-revalidate
CustomLog "invalidated-requests.log" common env=cache-invalidate

Please see relevant apache documentation for more details and also be aware that most ESET resources will be in "revalidated" category, not in "cached" as most of the requires are configured in a way that revalidation of data is required (i.e. fast check will be made whether cached data has not changed in the meantime).

Thanks, I should have thought about that earlier

[MartinK 378]

10 minutes ago, carmik said:

A clarification please. In Agent-to-Server above, by "Server" I presume you are not referring to ESMC itself, but to the ESET server infrastructure for updates etc

Actually I meant Agent-to-ESMC, which is using TLS+HTTP2. It is now possible to route this connection through proxies and that is why our own ERA Proxy was discontinued. But even it is forwarded through proxy, there is no caching possible -> encrypted connections are just forwarded so it can be used as a network router for our communication, even when it is is actually internal, i.e. it does not leaves company network.

[carmik 0]

Is this Agent-to-ESMC the traffic that goes to ports 2222-2223?

[carmik 0]

And another question, I frequently observe HTTP requests of the following form:

X.Y.Z.W - - [21/Mar/2019:12:34:45 +0100] "HEAD hxxp://update.eset.com/eset_upd/ep6.6/dll/update.ver HTTP/1.1" 401 - "-" "EES Update (Windows; U; 32bit; BPC 6.6.2089.2; OS: 10.0.16299 SP 0.0 NT; TDB 40782; CL 1.0.0; x32c; APP ees; ASP 0.0; FW 32.0; PX 1; PUA 0; CD 1; RA 1; PEV 0; UNS 0; UBR 125; HVCI 0; HWF: XXXXXXXXXXXXXXXXXXXXXXXXXXXX; PLOC el_gr; PCODE 111.0.0; PAR 1; ATH -1; DC 0; PLID XXX-XXX-XXX; SEAT ABCDEFG; RET 5003)"
X.Y.Z.W - - [21/Mar/2019:12:34:46 +0100] "GET hxxp://update.eset.com/eset_upd/ep6.6/dll/update.ver HTTP/1.1" 200 11065 "-" "EES Update (Windows; U; 32bit; BPC 6.6.2089.2; OS: 10.0.16299 SP 0.0 NT; TDB 40782; CL 1.0.0; x32c; APP ees; ASP 0.0; FW 32.0; PX 1; PUA 0; CD 1; RA 1; PEV 0; UNS 0; UBR 125; HVCI 0; HWF: XXXXXXXXXXXXXXXXXXXXXXXXXXXX; PLOC el_gr; PCODE 111.0.0; PAR 1; ATH -1; DC 0; PLID XXX-XXX-XXX; SEAT ABCDEFG; RET 5003)"
	

As you can see, the first request gets a 401 response code, whereas the GET one gets a document ok (200). Is the 401 HEAD response to be expected/normal?

 

Edited March 21, 2019 by carmik

[Marcos 5,074]

401 is received first because authentication is not performed. Subsequently we use digest authentication to authenticate against the update server which succeeds.

[carmik 0]

Thank you for addressing all my questions and concerns!

[MartinK 378]

10 hours ago, carmik said:

Is this Agent-to-ESMC the traffic that goes to ports 2222-2223?

Agent-to-ESMC goes by default to port 2222. Port 2223 is used for Console and API connections (also used when using "Server assisted" installation).

[MartinK 378]

4 hours ago, carmik said:

And another question, I frequently observe HTTP requests of the following form:

X.Y.Z.W - - [21/Mar/2019:12:34:45 +0100] "HEAD hxxp://update.eset.com/eset_upd/ep6.6/dll/update.ver HTTP/1.1" 401 - "-" "EES Update (Windows; U; 32bit; BPC 6.6.2089.2; OS: 10.0.16299 SP 0.0 NT; TDB 40782; CL 1.0.0; x32c; APP ees; ASP 0.0; FW 32.0; PX 1; PUA 0; CD 1; RA 1; PEV 0; UNS 0; UBR 125; HVCI 0; HWF: XXXXXXXXXXXXXXXXXXXXXXXXXXXX; PLOC el_gr; PCODE 111.0.0; PAR 1; ATH -1; DC 0; PLID XXX-XXX-XXX; SEAT ABCDEFG; RET 5003)"
X.Y.Z.W - - [21/Mar/2019:12:34:46 +0100] "GET hxxp://update.eset.com/eset_upd/ep6.6/dll/update.ver HTTP/1.1" 200 11065 "-" "EES Update (Windows; U; 32bit; BPC 6.6.2089.2; OS: 10.0.16299 SP 0.0 NT; TDB 40782; CL 1.0.0; x32c; APP ees; ASP 0.0; FW 32.0; PX 1; PUA 0; CD 1; RA 1; PEV 0; UNS 0; UBR 125; HVCI 0; HWF: XXXXXXXXXXXXXXXXXXXXXXXXXXXX; PLOC el_gr; PCODE 111.0.0; PAR 1; ATH -1; DC 0; PLID XXX-XXX-XXX; SEAT ABCDEFG; RET 5003)"
	

As you can see, the first request gets a 401 response code, whereas the GET one gets a document ok (200). Is the 401 HEAD response to be expected/normal?

 

As @Marcos mentioned, second (GET) attempt provides credentials and that is why it succeeds. Initial HEAD request is used as "authentication probe" and is configured so that even non-200 response won't invalidate cache, as would by default happen with GET request with 401 response.