Actually enabling advanced security has not impact on certificate validations - it just forces console to generate more secure certificates, but original ones would still work.
But what changes with enabling advanced security is that older TLS protocols (If I recall correctly, older than TLS 1.2) are disabled for AGENT connections, and also older and no-longer-safe cipher suites are disabled, which means that only devices with support for latest protocol versions would connection. Recent versions of AGENT do have this support, as they no longer rely on cryptographic primitives provided by operating system, but in case TLS introspection is used in between AGENT and SERVER, it might be blocked in case it does not support any of safe algorithm.
Regarding analysis, this seems to be network or TLS related, so I would recommend to analyze network communication using tools like wireshark. It is possible that problem is between TLS introspection component and SERVER, and not between AGENT and TLS component, so proper place for capturing of traffic will be required.