Jump to content

Protocol Mismatch detected RDP communication over non standard port [E0517]

Recommended Posts


New to Inspect so still learning the ropes so to speak

I seem to get this alert periodically on all of my internet facing servers.

Category: ESET Inspect alerts
Computer name: cleansed
Computer static group hierarchy: cleansed
Rule: Protocol Mismatch ‑ detected RDP communication over non‑standard port [E0517]
Time Detected: 11/18/22, 2:09:34 AM UTC
Process Detected: %SYSTEM%\ntoskrnl.exe
Severity Rating: Critical 

Is this someone port scanning and trying to be clever, and just noise, or do i have a bigger problem?

I have RDP allowed internally, but not from the internet.


Edited by FTL
Link to comment
Share on other sites

  • ESET Staff

This could be a sign of being port scanned from the outside world.  It is not uncommon to see port scans attempting to identify what service is available on each port.  This means that something is testing non-standard RDP ports to see if RDP is running.  Basically the equivalent of running "nmap -sV -Pn -p 445 %YourPublicIP%" where the port number does not have to be 445, but could be a range of ports, or a different port.

But, I am unable to tell which port if this is what the EI Rule triggered on, or if this was Inbound or Outbound traffic from a public or private IP.  As it is the ntoskrnl.exe process, I assume its potentially inbound on, 445 or something else.

  1. If you log into the EI Console, then navigate to the main "Detections" section (on the left)
  2. Clear any un-needed filters, and sent the filters to:
    • Resolved = Unchecked
    • Rule Name = [E0517]
    • First drop down list on top left = Ungrouped
  3. Next we will add a column by clicking the gear on the far right, then clicking "Select Columns" and choose the following column and then click OK:
    • Trigger Event
  4. Scroll to the right and locate the "Trigger Event" column

This will give you a nice list to look at where you can easily see

  • If it is "Inbound/Outbound"
  • If a "Private or Public IP" was involved
  • What port was involved

If it is inbound from a Public IP, it means the identified Port is exposed to the internet and the identified IPs are testing if RDP is running on that port.  If it is port 445, it should be closed off from the internet immediately.

If it is inbound from a Private IP, it means that private IP is potentially performing a port scan or has some software which is not using a standard protocol to communicate with other computers.  Or that you have altered the default RDP port on your network, and should follow Marcos' advice.

If it is outbound and mstsc.exe is the process causing the detection (which is not shown in the data you shared), then you should just verify if you allow users to connect to RDP servers on non-standard ports and potentially lock down protocol.


Hopefully this helps.  There could be more possibilities than what I listed, but I commonly do see where customers did not realize they had ports like 445 exposed to the world, and EI helped to point that out for them.


Link to comment
Share on other sites

Hi JamesR

Thankyou for this.

I have peformed the steps above and can confirm they are all inbound from public IPs and on port 443 (which is leitimately open on the firewall for these servers)


They are all ntoskrnl.exe apart from my PRTG monitoring server which is prtgserver.exe but again external IP and on port 443 which is legitimately open on the firewall.

Command lines are all None and username is "nt authority\system"


Link to comment
Share on other sites

  • ESET Staff


Thanks for the additional info.  Its starting to sound like your server may be hosting Remote Desktop Services with Remote Desktop Gateway.  Which will have ntoskrnl.exe listening on port 443 for RDP requests to forward to other servers.  While this might be intended, the trigger event info you shared shows an IP from Russia attempting the connection (I used this site to look up the location of the IP address: https://www.maxmind.com/en/geoip-demo).

I highly recommend reviewing the roles installed on the server to verify if "Remote Desktop Services" (previously called Terminal Services) is one of the installed roles.

These detections could very well be a sign of an RDP Brute Force attack from undesired IP addresses.  If this server is an RDP Gateway, and 443 needs to be open to the internet, I would recommend restricting which blocks of IPs you allow to connect.  Geo-IP Blocking could help reduce connections from attackers, but any compromised device in your country, could continue an attack.  And ensure you are using 2fa on any RDP logins (especially Admin logins).

With that said, there is still a chance its not Remote Desktop Services with an RDP Gateway.  That is just what I would expect at this point.

If you have a list of IP Addresses which are allowed to connect, that you want to exclude from triggering this detection, you can use the following exclusion as a template to modify and meet your needs.  Then you will only get detections on untrusted IP addresses:

        <!-- Describe process to apply exclusion too -->
        <operator type="and">
            <!-- SignatureType of 90 = Trusted -->
            <condition component="Module" property="SignatureType" condition="greaterOrEqual" value="90"/>
            <condition component="FileItem" property="FullPath" condition="is" value="%SYSTEM%\ntoskrnl.exe" />
            <condition component="ProcessInfo" property="ProcessOwner" condition="is" value="nt authority\system"/>
        <operation type="TcpIpProtocolIdentified">
            <!-- List of IP Addresses to exclude from triggering detection.  Accepts CIDR notation. -->
            <operator type="or">
                <condition component="Network" property="IpAddressV4" condition="is" value="" />
                <condition component="Network" property="IpAddressV4" condition="is" value="" />


Side note: RDP can be brute forced, and its not uncommon for someone to have setup a secondary admin account, with a weaker password and no 2FA, to use incase their primary account is not working (disabled due to bad passwords, or 2FA isnt functioning as expected).  It is not uncommon that a ransomware attack starts with a form of brute force on exposed services (RDP, SMB, vCenter/ESXi web console, etc...).  Also, if some form of Remote Code Execution is discovered, or only known to attackers, it could allow them to walk right in without authentication (In 2017 WannaCry used an RCE on SMB to spread without needing any credentials).

Link to comment
Share on other sites

Hi JamesR

No none of the servers have RDS installed.

They are IIS web servers, my PRTG monitoring server (which i cant easily lock down 443 access via IP for as the mobile app needs to connect over 443 for push notificaitons to work to my tech's, and an Exchange on prem server, which again has 443 open for its operation.


Of the 5 servers that are getting this alert:

IIS Web Server 1 - only has port 443 open and natted on the firewall for inbound internet traffic

IIS Web Server 2 - only has port 443 open and natted on the firewall for inbound internet traffic

IIS Web Server 3 - only has port 443 open and natted on the firewall for inbound internet traffic

PRTG Monitoring Server - only has port 443 open and natted on the firewall for inbound internet traffic

Exchange Server - has ports 25 and 443 open and natted on the firewall for inbound internet traffic.

No other ports are open on the firewall to these devices


Link to comment
Share on other sites

This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Create New...