Within IIS in the Bindings for your website, have you got a binding entry for the IP and port with no hostname?
Since IIS allows you to have multiple websites, using multiple combinations of hostnames, IP addresses and port numbers, it determines which website you're trying to reach by taking your request and finding the website that has a matching binding. The combination of those three elements is always unique across the sites hosted on that server.
So in your case you'd want your site to have two bindings. One for the hostname/IP/port combination you have now, and the second almost identical entry with an empty hostname entry.
Note, if IIS is already listening on port 80/443 using that IP address and no hostname elsewhere (for instance in a default site that's still running) then it won't let you add it, but that could also explain why you're getting a different result as a different website is responding to the query. So while the URL connection is throwing a 500 error indicating an issue with the website itself, the IP 404 is indicating it's being sent elsewhere, but somewhere that perhaps the default page isn't found hense the 404 file not found error.