It's not really clear what you are asking.
In the best case, the hacker has to know the IP address and the domain name in order to access the server. Someone who just happens to visit the IP address cannot easily discover the name of the web site if the server's default configuration does not reveal it.
The crucial question then is how exactly the system is set up and whether that configuration can be trusted to not leak the information you want it to protect. We have no information about this, so the short answer is simply "it depends."
Still, someone who knows which domain name you are likely to use and who is specifically striving to access this particular server could perhaps find out by some means which IP address you are connecting to. If they have visibility into the network you are connected to, they would just have to look where the packets from your computer are going in order to discover this.
Depending on the risk scenario, you might want to consider adding additional layers of security around the system. Perhaps require authentication in order to access the server (and/or require a private SSL certificate), and/or use a VPN to hide the destination of your outbound traffic.