A "CMS-agnostic form that is not part of a Drupal module" could also be considered a "third-party PHP script", which is a security risk.
Now, you wrote this code yourself, so you can vouch for it. But, you are basically asking, "how can I inject a PHP script into Drupal?", which is something that Drupal doesn't want you to do as it can circumvent Drupal's various security features.
This means that the answer to your question is "It's hard and there will be compromises."
Here are three approaches you can take.
Approach: Host the script on the same server, but separate it from Drupal
One approach is to host the script in its own directory on your server, and then exclude that directory from Drupal. This exclusion can be done at the server level so Drupal never sees the script and the script doesn't see Drupal.
In this case, the script will not be part of the Drupal site, so you won't have the theme.
The server configuration depends on your server; you can Google it.
Approach: Embed it in an iframe
You said you didn't like scrolling in your iframes, but there are many JS libraries to work around this, such as
iframe-resizer. This is a Stack Overflow question though.
Approach: Use Webform
Re-write your form to use the Webform module. Yeah, it won't be CMS-agnostic anymore. But Webform is one of the premier Drupal contrib modules and can do basically anything-- many organizations choose Drupal because of this module. So it can obviously handle a contact form of any complexity, and it will give you full integration with Drupal.