Generally it is possible to reverse-engineer a protocol by looking at messages. Commonly, protocols are build upon existing technology, such as X.509 certificates, ASN.1 BER/DER, binary encodings such as CBOR, textual encodings such as XML or JSON etc. These kind of of encoding structures contain a lot of information about what kind of information is being transmitted.
In most cases no effort is taken place to hide the protocol - it is simply not needed. There are many open protocols being used, and it would be easy to detect those. Keys can usually also be detected, even if meta information such as above is not available. If ECDH is used then the participants need to exchange public keys. It is usually possible to detect the keys from the handshake messages, e.g. by checking that the keys are specifically on a curve.
That all said, it is likely possible to create a protocol which is hard to reverse engineer if you just get the bytes from intercepted messages. But remember: even if the protocol is kept secret, it only requires a single leak to find what protocol is being used. As such, the hiding the protocol itself can be thought of as an obfuscation effort; it provides only limited security (this is usually referred to as the Kerckhoff principle).
Conclusion: sometimes it might be tricky to find out how a protocol is behaving when looking at the actual bits and bytes. However, generally the protocol descriptions are readily available and the security is determined by the keys and algorithms. Even if the protocol description is not directly available it is possible to figure out a lot of detail by reverse-engineering the messages (in the handshake).