At this point, how does the attackers system know it has broken the encryption, especially if it has attempted many previous attempts & at insane speeds? Corollary: how can an attacker know they have decrypted successfully since the decrypted text would look exactly like encrypted text of the next layer of encryption?
It doesn't. But note that to verify that the encryption is broken only takes that the other cipherstream is distinguishable from random. That may be less costly than a full attack that requires decryption. Furthermore, depending on the construction, meet-in-the-middle attacks may be feasible.
Assuming various agencies, countries or hackers, are capable of breaking various encryption techniques, would such a system (from a high level viewpoint, based on question one logic) add any level of security?
Yes, but how much depends on the details. Meet-in-the-middle attacks were already mentioned. However, it could also be that both ciphers have a common structure, which means that if one can be broken then the other might get broken as well. In that case the effort is doubled, if that. If Grover is used for quantum cryptanalysis then we can assume that doubling the number of qubits would get you pretty far from breaking the double-encryption system.
Would reverse engineering by decompiling such a system enable all layers of encryption to be broken easily?
You cannot "decompile" a cryptosystem, that's term is not defined within cryptography. The algorithms are considered known, while the keys are considered secret.
