Why does "hex encoding of the plaintext before encryption", "harm security"?

Hex encoding of the plaintext doubles it's size, while preserving entropy. Hence entropy per bit is halved.

For weak encryption algorithms (only), this typically makes attack easier. For example: the original context was XOR with a repeated key. For uniformly random key the size of a single plaintext, that is perfectly secure. But it's no longer after hex encoding.

Detailed example following comment: The cipher is XOR with a 5-bytes uniformly random key repeated as necessary to match plaintext length. Without hex encoding, given the ciphertext for a single 5-byte plaintext, we can't deduce anything about the plaintext, e.g. decide if it's for ALICE or OSCAR (in ASCII). But with (uppercase) hex encoding these become 414C494345 or 4F53434152 (in ASCII). These are 10-byte each, so that the same key byte must be reused for the first and 6^th characters. These are 4 and 9 (34_h and 39_h) for ALICE, 4 and 3 (34_h and 33_h) for OSCAR. Thus we need only XOR the first and 6^th bytes of ciphertext: we'll get 0D_h (34_h XOR 39_h) for the plaintext ALICE, versus 07_h (34_h XOR 33_h) for OSCAR. More generally, XOR of j^th and 5+j^th bytes is independent of key, and characteristic enough of plaintext that's a serious issue.

^{Rebuttal of this comment: in the above example, the addition of hex encoding does NOT change the size of the effective plaintext, which remains the same 5 characters. That example does illustrate that all things being equal, encoding the plaintext in hex tends to weaken an already weak cipher. At the expense of a much more complex example, it would be possible to come up with a cipher with fixed-size key/password that's practically secure for arbitrarily large plaintext with the usual level of redundancy in English, but not for the increased redundancy of hex-encoded text.}