Are there any security drawbacks or other things I forgot to consider?
Yes, GCM has security issues with how SRTP would want to use it.
SRTP is designed to have as low a bandwidth-overhead (that is, packet size) as possible; SRTP is sometimes used over wireless, and there, packet size is quite expensive. IIRC, the only bandwidth SRTP adds is the integrity tag (the IV, if the mode needs one that never repeats, can be a function of the position in the stream, which is included in the unencrypted section of the packet already); we would like to reduce the tag to be as short as possible (both to reduce the cost, and for SRTP, having someone introduce a single bogus packet isn't all that bad - for audio use, they might be able to modify 20msec of the voice connection at a not-too-high ($<10^{-6}$) probability, and that was considered tolerable for this use case).
When we get to GCM, we have a problem with short tags. GCM has this property: if we have an encrypted GCM message with IV, ciphertext/tag $(IV, C, T)$ and someone finds a bogus message $(IV, C', T')$ that is accepted (note: same IV was used, the ciphertext and tag was modified arbitrarily), then for any future encrypted message $(IV", C", T")$, then the message $(IV", C" \oplus C \oplus C', T" \oplus T \oplus T')$ will also be accepted. That is, by finding a single forgery, the attacker can introduce forgeries from here on out.
And, if you have a short tag, then the attacker has a decent probabilities of finding an accepted bogus message $(IV, C', T')$ by simply guessing various $T'$ values.
A single forgery was considered tolerable; allowing the attacker to generate a continuous stream of forgeries is less so. With GCM, we can avoid this issue by using a long tag (where the probability of randomly guessing it is sufficiently low), however the use cases for SRTP doesn't like long tags.
HMAC doesn't have this property; if the attacker does find a forgery $(C', T')$ (HMAC doesn't have an IV), that does not help him to generate any future forgeries.
