Searches here indicate that D/H primes of 2048 bits are "safe"? How do we know that this is true?
Actually, we don't - we don't know that Diffie-Hellman with any specific sized prime is secure [1]. That is, we don't know that the Diffie-Hellman problem is hard, that is, if the attacker sees the values $g^x \bmod p$, $g^y \bmod p$, and also knows the values $g, p$, it might be an easy problem to find the common value $g^{xy} \bmod p$.
Now, for a properly chosen value $p$ of 2048 bits in length (and also a good value of $g$ and $x, y$ chosen from good distributions), well, a lot of clever people have thought about it, and no one knows a way. Now, it is certainly possible that everyone missed something, however that's generally always true in cryptography.
So, you ask, why don't we go ultraconservative and use huge (say, 60,000 bit) primes? Well, mostly because of the costs:
60,000 bit primes would imply 60,000 bit = 7.5kbyte keyshares - that would need to be exchanged each time. Exchanging 15k of data for each new exchange is more costly than we'd prefer.
The time taken - you said that a DH operation would take 'only a few second'; however if you're on a big TLS server making tens of thousands of connections per second, a 'few second' per connection is rather a big deal.
[1]: The term 'safe prime' is most often used to denote a specific meaning; it is a prime $p$ with the value $(p-1)/2$ also being prime. I assume that you don't intend that specific meaning of 'safe' and that you really mean 'secure'