TLDR: The size of the group/ring is dictated by the fastest currently-known attack (as explained in this Wikipedia article).
Details. For the case of discrete-log in $\mathbb{Z}_p^*$ and factoring $\mathbb{Z}_N^*$, the fastest currently-known algorithm is the general number field sieve (GNFS). GNFS has a run-time of (roughly) $L_n(1/3,2)$, where $$L_n(\alpha,c):=e^{(c+o_n(1))n^\alpha\ln^{1-\alpha}(n)}$$
is the $L$-notation and $n$ denotes the bit-length of (the standard representation of) $p$ or $N$ (i.e., $\lceil(\log(p)\rceil$ and $\lceil(\log(N)\rceil$, respectively).$^*$ Since $b$-bit security for a scheme means that it should take any algorithm $2^b$ operations to break it, to compute the $n$ for $\mathbb{Z}_p^*$ and $\mathbb{Z}_N^*$ that achieves $128$-bit security one has to solve for $$2^{128}\approx e^{2n^{1/3}\ln^{2/3}(n)}\Leftrightarrow n\ln^2(n)\approx64^3.$$ This will give an ball-park figure of what the size of the modulus should be -- as computed in this answer this turns out to be around $3072$ bits (or $4096$ bits to be on the safer side?). Since we do not know any better means to solve DDH/CDH (the problems that underlie El-Gamal-type schemes) than to compute discrete logs, El-Gamal in (quotiented) $\mathbb{Z}_p^*$ needs to be deployed with primes of size $\approx3072/4096$ bits.
Similarly, since we do not know any better means to solve the decision quadratic residuosity problem in $\mathbb{Z}_{N^2}^*$ (the problem that underlies Paillier's cryposystem) than to factor $N$, by the same argument above, we need to work $N$ of size $\approx 3072/4096$ bits.
For discrete-log in elliptic curves, I believe we do not know anything better than generic discrete-log algorithms (e.g., Pollard's rho) which run in time square-root the size of the group. Therefore, for $128$-bit security of EC-El-Gamal, it suffices to work with elliptic curves over a field of size $2^{256}$. (This also means that EC-El-Gamal is significantly more communication-efficient than El-Gamal in $\mathbb{Z}_p^*$.)
$^*$The original GNFS is a heuristic. But as pointed out by @djao (see this comment), there are provable variants which run in $\approx L_n(1/3,3)$.