The random number required is an integer between 0 and $\ell$. Each person picks a random number $x_i$ between 0 and $\ell$, and each person picks an $n$-bit random blinding factor $b_i$ where $n\geq 128$ and $n$ must be agreed in advance. Each then announces a hash commitment calculated as $H(b \mathbin\|x)$. They then each open their commitments by announcing their values of $b$ and $x$, and calculate their random number as $$x_1 + x_2\ \pmod \ell.$$ This is quantum-secure if the hash and choice of $n$ are together quantum-secure.
As @poncho points out, the person that opens their commitment last can simulate a failure to complete the protocol if they are unhappy with the particular shared random value that would be generated during that invocation of the protocol. This could be a problem particularly if $\ell$ is small.
One way of avoiding this problem is to ensure that there is a forfeit for failure to complete the protocol. For example, each person could deposit funds with a trusted arbitrator that would confiscate those funds from the person that did not complete the protocol. The forfeit should equal or exceed the maximum possible gain from simulating a failure.