If (with a very small chance) there exist $x_1$ and $x_2$ that $H(x_1) = H(x_2)^2$, then Bob could discover it ... the simulation should always succeed, and should not depend on the input $\{x,y\}$.
I agree with your friend that this observation doesn't violate semi-honest security.
In the semi-honest model, the inputs are independent of the random oracle. In other words, the inputs are fixed first, and then $H$ is sampled. The environment does not have access to the random oracle (in the local random oracle model), so its choice of inputs for the honest parties' doesn't depend on the random oracle. So the event that $H(x_1) = H(x_2)^2$ does not depend on $x_1, x_2$. It is negligibly likely for all inputs, so the simulator can safely ignore this case.
In the PSI protocol (Fig.3) of [3],
I'm not sure what your question here is. In [3] they need a malicious-secure PSI protocol, and the classic DH-PSI protocol is not. So they use the more complicated protocol of Jarecki & Liu.
The bigger question seems to be about whether classic DH-PSI "can't be simulated," where presumably you mean against malicious adversaries. I agree. Just think about the case where the parties have 1 item each. Consider the case where a corrupt Alice queries the random oracle at $x_0$ and $x_1$, flips a coin $b$, and sends $H(x_b)^a$ for random exponent $a$. The random exponent $a$ makes the simulator's view (random oracle queries $x_0, x_1$ and protocol message $H(x_b)^a$) perfectly independent of $b$. But the simulator must guess $b$ in order to extract correctly.