In FORS few time signature, How Adversary A who observed the signatures of r messages, finding a message $ m_{r+1} $ that is in an r−subset cover relation with the other r messages $ (C^{r-FORS}_{k} (m_{1}, m_{2}, . . . , m_{r+1})) $ has probability of success $(r/t)^κ $ ???
Actually, there are two potential attacks against FORS - one is a simple second preimage attack on the hash that hashes all the roots together; the difficulty of that attack is $O(2^n)$, where $n$ is the length of the hash function and we assume that the hash function allows no better attacks than brute force search.
However, that's not the attack you're asking about. In the attack you are considering, the attacker has $r$ valid signatures for this FORS public key - each signature will reveal one of the leaf nodes of each FORS tree (of which there are $k$ of them). Hence the attacker knows, for each tree, at most $k$ of the leaves (there is a possibility that two different signatures will reveal the same leaf - we will make the conservative assumption that doesn't happen, both because it is unlikely for the parameters we use in practice, and because such a collision makes things harder for the attacker).
Now, what the attacker does is take a guess and hash that into $k$ leaves of the FORS trees - he wins if each of those leaves happens to have been revealed; if that happens, he can copy the authentication paths from the valid signatures, and thus produce a forgery. Now, each FORS tree has a total of $t$ leaves of which $r$ is revealed; assuming the attacker cannot control which leaf is selected (that is, if the hash function that selects the leaves is strong), then he wins with each FORS tree with a probability $r/t$. Now, there are $k$ FORS trees, and he has to win all of them with the same guess; if these are independent (again, the hash function that selects the leaves is strong), then his total winning probability is $(r/t)^k$ per guess.
Or, bottom line, the expected work effort of the attacker trying to generate a forgery is $\text{min}(2^n, (t/r)^k)$