What makes Argon2 also slow?

So as I understand Argon2 is a memory-hard function, in other words it has to use a certain amount of memory, thus making it effective against GPUs.

So would this in theory, make Argon2 useless against an adversary with infinite memory?

But also if this is the case, then why does giving Argon2 a higher memory requirement also make it slower? Shouldn't it just take the same time but with more memory? Is there some more processing going on too? Or is it slower because of RAM read/write times?

Giving Argon2 more memory does indeed make it slower. When Argon2 fills the memory that it uses, it calls the G function, which modifies data based on the BLAKE2b round function. Increasing the memory also increases the number of columns iterated over in each iteration, each of which also involves a call to the same G function. As a consequence, doubling the memory also means effectively doubling the amount of work required. You can see much of the algorithm pseudocode on Wikipedia.

Argon2 would not be useless against an adversary with infinite memory unless they had some way to make operations much faster for their additional memory. In practical computer designs, even with very performant vector operations, this would still add additional cost.