P in this context means the ratio of CPU time this job got over total time elapsed. Near 100% means almost all of the time it was on CPU, and so was CPU constrained for those runs. In contrast to the other runs where something else was the limiting factor. More system (aka kernel) time than system time, as is typical of I/O heavy tasks.
Given the workload was copy a 6 GB file, we can infer the 11 second runs averaged more than 0.5 GB writes per second. O column confirms the same number of writes each time, consistent with a simple copy one file process.
Input column has major swings, however. Slow runs have about equal reads as writes. But the fast runs don't do any reads! I assume the file is still cached in RAM from when it was last read. DRAM is much faster than even solid state storage. Which is a great speed boost, until under memory pressure the OS drops the cached data, and has to read from slow storage again.
So this is a 200 second task, which occasionally can take 12 seconds. Likely due to Linux page cache.
Finding root cause of a performance problem often requires a deeper understanding of the overall system, beyond any particular set of metrics.
The filesystem being used is a remote, network attached storage device.
Note your copy thing is over networked storage, so it could also be anything on the remote system or the network in between. Remote storage performance. Network (probably IP) speeds and utilization. Or it could be local to this VM, where the guest is competing for resources with everything else running on your infrastructure.
Always possible to go deeper into how things work. Does network storage (NFS?) matter at all, or do you also see this for local disk? 0.7 seconds of user CPU time is quite a bit of work actually, how much is accounting to manage many system calls? What does CPU busy actually mean when most of this is waiting for slow memory and very slow storage? Not easy questions to answer, however perhaps no need to dig too deeply once the thing is performing adequately.
