That is Linux being economic with memory. You have high memory usage at some point. Thus, linux swaps memory to the disk. When later, you have less memory requirements, linux leaves the swap on disk, so the memory is available for disk cache and for anything else you are doing, or plan to be doing. Only when you need the memory residing in the swap space, will linux retrieve it from swap. Not earlier. Since you do not need it anyway.
Lowering swappiness effectively will reduce the tendency of the OS to swap pages to disk, and increase the tendency of the OS to cache disk files to memory.
that just gives the system the order in which pages are to be evicted from physical memory, not how soon to start evicting pages
Not quite correct, in my opinion. Memory management is complex in reality]. This is the definition from the Linux documentation source:
"This control is used to define how aggressive (sic) the kernel will swap memory pages. Higher values will increase aggressiveness, lower values decrease the amount of swap. A value of 0 instructs the kernel not to initiate swap until the amount of free and file-backed pages is less than the high water mark in a zone.
You can indeed lower swappiness: it will reduce the tendency of the system to swap out paged memory. In return, your hard drive will have more work physically writing files, because these will be cached less. What the net effect is on disk strain, depends on actual usage.
