It seems to me you are copying 4 keys to your server.
This is taken from the online documentation located at
https://www.ssh.com/academy/ssh/copy-id
-i Specifies the identity file that is to be copied (default is ~/.ssh/id_rsa). If this option is not provided, this adds all keys listed by ssh-add -L. Note: it can be multiple keys and adding extra authorized keys can easily happen accidentally! If ssh-add -L returns no keys, then the most recently modified key matching ~/.ssh/id*.pub, excluding those matching ~/.ssh/*-cert.pub, will be used.
Update from the comments:
If you which to remove a certain key just use:
$ rm .ssh/id_target_key*
One of the backup options can be:
$ mv .ssh/id_target_key id_target_key.bak
Remember that there are always 2 keys. One private and one public. The latter has the extension .pub
Where to locate these keyfiles?
On the server open /etc/ssh/sshd_config and look for
AuthorizedKeysFile .ssh/authorized_keys
This file can be edited in order to remove your key manually, there might be other users keys installed.
On the client open /etc/ssh/ssh_config and look for an alternative path from there. (Mine is all commented out so ~/.ssh is the default location) ~/ is a relative path which changes per user. Executing the ssh command set with sudo will result in a different path: /root.
