While it's certainly possible to install grub to your internal disk on a FAT partition (even the EFI), there is an easier way to almost get what you want (boot Windows when external SSD is not present, get grub and a choice of OS when SSD is attached). The basic problem is that both your Windows installs are bitlocker encrypted, so grub will not be allowed to boot them.
Copy all the files and directory structure from the internal EFI to the external EFI. Nothing needs to be changed. The only thing to check is that the EFI/Boot/bootx64.efi 's size is the same as EFI/ubuntu/shimx64.efi (or maybe grubx64.efi if you don't use secure boot). bootx64.efi is the default device bootloader, and that will give you grub's menu when run. The grub installer usually replaces the existing bootx64.efi (a copy of the Windows bootloader, with shimx64.efi. Now, check the internal disk's EFI/Boot/bootx64.efi's size, and ensure it is the same size as the internal disk's EFI/Microsoft/bootmgfw.efi. If not, there should be a "bkp" backup of the original, or just copy the bootmfgw.efi to the bootx64.efi to ensure it is a Windows boot.
Select how you want to boot:
If you put the external SSD first in the boot-order using your machine's UEFI settings for changing the bootorder, your machine will boot the external grub, and from it you can (only) boot Ubuntu. Override this to boot Windows by using the EFI menu (some key at powerup to allow you to select the device or Os to boot). Select Windows from the choices (or even the internal disk since it should have the bootx64.efi as a copy of bootmgfw.efi.
Put Windows second in the bootorder, so when the external SSD is absent, Windows will boot.
The internal disk's EFI/ubuntu files will no longer be used, but wont hurt anything if left there.
The external device's bootloader, ERI/Boot/bootx64.efi, you say is the same size as shim -- that's good. I like to have the file grubx64.efi in the same directory, but for 23.04 that didn't seem to be a requirement. The UEFI boot order should have the device at the top of the boot order, not the "ubuntu" entry. The "ubuntu" entry is in non-volatile ran (nvram), and deleting the UEFI files will not delete the nvram entry (which definitely cannot work without the files)
With bitlocker enabled, grub wont boot either Windows, so try putting the internal hard disk Windows first in the boot list, which should allow windows booting with or without the second disk -- no grub involved. To boot Ubuntu, when the second disk is attached, use the EFI menu key at powerup (some function key or del or esc...) to allow you to select device or Os to boot). Select Ubuntu from that (or device if the rootx64 default is shim/grub). grub should then run, and should be able to boot Ubuntu (but not Windows even if it is present in the menu, since bitlocker is enabled).
To repeat, I don't think you can boot a bitlocked Windows from grub, so if that's really what you want to do, consider disabling bitlocker on your Windows installs.