After a few days of pain I think I have a solution. In case it is of use to anyone else.
Firstly: Colorbox 2.0 (Drupal 9/10 version) cannot load entities. By default, it is only an image formatter. From the project site:
Colorbox for Drupal 8 is an image field formatter and nothing more, or
less.
Solution 1: HTML Formatted Media type Caption
It is possible to have colorbox load a Media entity image field and pass another custom field as the caption. With some tweaking, you can get html working as well. This can be setup on the entity display settings or in a view (as a formatter for the image field). It is not very flexible. The caption will, by default, also be the image title which can cause issues if you are loading html. You will probably also need to fiddle with the allowed html tags in the colorbox module's settings.
If that is what you need, that is doable. If you need anything more complex read on.
Solution 2: Adding full entities
There is a mobule that extends colorbox to allow the loading of entities it is called Colorbox Entity Display
It doesn't completely integrate with colorbox. In order to have the entity load you need to add a class to a link.
It DOES NOT work for media entities; at least I couldn't get it to work. You will need to make a normal content type and configure it to look how you want it to appear in the modal.
Colorbox will attempt to load the node portion of the content so you can just override the node template with whatever format you want.
The process for getting entities into a view based photo gallery is:
- Create view with at least - Image field, Link to content (as text and hidden), field for link title (hidden)
- Re-write the output of the image field as a link and pass the two other fields as well as the custom colorbox-entity class to the link.
After that, you should have a colorbox gallery with the ability to load entities into the modal.