How to implement Continuous Integration with Puppet and multiple services

We're trying to implement a Continuous Integration pipeline in our environment. We have a lot of different services, each with its own Git repository. Deployment is done via Puppet, using an external node classifier that determines what classes to deploy to each host type. And the Puppet files are sitting in their own Git repo, as depicted here:

Only, it's not just 3 services, it's more like 100. So the Puppet project is this monstrous monolith of multiple manifests, and of course it's in its own independent Git repo.

Now, along comes lil' ol' me, tasked with setting up a pattern for CI, so that when someone requests to merge a branch from, say, Service A, into master, we should be able to kick off a CI build that will spin up a virtual environment, deploy Service A to some VMs, and ensure that the new branch passes all automated tests. Of course, the problem is that to deploy a new build of Service A, I not only have to build it, but I also have to update the Puppet manifest to refer to the new build version...and the Puppet files are sitting in a completely independent repo, not on my branch. So I don't have any easy way to tell the Puppet Master that for this branch, we need to use the CI build, not the master version.

I can't be the first person ever to want to set up CI for an environment like this, but I've searched the web for solutions and come up empty. Perhaps I'm using the wrong search terms.

Can anyone suggest a suitable design pattern that will enable me to implement CI for all my service repos?

The easiest (but not necessarily the best) way I can see to do this is to set something up that requires that for the CI to build out the test environment, it requires matching branch names or something. That would force developers to create a branch in the Puppet code to deploy their build version.

Effectively:

1. Dave the Developer creates a new feature branch of a service, based on a ticket: e.g. feature/JIRA-999-some-task
2. Dave pushes his code
3. CICD detects new branch or PR or whatever, starts running
4. Stage of the pipeline looks for the same branch name in the Puppet repo
5. If branch doesn't exist, pipeline errors and tells Dave to go create the branch
6. If it does exist, maybe add some validation to check it has been changed at least?
7. Tests run
8. Tests pass, everyone is happy

It's not an easy problem to solve, and this approach you'd have to ensure Devs knew where to update the version (maybe in a Hiera data file?).

The only other way I can see this working is a much bigger shift to something like containerization, allowing you to manage one layer (the infrastructure) with Puppet, and then deploy containers on top. You could then give developers the tools to deploy the containers whilst retaining control of the platform.

One final random thought - could you do some of the testing in the cloud to avoid using the Puppet code for anything except prod/staging? Or leverage cloud-based puppet in some way?

The challenge here is that the code is disconnected from the deployment.

My approach would be to pass Git commit hash to Puppet, and then Puppet would use that commit hash to pick up the code to deploy. However, I don't know if Puppet supports passing such arguments to jobs.

During check-out phase of the actual code repository, the hash would be recorded and then passed to Puppet.