Libreswan and Mac OS X Big Sur client issues

I'm resorting to asking for help after a brutal amount of time troubleshooting connection problems between client and server.

Troubles

Mac OS X Catalina, and Linux clients work fine connecting to the server, but Big Sur does not. I haven't yet tested Mojave (which would in theory be more lax with security anyway), nor have I tested Windows 10. The reason for all the different clients is complex, but essentially we're a small consulting company where people can choose their platform albeit older Mojave is simply because they haven't upgraded yet. Either way that makes my job (as the sysadmin) significantly more complex, but there it is.

Servers

• RedHat 8.2. One is configured in FIPS mode, another is not, due to OpenVPN. Both are of course running Libreswan.
• Libreswan default install from yum. Version: 3.29-7
• IKEv2 using NSS, with RSA certificates for authentication.
• Custom Root/Intermediate CA infrastructure. CRL is valid, integrated, and stored in NSS.
• Firewall, routing table, sysctl settings, etc., all seem to be configured correctly. I'm using iptables with a custom rule set, rather than firewalld.

Clients

• Mac OS X Mojave, Catalina, and Big Sur
• Windows 10
• Ubuntu and RedHat Linux clients

Configuration

Client Side (Mobileconfig to package it all together)

I'm using the built in network client, no external clients to simplify things. Most for-pay clients do not seem to support IKEv2 anyway, although I did briefly try GreenBow VPN.

VPN Payload tab (Apple Configurator 2, both Catalina/Big Sur same data/configuration)

1. Connection Type: IKEv2
2. Server/Remote Identifier: IP of server (Server certificate has SubjectAltName IP: server IP)
3. Local Identifier: Email address of user seems to work well here assuming it's not Big Sur. Setting it as blank also worked at one time. I'm not sure what I changed or if it was even me that did, but Email needs to be present now. Ive begun adding this as a subjectAltName email field client side as well, at suggestion from coworker.
4. Machine Authentication is certificate, RSA for type, and CA issuer name is set correctly.
5. All the other security stuff, like DH params and crypto are mid grade, AES-256 and DH21. I had originally configured with AES-256-GCM to try to avoid anything related to the SHA2 truncation bug. That does not work either.

Certificates Payload tab

1. I've configured this in different ways. Standard stuff applies here though: A .p12 identity, with both a private key password for the user, as well as an export password (both 23 characters long). Length of time is 920 days. I did read somewhere that this could be the problem, but the logs I've found client side do not reflect being upset with the length of time it's valid. Also, I did test a shorter certificate time, 500 ish days to see if that did anything nice; It did not.
2. The root/Intermediate CA has been included in two configurations here during the PKCS12 creation: Just the Intermediate CA, versus the full ca-chain. Neither configuration has worked in Big Sur.
3. Manually imported the root/intermediate CA's; I've marked both as trusted for all use cases. I've imported/copied them from the login keychain, to the System keychain. I've done everything that I can find on this subject, and nothing has worked.

Server side Configuration

auto=add
  authby=rsasig
  dpddelay=30
  dpdtimeout=120
  dpdaction=clear
  fragmentation=no (Big sur gives a different error with this on)
  modecfgpull=yes
  modecfgdns="Internal DNS"
  pfs=yes
  rekey=yes 
  salifetime=8h
  ikelifetime=8h
  left=PUBLIC IP
  leftcert=PUBLIC IP (I've tried varied forms here; @PUB... etc)
  leftid=PUBLIC IP
  leftsendcert=always
  leftsubnet=0.0.0.0/0
  leftrsasigkey=%cert
  leftmodecfgserver=yes
  rightaddresspool=Internal address pool
  right=%any
  rightrsasigkey=%cert
  rightmodecfgclient=yes

Client Side Logs

First off, the only GUI error popup I see here on connection attempt is "User Authentication Failed". I then use the following command to obtain better/more useful logs on Mac OS X:

log show --start DATE --predicate 'senderImagePath contains[cd] "NetworkExtension"'

These logs are, in large part, 1000 lines of trash. That said, this seems relevant:

2021-08-10  0x1aa44    Error       0x0                   2375   0    NEIKEv2Provider: (NetworkExtension)  [com.apple.networkextension:] Certificate evaluation error =  kSecTrustResultRecoverableTrustFailure
2021-08-10   0x1aa44    Error       0x0                  2375   0     NEIKEv2Provider: (NetworkExtension) [com.apple.networkextension:] Certificate is not trusted
2021-08-10  0x1aa44     Error       0x0                  2375   0    NEIKEv2Provider:  (NetworkExtension) [com.apple.networkextension:] Certificate  authentication data could not be verified
2021-08-10   0x1aa44    Default     0x0                  2375   0     NEIKEv2Provider: (NetworkExtension) [com.apple.networkextension:] IKEv2IKESA[2.2, C4CABCADAB06C2CF-75E7F26177F83187] state Connecting  -> Disconnected error (null) -> Error Domain=NEIKEv2ErrorDomain  Code=8 "Authentication: Certificate authentication data could not be  verified" UserInfo={NSLocalizedDescription=Authentication: Certificate  authentication data could not be verified}
2021-08-10   0x1aa44    Error       0x0                  2375   0     NEIKEv2Provider: (NetworkExtension) [com.apple.networkextension:] IKEv2Session[2, C4CABCADAB06C2CF-75E7F26177F83187] Failed to process IKE  Auth packet (connect)

Notes

As described in the earlier comments about client configuration .. I've done the work to get this right, seemingly. It shows even in the Apple Configurator display, that this is an acceptable configuration package. Obviously it's not signed by Apple, but no surprise. The standard, manual IKEv2 configuration is simply not comprehensive enough for our needs. And it just doesn't work, to be sure, either. At least for anything more complex than a child's toy of a server.

I've also researched the Certificate Transparency problem; Unfortunately this is not a solution for Mac OS X. It is only a solution applicable to devices such as iPads, etc. Attempting to install a mobileconfig to OS X with CT enabled and excepting certain certificates, it's rejected and will not install the mobileconfig.

Server Side Logs

I have the external server in debug mode, to generate more logs than normal. That said, in this instance the server seems perfectly happy with the transaction, it's client side that hates ........ Something. I don't know what.

Here's what I see though:

Aug 10  serverName pluto[]: | offered CA: 'C=US, ST=US, O=companyName, OU=OUCA, CN=CAName, [email protected]'
Aug 10  serverName pluto[]: "remote"[51] serverIP #90: IKEv2 mode peer ID is ID_USER_FQDN: '[email protected]'
Aug 10  serverName pluto[]: | received v2N_INITIAL_CONTACT                              
Aug 10  serverName pluto[]: | Received unknown/unsupported notify v2N_NON_FIRST_FRAGMENTS_ALSO - ignored
Aug 10  serverName pluto[]: | received v2N_MOBIKE_SUPPORTED while it did not sent       
Aug 10  serverName pluto[]: | received CERTREQ payload; going to decode it              
Aug 10  serverName pluto[]: | CERT_X509_SIGNATURE CR:                                   
Aug 10  serverName pluto[]: |   10 81 9a c1  4c a6 94 70  ca d1 7d 77  e1 5a ab 36      
Aug 10  serverName pluto[]: |   93 3d cd 39                                             
Aug 10  serverName pluto[]: |   cert blob content is not binary ASN.1                   
Aug 10  serverName pluto[]: | verifying AUTH payload                                    
Aug 10  serverName pluto[]: | required RSA CA is '%any'                                 
Aug 10  serverName pluto[]: | checking RSA keyid 'serverIP' for match with '[email protected]'
Aug 10  serverName pluto[]: | checking RSA keyid 'C=US, ST=US, L=City, O=companyName, OU=OUCA, CN=UserCN, [email protected]' for match with '[email protected]'                                                                                                                         
Aug 10  serverName pluto[]: | checking RSA keyid '[email protected]' for match with '[email protected]'
Aug 10  serverName pluto[]: | trusted_ca_nss: trustee A = 'C=US, ST=US, O=companyName, OU=OUCA, CN=CAName, [email protected]'
Aug 10  serverName pluto[]: | key issuer CA is 'C=US, ST=US, O=companyName, OU=OUCA, CN=CAName, [email protected]'
Aug 10  serverName pluto[]: | an RSA Sig check passed with *AwEAAbi14 [remote certificates]
Aug 10  serverName pluto[]: "remote"[51] serverIP #90: Authenticated using RSA          

The server logs show, that the Mac OS X Big Sur client, authenticates properly. It proceeds to then build the SA relationship, etc., as you'd typically see. The client, is who rejects the connection out of hand. What I don't understand is what changed between Catalina, versus Big Sur ??

I have found no information on this OS change. I do not understand why this isn't in 60 point font, in bold, on the front page of the Apple website that oh yea, fun fact, we totally changed the way IPSEC works in this version of the OS. And, I'm tired of bloodying my head either pulling hair out, or banging it against a wall repeatedly.

Any help would be appreciated.