Please don't do anything you'll regret later. Let's go through these CVEs and see what they're about:
CVE-2023-3817 & CVE-2023-3446
Published: 31/19 July 2023
Issue summary: Checking excessively long DH keys or parameters may be
very slow.
...
An application that calls DH_check() and supplies a key or parameters obtained from an untrusted source could be vulnerable to a Denial of Service attack.
CVE-2023-2975
Published: 14 July 2023
Issue summary: The AES-SIV cipher implementation contains a bug that
causes it to ignore empty associated data entries which are
unauthenticated as a consequence.
...
As this issue does not affect non-empty associated data authentication and we expect it to be rare for an application to use empty associated data entries this is qualified as Low severity issue.
The first 2 (CVE-2023-3817 & CVE-2023-3446) causes a long response time under certain conditions (which could theoretically lead to a DoS situation), and the last one (CVE-2023-2975) causes unauthenticated empty data.
All those CVEs are rated as low severity, meaning they'll get a patch when the Ubuntu Security Team believes this can fit into the releases schedule.
If you think you can make a better judgment of the CVEs than the Ubuntu Security Team, you are of course welcome to compile a newer version of the library yourself, but in this case you'll also have to maintain this for yourself going forward.
Most users are better off trusting the Ubuntu Security Team in these matters.