(This answer pertains to Signal protocol, which underlies applications like Signal and WhatsApp. As far as I know, WhatsApp software is not open source and therefore it is hard to ascertain whether/how they actually implement the Signal protocol. The source-code of Signal application is, on the other hand, available.)
On a high level, the Signal protocol builds on the off-the-record messaging protocol [BBG]. The security analysis of Signal protocol for end-to-end messaging (compared to the protocol for group messaging, which is addressed below) was carried out in [CCG,CCDGS,B+]$^*$. Briefly, these works realised that the core component of the signal protocol is a cryptographic primitive called (unidirectional) ratcheted key exchange/ratcheted encryption (you can read more about ratchets in this post). A ratcheted key-exchange (e.g., Diffie-Hellman ratchet) protocol, in addition to the guarantees provided by a vanilla key exchange protocol (e.g., Diffie-Hellman and its derivatives like MQV), provides security properties like forward secrecy and post-compromise (or backward) security. Further analysis of ratcheted key-exchange was carried out in [PR,ACD]: they strengthened the results in [CCG,CCDGS,BCJNS] to take into account bidirectional or double ratchets.
The group messaging protocol used in Signal/WhatApp, is being currently standardised by IETF: see MLS working group for latest updates. The underlying protocol is called continuous group key agreement$^{**}$ and it aims to extend the ratcheting key exchange -- which is limited to two users -- to dynamic groups with multiple users. One of the main goals here is to have the communication complexity of the protocol scale sub-linearly with the number of members in the group.$^{***}$ You can read more about the security analysis of the proposal (which build on protocols called ART [CCGMM] and TreeKEM [BBR]) in [ACCKKPW,ACDT] and the papers they refer to.
$^*$ You can read more about the differences between the results in [CCG,CCDGS,BCJNS] in the related works section of [CCDGS] and [BCJNS]
$^{**}$ The end-to-end ratcheting key exchange is also sometimes referred to as continuous key agreement.
$^{***}$ Group ratchets can be simulated using end-to-end ratchets if one does not care about the blow-up in communication complexity.
[ACCKKPW]: Alwen et al, Keep the Dirt: Tainted TreeKEM, Adaptively and Actively Secure Continuous Group Key Agreement, EuroS&P 2021
[ACD]: Alwen, Coretti and Dodis, The Double Ratchet: Security Notions, Proofs, and Modularization for the Signal Protocol, Eurocrypt 2019
[ACDT]: Alwen, Coretti, Dodis and Tselekounis, Security Analysis and Improvements for the IETF MLS Standard for Group Messaging, Crypto 2020
[BCJNS] Bellare et al, Ratcheted Encryption and Key Exchange: The Security of Messaging, Crypto 2017
[BBR] Barnes, Bhargavan and Rescorla, TreeKEM: Asynchronous Decentralized Key Management for Large Dynamic Groups
[BBG] Borisov, Brewster and Goldberg, Off-the-Record Communication, or, Why Not To Use PGP, CCS 2004
[CCDGS]: Cohn-Gordon et al, A Formal Security Analysis of the Signal Messaging Protocol, EuroS&P 2017
[CCGMM]: Cohn-Gordon et al, On Ends-to-Ends Encryption: Asynchronous Group Messaging with Strong Security Guarantees, CCS 2018
[CCG]: Cohn-Gordon, Cremers and Garett, Post-Compromise Security, CSF 2016
[PR]: Poettering and Rössler, Asynchronous ratcheted key exchange, Crypto 2018
