How can I understand the math behind crypto, especially that used in blockchains / distributed ledgers?

Honestly speaking, the amount and sophistication of cryptography used in DLTs depends a lot on which project you are studying. First of all, please note that blockchain/DLT has also become a marketing term today, so many projects out there are just nothing more than legacy databases with a cool name. Public, unpermissioned cryptocurrencies are another matter, and imho they are what you have to study if you are really interested in this brand new field. (But first check you really need DLT. You get many features and lose many others, so first check you really don't need just a DB.) If you haven't changed your mind, let's go on with just a bunch of examples and references :)

• In Bitcoin, you find standard pubkey/privkey signatures over elliptic curves and hashing (e.g., the core of PoW), both of which nowadays can be considered standard cryptography (nothing more than what you can find in a modern X.509 certificate of a web server domain). Much more challenging for a beginner are imho incentives and game-theory related concepts. A book I loved when I began my journey in this world is "Mastering Bitcoin 2nd. Ed." by Andreas Antonopoulos, which introduces the many concepts (the few cryptography ones as well) you need to master to dig into this world. But be careful to not consider Antonopoulos's book math-rigorous: as @kelalaka points out in comments, there are misleading problems with definition of EC scalar multiplication, in page 68 of first edition and 63 of second.
• Usually privacy-oriented coins use fancier cryptography. For example, in Monero you find Diffie-Hellman exchanges, Ring Signatures, and other cool ideas (e.g., Pedersen Commitments and Bulletproofs). I suggest you check https://www.getmonero.org/library/ : "Zero to Monero 2nd Edition", a complete reference of today Monero ("today" because hard forks, aka non back-compatible changes are common in Monero) with more math footprint than "Mastering Bitcoin". I think good companion documents for it are the cheatsheets you can find in the same web page (disclaimer: they are mine ;-) )
• The future of Ring Signatures seems to be Triptych and Seraphis on Monero side, and Lelantus Spark on Firo side (a "new" cryptocurrency, formerly Zcoin but with completely different protocols/math)
• Other very cool cryptographic stuff used in cryptocurrencies are Zero Knowledge Proofs; be careful because they have so many flavors. Bulletproof used in Monero is one of them, but for sure you will find ZK-SNARKs of Zcash, ZK-STARKs and ZK-Rollups for Ethereum's future... I avoid references at application level because they are a world into a world, and imho to really understand them you have to start from the foundations, for example "Foundations of Cryptography Volume 1" by Oded Goldreich and then dig into models/heuristics of non-interactiveness.... really a mess (this is just a simplified idea: https://github.com/AdamISZ/from0k2bp/blob/master/from0k2bp.pdf - btw it contains a very understandable intro to Commitments)