First, do you really think that there are more updates than let us say, credit card transactions per seconds in the world, managed finally by only 2 or 3 companies? Those work, so it is a solvable problem :-)
Second, you might be confused on how updates are happening, because it is a non often understood part where two planes intersect: the registration plane and the publication plane. You might as well be confused by what exactly happens on registry nameservers (only maintaining the NS records for delegations of domains, and NOT the full content of those zones).
But before going into that you also seem to assume that updates should be real-time where they both don't need too and often aren't. Nothing is real time anyway in the DNS, because of the TTL.
So back to your two planes:
- registrars send updates to registries, when nameservers change and other operation that have DNS side effects (ex: setting up EPP
clientHold status); this is the registration plane, it is completely UNCORRELATED to DNS publication, and typically uses a protocol called EPP; when registry replies "update accepted" it absolutely does not mean it is published already on their DNS infrastructure, it is just a guarantee "it will be at some point"
- registries maintain the DNS publication plane, making sure their nameservers do publish the correct
NS records for all delegations, aka all domains registered under the TLDs they manage.
As such, the number of changes is probably far smaller that you have in mind: if a .com owner changes the content of its zone, with new records, in most cases, there is nothing to do through the registrar and nothing changes at the registry authoritative nameservers.
And those changes do NOT happen through DNS Update mechanism. Changes are pushed by registrars using a specific protocol, EPP, changes are stored somehow in some registry database, after which the new data is published by the registry on its authoritative nameservers.
You seem also to think "real-time" is mandatory. It is not, at least not technically, and may even by an ineffective design (consider if you want to do tests that the new changes make sense, as some registries are doing by checking that the name nameservers are correctly configured for resolution of the name they will soon be listed as authoritative for, or DNSSEC tests for example...).
A typical way of setup, used by many registries providing things like a "10-minutes delay for updates" or "1-hour", is to store in some internal buffer all changes requested in that given period of time, then once for all generate the new zone and publish it, while starting a new buffer to collect all the changes that will came in the next period.
I would assume that ISC's BIND is used at that level.
Not at all. Verisign runs its own proprietary nameservers software, called Atlas. See for example https://www.enterpriseappstoday.com/news/verisign-accelerates-dns.html
Note that in this 2004 article it is already said:
VeriSign Naming and Directory Services (VNDS) promises to update the core 13 .com and .net authoritative name servers in less than five minutes. The current rate is about twice per day.
Of course it improves since then. But I believe even once per 5 minutes to be largely good enough for any real practical use of the DNS.
There could be also contractual obligations, and specifically for gTLD registries and registrar that are all under contract with ICANN.
Current Verisign-ICANN contract is at https://www.icann.org/en/registry-agreements/details/com
You can find in §6.6 of the appendix at https://www.icann.org/en/registry-agreements/com/com-registry-agreement-appendix-7-1-12-2012-en details on the update constraints:
6.6 Update Frequency. The Registry Operator makes timely updates to the data on the DNS Name Servers and Whois. ICANN-Accredited Registrars record these updates through the SRS. The SRS then updates the DNS Name Server and the Whois. Registry Operator processes this updates on a near real time basis.
The committed performance specification with regards to Update frequency for both the DNS Name Server and the Whois is 3 minutes for 95% of the transactions during a Monthly Timeframe. That is, 95% of the updates to the DNS Name Servers and Whois during a Monthly Timeframe will be completed within 3 minutes. Update frequency is measured from the time that the Registry Operator confirms the update to the time the update appears in the DNS Name Server and Whois. Update frequency performance will be reported on a monthly basis to ICANN in accordance with Appendix 4.
Do note the usual mark at "95%" used for lots of SLA. So it is near real time when feasible, but in practice typically 3 minutes (hence the typical buffer setup described above).
I have a fairly clear picture on how someone would build scalable infrastructure using BIND. The part I am unclear on is how would someone design a system that can handle the addition, deletion, and change of hundreds of DNS zones every few seconds.
Verisign has only a couple of zones: .com, .net, some IDNs, etc. They don't manage "hundreds of zones". And certainly not hundreds of them with lots of changes every seconds.
You may/want to be more interested by DNS providers that do hosts million of zones with possible high frequency of updates. Here is one article from CloudFlare where they explain what they do performance wise regarding their authoritative DNS service: https://blog.cloudflare.com/how-we-made-our-dns-stack-3x-faster/
There are countless domain registrars in the world
No, not all. Far from countless. Less than 2000 in fact, and probably only like 500 really active with high volumes of changes. All registrars for gTLDs have to be accredited by ICANN. You can find the whole list of them at https://www.icann.org/en/accredited-registrars
Am I right to doubt that a TLD operator is using text files and constantly having BIND reload the changes every time a new .com is registered?
No sane high level of transactions nameserver software would be backed by text files. Even Bind isn't: when dynamic updates are enabled, you have a "journal" file (which is binary) and you specifically shouldn't edit the text file version of the zone (without first freezing updates and then allowing them again after the edit).
If so what do they do? Is it an SQL or LDAP database backed? does that even scale?
I doubt that either SQL or LDAP are good storage engines for DNS. Remember the DNS is hierarchical in nature, which poses various constraints.
