You should enter the fully qualified host name for the name server.
This is because the delegation is made through NS DNS RR, which accepts a name, not an IP. The form you are filling is supplied as the data field of this record.
The registrar essentially adds something like the following records into the parent zone (which is com in this example):
example.com. NS alpha.example.com.
example.com. NS beta.example.com.
example.com. NS gamma.example.org.
alpha.example.com. A 192.0.2.66
alpha.example.com. AAAA 2001:db8::6a:7bc:8def
beta.example.com. A 203.0.113.77
Exactly same records should exist in your zone too (and the zone must be the same on all servers, in the sense that all of them should give the same replies to the same queries).
The DNS standard requires that each NS RR should contain exactly the name of the authoritative server. I used names alpha.example.com, beta.example.com and gamma.example.org for the example, someone might use just ns1.example.org and so on, this is arbitrary. Registrar doesn't know which names you have chosen for your name servers, so you must always tell them those names.
The additional glue records are only required for the names of authoritative servers that live within the delegated zone itself, as are first two servers in the example. This is the case when you need to specify IP and IPv6 addresses of name servers in addition to, but not instead of host names. If some (or all) servers your zone is going to be delegated to have names not within the zone itself, you don't specify any IP or IPv6 addresses for them at all, only names, because those addresses are specified elsewhere (in their own zone). This is the case of gamma.example.org.
Only specifying the IP addresses alone is never sufficient, because without the name the registrar is unable create the required DNS record for the delegation in the parent zone. Honestly, just as Patrick mentioned in the comment, I doubt you were able to only specify IP addresses and never entered any names.
Theoretically, it is possible to "infer" the names of name servers, if they are already configured to serve this to-be-delegated zone and correctly replying at the moment of filling the form, so the registrar can send a query to the specified IPs to retrieve the list of NS records, but nobody requires that registrars should implement such heuristics. This isn't what you should expect to always work. I've never seen registrars that do something like this.
