I have been trying to figure out why requesting an access token through OpenStack Keystone is relatively slow (lowest is about 400 ms to in some cases more than a second).
To get down to the cause I used OSProfiler and it seems most of the time is spend in talking to the database. I can see Keystone doing about a hundred database calls when requesting a token. The individual calls to the database are relatively quick (between 2-6 ms) but doing that times 100 can easily result in 500 ms of latency just in database requests.
Right now the database is accessed via the network (there's about 0.3 ms in network roundtrip times). The time for each database query makes sense when looking at TCP and MySQL overhead so isn't that crazy.
The main thing I'm wondering about is whether this is expected behaviour, is it normal for Keystone to do this many queries to request a token. Caching is enabled and I do know caching is working as I can see Keystone using the database and when debugging the caching library it does in fact indicate it gets hits when reaching out to the cache.
Right now the only option I see is to completely eliminate the network and move the database to the same machine as the Keystone API.
Update: This is on Ussuri using Fernet tokens
