This is what the manual says about the file:
One of the main activities of the chronyd program is to work out the rate at which the system clock gains or loses time relative to real time.
Whenever chronyd computes a new value of the gain or loss rate, it is desirable to record it somewhere. This allows chronyd to begin compensating the system clock at that rate whenever it is restarted, even before it has had a chance to obtain an equally good estimate of the rate during the new run. (This process can take many minutes, at least.)
You don't need a drift file. A drift file will allow Chrony to figure out how the local clock behaves, and discipline local clock even without any reference. It's a tradeoff. Having it in memory would mean that after a reboot the local clock will not be as accurate.
Ultimately it depends on how accurate you need time to be. Do you use it to know when TLS certs expire? If so, five minutes probably doesn't matter to you. Do you use it to timestamp bullet impacts to triangulate the sound? If so, accuracy is vital. How often do you expect your device to be powercycled? In short: You have to decide based on your application.
