Score:1

Why are new writes to files not surviving a power loss

gl flag

I have a similar type of issue to: Raspberry Pi: Filesystem writes files, but after reboot the old data is back but not quite the same.

I am using Armbian_23.02.2_Orangepizero_jammy_current_5.15.93 running on an OrangePi with an SD card for storage.

I built a race timer for a car using C++. It uses GPS and logs data at 10Hz to a file. It also runs a server and I can change some configuration on a webpage that is hosted and this configuration is then stored as JSON in a file so that it survives reboots. The OrangePi is powered from the car and when the car is turned off after a run the OrangePi loses power until the next time the car is started (could be problematic?)

The problem I am having is this: I can launch the server, change some configuration, ssh on, cat the file and see that the new configuration is stored in the file. If I then pull the power, and plug it back in. The file returns to the state it was before the config was changed.

If I do sudo reboot now instead of pull the power the new configuration persists. It seems like the new file data isn't actually being written and held in RAM.

Storing data like this:

void store(const config& c)
{
    const nlohmann::json json = {
        {"key1", c.val1},
        {"key2", c.val1},
        {"key2", c.val1}
    };

    std::ofstream f("/path/to/file.json");
    f << json;
}

When logging data at 10Hz, I append to the end of the file with the new data at 10Hz, I have noticed that sometimes the end of the file is missing some "logs". Also if the current run is faster, the log file for that run will be copied to a different location, I have noticed that the copied data can be "funky". For example I have seen it contain parts of the previous and new log and have the previous file name.

Copying files like:

std::filesystem::copy_file(
    from_path,
    to_path,
    std::filesystem::copy_options::overwrite_existing,
    ec);

Is there anything I can do to force the files to be written straight away?

Is appending roughly 100 characters to a file at 10Hz possibly too ambitious for a single-board computer running on an SD card?

marcelm avatar
cn flag
_"... when the car is turned off after a run the OrangePi loses power until the next time the car is started (could be problematic?)"_ - Definitely problematic. Computers are not designed to reliably survive power loss. You may lose some data, as you already found, but you also risk filesystem corruption. OSses have some safeguards against this, so most of the time you'll be fine, until that one time it craps itself. Ensure battery backup and proper shutdown.
Dreamer avatar
nl flag
A slightly different approach: since an OrangePi is drawing only limited amounts of power, you may be able (1) to use a relatively small rechargeable battery as a kind of uninterruptible power supply (USP) that keeps the OrangePi running even after there's no longer any power from the car, and (2) use the car power loss as a signal to trigger a clean shut down of the OrangePi. That way, you do not need to manually `sync` and still get all data cleanly written to disk.
av4625 avatar
gl flag
@Dreamer I think this is the best long term answer, as it really solves all problems and makes reliability better. I will need to research the best way to do this in a small package and redesign my PCB
rfm avatar
mk flag
rfm
What you want to do is frequent fsync() calls on the file as you're writing it. This is less drastic that a "sync", since it only flushes that one critical file, not everything. Might not need to do it 10 times a second, maybe once a second? I wouldn't worry too much wearing out the SD card, but you should make backups frequently (daily?) and consider simply replacing the SD card on a schedule.
Score:9
om flag

The problem I am having is, I can launch the server, change some configuration, ssh on, cat the file and see that the new configuration is stored in the file. If I then pull the power, and plug it back in. The file returns to the state it was before the config was changed.

All modern operating systems cache data before writing it to disk, to optimize performance. Disks are slow compared to memory, and modern OS doesn't crash often enough to warrant synchronous writes.

If I do sudo reboot now instead of pull the power the new configuration persists. It seems like the new file data isn't actually being written and held in RAM.

Correct. And reboot flushes caches to disk. You can run sync to do the same; it forces the OS to write data held in caches to disk.

Is appending roughly 100 characters to a file at 10Hz possibly to ambitions for a single board computer running on an SD card?

The computer will be fine doing it. But you may not want to do it, to ensure longevity of storage medium, as flushing forces the storage media to rewrite an entire block, which can be detrimental on flash memory.

av4625 avatar
gl flag
`sync(2)` works like a charm and ensures that the data is actually written to disk, thank you!
Criggie avatar
za flag
I have an early pi4 doing backup stuff to external disk, and it will eat a 16 GB micro SD card in about 3 years. Symptom is the SD accepts writes and they "complete" ok but nothing on disk changes. At that point the SD is dead and needs replacing.
br flag
You can also alter some kernel tunables that control *how long* dirty data is allowed to hang around before it gets written out; if you make that time period shorter then you have a shorter window for data loss, at the cost of some performance for processes that want to do writes.
av4625 avatar
gl flag
@Criggie I have seen that there are a few OrangePi's with onboard storage, I will have to look into these to see if they are possibly more reliable and if there is a small version with onboard storage and 2 serial ports
av4625 avatar
gl flag
@vidarlo Above I pointed out I wanted to do some research to see if its more reliable than an SD card as Criggie said his project was eating SD cards, maybe its not but I'm not sure
av4625 avatar
gl flag
@vidarlo if you follow the comment chain, you'll see that I said `sync` solved my initial issue, and you are right that initial issue has nothing to do with the type of disk. But in my question I asked if this amount of writing to disk was ambitious for an SD card. Comments in this chain are relating to this and I simply said I need to do research to see if onboard storage would be more reliable. I don't get your problem, everyone else has been helpful
Score:6
ru flag

You can use the sync command as root. sync synchronizes cached writes to persistent storage. You can type man sync to get some help on its command line switches if you need something more specific.

Score:3
cn flag

You should also make it do an orderly shutdown though. This is often accomplished by using one of the GPIOs to input a shutdown signal, and enough power storage to allow the RPi to act on the signal. You could, for example, power it off a USB battery pack that's charged by the car (ensuring you buy one that will charge and deliver power at once - not all do, including some of the better makes). The GPIO would be driven (I suggest via an opto-isolator) from the 12V ignition on line of the car.

Because it can't monitor the GPIO while it's shut down, you can't start it in the same way. Flipping a switch in the power line is quite an easy way to get it to reboot.

There are also solutions using relays, and no battery beyond the car's own. For example:

  • power the RPi's PSU off an always-on 12V line.
  • but that line is routed through a relay
  • the relay coil is powered, through a diode, by the 12V ignition on line
  • the coil is also powered, again through a diode, by a transistor that's switched by a GPIO (the GPIO will go low on shutdown). Again, I'd use an opto-isolator here.
  • You still sense the 12V ignition on line using a GPIO, opto-isolated.

This setup means that if either the ignition on signal is 12V OR the GPIO driving the relay is high, power is supplied. That means it will start automatically when you start the ignition

I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.