Linux script with logfile that changes names

How can I rename a log or error file to prevent overwriting? I have a bash script that produces an output log file. When re-running the script I want the old log files to be preserved by changing the name of the every logfile depending on the name of the last file created. For example:

1st run: script.sh > log.file produces log.file1 2nd run: script.sh > log.file produces log.file2 3rd run: script.sh > log.file produces log.file3 Basically every run needs to check the last log file and count one up.

Another (better?) way to do this is to include the date and time in the name of the log file. That gives you a unique name every time which is more informative than just a number, and it doesn't depend on checking for a pre-existing file.

Example: output the data to a file named "$(date +"%Y_%m_%d_%I_%M_%p").log"

in bash:

outputfilename="$(date +"%Y_%m_%d_%I_%M_%p").log" && generatelogstuff > "$outputfilename"

See: https://unix.stackexchange.com/questions/278939/how-do-you-put-date-and-time-in-a-file-name

One option (that doesn't require explicit calculation of a new file name each time) would be to write to a temporary log file, then copy or move it with the --backup=numbered option:

$ tmplog=$(mktemp)
$ ./script.sh > "$tmplog" && cp --backup=numbered "$tmplog" log
$ ./script.sh > "$tmplog" && cp --backup=numbered "$tmplog" log
$ ./script.sh > "$tmplog" && cp --backup=numbered "$tmplog" log
$ ./script.sh > "$tmplog" && cp --backup=numbered "$tmplog" log
$ rm "$tmplog"

resulting in an automatically numbered sequence of log files,

$ ls -1tr log*
log.~1~
log.~2~
log.~3~
log

Only if,

either you know for sure that the existing log.file... filenames numbering start with 1 i.e. log.file1 and follow a sequential order i.e. log.file1 log.file2 log.file3 … etc.

or no files with this naming pattern exist yet.

and no directories named log.file... exist in ether cases in your current working directory, then you can use something like this(in bash):

$ f=(log.file*); [ -e "${f[-1]}" ] && lf="log.file$((${#f[@]}+1))" || lf="log.file1"
$ ./script.sh > "$lf"

• f=(log.file*) will read existing filenames pattern and store them in an array f.
• [ -e "${f[-1]}" ] will check if the last item in the array is an existing file.
• In case log.file.. files exist, lf="log.file$((${#f[@]}+1))" will increment the new log.file number based on the (array length+1).
• In case no log.file.. files exist, lf="log.file1" will set the filename to log.file1.

I am new to this stackexchange group, and apparently I'm not allowed to comment, so I will have to phrase this as an answer:

Several of the solutions I see at the time I am posting this have race conditions that can cause problems.

E.g. incrementing a counter: more than one instance of the script may be running between the time you check -f $filename$i and the time you create such a file name if it does not already exist. In this instance since you are using >, one instance might be lost.

E.g. Placing a Timestamp in the filename: similarly, two instances may be running at precisely the same time, whatever granularity your time Sam indicates.

E.g. write into a temporary file name that is the same, and then moving it to a versioned file name… Similar problems.

Probably the most standard solution is to embed the process ID in the file name - eg it might look like base.seq#.timestamp.pid. Most UNIXes require the PID to be unique. ... until it gets recycled (unlikely). ... or if you are accessing a network file system, from separate OS instances that do not have the same process ID space. Add hostname as a bandaid? ... but host names are not necessarily unique, not even with full domains, nor are IP addresses with NAT or MAC numbers.. Even on a single system, sometimes people use their own user level cooperative multithreading libraries that the OS is not aware of... and, yes, some of them will permit you to do system calls.

The most general solution is something that removes the window/race condition: something that allows you to test that a filename$i already exists, and returns to a newly opened file descriptor if it does not. Atomically. But even if your OS supports that, nit all filesystems do. (One of the reasons why the "rename" approach is popular is that the rename() syscall when it was eventually added had better guarantees in this regard.)

Anyway, I would rather have made this a comment, because right now I cannot recall if there is a single BKM for UNIX families and in ubuntu in particular. As an old fogey I know that this has been a long-standing source of bugs, but I don't know if it has been completely resolved, particularly for the most general network and cloud file systems.

But having pointed out that there can be dragons here, I can backpedal and say that you can frequently get away with some of the solutions suggested by the answers that triggered my multiprocessor red flags, esp filename.timestamp.PID, in shell scripts if you know things about the environment, such as you are not running on a network filesystem. And especially if you know that you don't need to worry about DIY multithreading packages, only OS standard multithreading.

I hope that somebody will correct me by providing the guaranteed always to work on all Ubuntu / UNIX-like systems code sequence... with or without constraints wrt network filesystem or cloud file system or FUSE file system or…