On a linux server dedicated to running one application that processes financial transaction input data files that arrive every minute using sftp, what is the recommendation for writing out log files?
The logs are very detailed and therefore large, and are deleted or archived offline every 48 hours.
What is the difference in write performance if writing the log files to a dedicated partition on the main system disk vs writing them to a separate physical disk?
Establishing dedicated IO paths (separate disks) can be beneficial when you need guaranteed IO performance for specific tasks in your application workload.
Like you already suggested, writing logs to a different device can benefit the other aspects of the workload.
That was much more a valid strategy and requirement when enterprise workloads were limited to spinning disks. With how both the price , performance and write capacity of SSD / flash / NVME have evolved you can easily afford storage that will easily meet the majority of IO demands.
You’ll know when you start measuring IO performance if your application does indeed come close to maxing out the available IO performance of a single drive or not.
But you’re going to need real numbers not vague statements and only an intuition.
buffer=32k flush=1m to the access_log. This will reduce the amount of I/O since it will try to flush not each request directly, but buffers a bit. These kind of configurations might also reduce not only the I/O load on the disk but also increase performance and extend the life span of the disk.