12

I have some embedded boards running American Megatrends bios with embedded linux as the OS. The problem I have is that the industrial flash ide's will be corrupted on power loss. I have them formatted as ext4. Whenever this happens, I can usually fix the flash with fsck, but this will not be possible in our deployments. I have heard that disabling the write-caching should help, but I can't figure out how to do it. Also, is there any thing else I should do?

More Info

The drive is a 4gb ide flash module. I have one partition which is ext4. The O.S. is installed on that partition and grub is my bootloader.

fdisk -l shows /dev/sda as my flash module with /dev/sda1 as my primary partition.

After a power loss I usually cannot make it entirely through the boot init scripts.

When I mount the drive on another P.C. I run fsck /dev/sda1. It always shows messages like 

"zero datetime on node 1553 ... fix (y)?"

I fix them and it boots fine until the next power loss.

When I get to the office tomorrow, I will post the actual output of fdisk -l

This is all I know about how the system works. I am not a systems guy, I am a Software Engineer that has a habit of getting into predicaments that are outside of his job description. I know how to format drives, install a bootloader, write software, and hack on an operating system.

Here is the output from dumpe2fs

#sudo dumpe2fs /dev/sda1 dumpe2fs 1.41.12 (17-May-2010) Filesystem volume name: VideoServer Last mounted on: / Filesystem UUID: 9cba62b0-8038-4913-be30-8eb211b23d78 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize Filesystem flags: signed_directory_hash Default mount options: (none) Filesystem state: not clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 245760 Block count: 977949 Reserved block count: 48896 Free blocks: 158584 Free inodes: 102920 First block: 0 Block size: 4096 Fragment size: 4096 Reserved GDT blocks: 239 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 8192 Inode blocks per group: 512 Flex block group size: 16 Filesystem created: Fri Feb 4 15:12:00 2011 Last mount time: Sun Oct 2 23:48:37 2011 Last write time: Mon Oct 3 16:34:01 2011 Mount count: 2 Maximum mount count: 26 Last checked: Tue Oct 4 07:44:50 2011 Check interval: 15552000 (6 months) Next check after: Sun Apr 1 07:44:50 2012 Lifetime writes: 21 GB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 28 Desired extra isize: 28 Default directory hash: half_md4 Directory Hash Seed: 249d2b79-1e20-49a3-b324-6cb631294a63 Journal backup: inode blocks
Improve this question

2 Answers 2

Reset to default
7

The write cache has usually nothing to do with the BIOS, mostly there is no option for switching disk cache settings in there. With linux, using hdparm -W 0 should help.

The setting is persistent, so if you don't have hdparm to play around with in your production systems, you should be able to disable the disk write cache on a different system and replug the disk.

BTW: I'd second the idea of a non-writable root filesystem (so your system could boot in a kind of "recovery mode" and allow for remote access even if the writable filesystem is not mountable for some reason). And if you can change the hardware design, consider using mtd devices instead of IDE/SATA disks with a flash-aware filesystem like jffs2. We've been using this combination with several embedded devices (mostly VPN router solutions in the field) for several years with good results.

Update: the root of your problem seems to be that you are running an ext4 filesystem with journaling disabled - has_journal is missing from the Filesystem features list. Just shut down all services, check if anything still has open files using lsof +f -- /, remount your root partition read-only with mount -o remount,ro /, enable the journal with tune2fs -O has_journal /dev/sda1 and set up the "ordered" journal mode as the default mount option using tune2fs -o journal_data_ordered /dev/sda1 - you will have to re-run fsck (preferably from a rescue system) and remount root / reboot after this operation.

With these settings in place, the metadata is guaranteed to be recoverable from the journal even in the event of a sudden power failure. The actual data is also consistently written to disk, although you may see data of several seconds before the power outage lost on bootup. If this is not acceptable, you might consider using the tune2fs -o journal_data /dev/sda1 mount option with your filesystem - this would include all data written to disk in the journal - this obviously would give you better data consistency but at the cost of a performance penalty and a higher wear level on your SSD.

Improve this answer
12
  • So is the write cache my problem or something else? Commented Oct 3, 2011 at 23:14
  • Well, how should I know, it's your system after all :-) You should give some details on the file system mount options used (did you enable extents? what kind of data / journal mode?) and the kind of corruption you're seeing (fsck output would be best) for a more detailed analysis. Commented Oct 3, 2011 at 23:23
  • OK, thanks. I am a a helpless software engineer you know :). I'll get some details. I am adding some details within the minute. Commented Oct 4, 2011 at 1:19
  • I don't know what extents are and I am not sure what a Journal mode is. Commented Oct 4, 2011 at 1:32
  • Ah, I see. Just post the first lines of the output of dumpe2fs /dev/sda1 (or whatever your device/partition name for this system would be) - they should contain all relevant information. And the mount options for the root filesystem from /etc/fstab should help as well. Commented Oct 4, 2011 at 11:56
5

The write cache suggestion is a good start but this sounds like an architectural design flaw. On an embedded system the internal flash should probably NOT be mounted R/W except in rare circumstances. You should really be doing most of the work in a memory filesystem and syncing changes back to the RW flash upon some user command or regular interval. It is really uncommon for an embedded system to use a regular filesystem (like ext4) in rw mode during normal operation. If there is some application requirement where you need lots of storage space you should consider having your system partition be different and designing it such that the data partition can be fsck -y'ed as part of startup.

If you need some starting points I would look at how people setup Diskless Linux systems:

http://frank.harvard.edu/~coldwell/diskless/

and start from there. The general idea is that your system binaries and data can be mounted read-only so your filesystem won't be corrupted. However you need to be able to write to certain areas, so you need something to usually memory filesystem /tmp, /var/tmp. Even if certain things need to be writable you just create a script to mount the partition as r+w and then commit the changes, then go back to read-only.

A really great example of this is the Cyclades hardware, its embedded linux and whenever you make configuration changes you have to execute a save script which actually rebundles the configs and writes them out to the flash.

Improve this answer
6
  • There are configuration files that need to be edited by the application as well as the /etc/networks and the hostname file. Could you give me a recommendation i.e. something like, you need one partition with such and such type and another for your config files of another type and so on? I really have no idea about these things. I write software and am magically expected to know exactly (not that I don't know enough to write *nix software, but I certainly don't know as much as a dedicated systems guy) how the hardware should work by my employer. Commented Oct 4, 2011 at 2:23
  • Sure, I updated the answer to include some more info. This is a pretty complex topic to cover in one question though since it deals with so many linux internals. You might want to try and contract someone who has done diskless/pxe/embedded systems before to understand your application requirements and architect a solution that would be reliable. Commented Oct 4, 2011 at 2:30
  • Worst case, you can use a system partition (never writable) and two configuration partitions. If the primary partition is unreadable or incomplete, boot from the secondary, reformat the primary, and copy the secondary into it. Update the primary and secondary in non-overlapping operations. Commented Oct 4, 2011 at 4:36
  • Ok, I updated my answer. I will probably take your advice and take this to an old professor of mine from my graduate program. In the meantime, is there a quick and dirty that will at least get me in a better position that doesn't include my ass in a frying pan? Commented Oct 4, 2011 at 14:45
  • Turning off write caching or running 'sync' on a regular basis would probably help in the short term. Commented Oct 4, 2011 at 21:24

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.