9

I wonder how much data in total does a freshly installed vanilla Linux system (like 32-bit CentOS 5.10) read in order to get to a virtual console shell prompt? You know, reading all the configuration files, loading binaries, kernel image, etc.

I am booting 100s of Linux machines, and 95% of their system contents are identical and will remain identical - the machines are clones. I would like to offload the identical/read-only shared part of the filesystem, to an NFS storage, mount it from there, and boot like that. Only writeable part of the filesystem, like /var, /tmp and /home, will remain local to each machine. Since potentially hundred machines may boot simultaneously as part of a "cluster", I need to estimate whether the accessed NFS storage link will be a bottleneck while booting from.

I am looking for order-of-magnitude estimates. I am aware that Linux boot varies greatly with regards to details of the process. Are we talking 10Mb? 100Mb? 1Gb?

Improve this question
7
  • 5
    Why are you asking? Commented Mar 5, 2014 at 17:20
  • 2
    The variability is (can be) orders of magnitude between systems - loading the kernel & drivers is the tiniest fraction of the boot process, and the init scripts on a system can do literally anything before you get a login prompt. Please explain the situation you're dealing with in terms of an actual, practical problem that we can help you solve. Commented Mar 5, 2014 at 17:29
  • 1
    @amn Can you put the reason in your initial question? It'll help with context. Another reason people would ask a similar question is if they're using cycle-limited storage. More detail is always better. Commented Mar 5, 2014 at 17:47
  • 1
    You're looking at the vicinity of 20 - 50 MB: kernel + modules (~10, but can be more depending on hardware), glibc, additional basic libs, around dozen of executables). Commented Mar 6, 2014 at 1:43
  • 5
    I need to estimate... then do one and measure it. Commented Mar 6, 2014 at 12:44

2 Answers 2

Reset to default
8

Install one system, boot it and check out the block layer statistics from /sys/block/${DEV}/stat e.g. /sys/block/sda/stat.

Quoting from the documentation:

The stat file consists of a single line of text containing 11 decimal values separated by whitespace. The fields are summarized in the following table, and described in more detail below:

Name units description ---- ----- ----------- read I/Os requests number of read I/Os processed read merges requests number of read I/Os merged with in-queue I/O read sectors sectors number of sectors read read ticks milliseconds total wait time for read requests write I/Os requests number of write I/Os processed write merges requests number of write I/Os merged with in-queue I/O write sectors sectors number of sectors written write ticks milliseconds total wait time for write requests in_flight requests number of I/Os currently in flight io_ticks milliseconds total time this block device has been active time_in_queue milliseconds total wait time for all requests

read sectors, write sectors

These values count the number of sectors read from or written to this block device. The "sectors" in question are the standard UNIX 512-byte sectors, not any device- or filesystem-specific block size. The counters are incremented when the I/O completes.

You can use this one-liner to get the number of bytes more easily:

awk '{printf("read %d bytes, wrote %d bytes\n", $3*512, $7*512)}' /sys/block/vda/stat

Results for Scientific Linux 6.1 i386

I tested this on a KVM/qemu virtual machine running Scientific Linux 6.1 i386 (which is similar to RHEL). The following services were enabled: acpid, auditd, crond, network, postfix, rsyslog, sshd and udev-post. The swap is on a separate disk, so it's not taken into account.

The stats for 85 boots, taken remotely with SSH a couple of seconds after the login prompt appeared, were:

 Name Median Average Stdev ------------- ------ ------- ----- read I/Os 1920 1920.2 2.6 read merges 1158 1158.4 1.8 read sectors 85322 85330.9 31.9 >> read MiBytes 41.661 41.665 0.016 read ticks 1165 1177.2 94.1 write I/Os 33 32.6 1.7 write merges 64 59.6 7.4 write sectors 762 715.2 70.9 >> write MiBytes 0.372 0.349 0.035 write ticks 51 59.0 17.4 in_flight 0 0.0 0.0 io_ticks 895 909.9 57.8 time_in_queue 1217 1235.2 98.5

The boot time was around 20 seconds.

Improve this answer
1
  • 2
    Note that this only seems to give you the transfer demand (quantity), not the throughput demand (rate). You could divide by the uptime to get an average number though. Commented Mar 5, 2014 at 19:16
15

You say in your comments that you're evaluating a netboot / network root environment.

The first thing you must realize is there is no such thing as "vanilla" - you're not going to run CentOS 5.10 right out of the box with zero changes (if you think you are you're deluding yourself: NFS Root is already at least Strawberry, verging on Pistachio).

If you want an answer for your specific environment (which is what really counts) you're going to need to set up an NFS server and a client machine, boot it, and measure:

  1. The transfer (quantity)
  2. The throughput (rate)

Both values will be critically important for performance. You'll probably also want to set up several clients at some point and simulate normal use of the system to see what kind of steady-state demand they put on your NFS server / network when people are using the systems as they would in their everyday work.

See also: Our series on Capacity Planning - we don't talk specifically about NFS, but the general principles of "Build it, Test it, Stress it" apply.

Improve this answer
3
  • 1
    If there is vanilla icecream, there is vanilla Linux! ;-) Seriously though, it is a fairly unaltered CentOS 5.10, and whatever was altered is part of the writeable filesystem, that which won't be mounted from NFS, so it's not a factor - yes, there is a gigantic Postgres database in /var/lib but /var is not mounted from NFS but is on the local physical startup disk. And if I wanted to profile it, I wouldn't be asking the question here :-) Commented Mar 5, 2014 at 18:01
  • 10
    @amn I'm sorry you don't want to do profiling, but you gotta do what you gotta do - we can't pull applicable numbers out of our butts for you. Your solution (NFS root) is a sound, time-tested one, and honestly you can probably just deploy it blind with no problems (tens of thousands of Sun Microsystems environments were deployed blind like that back in the heyday of NFS-root & Netbooting Solaris & worked great). If you're concerned about performance though you'll need to do profiling to determine the demand and bottlenecks for your specific environment - that's just the way of the universe. Commented Mar 5, 2014 at 18:58
  • 1
    @voretaq7 Can't argue with the profiling argument, and never did. I just wanted the next best thing before I roll up my sleeves and set up the NFS. Thanks for your valuable input. Commented Mar 6, 2014 at 9:54

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.