I/O Read overkill

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

I/O Read overkill

Fabien Wernli
Hi,

Yesterday I migrated my dual-xeon 4GB RAID1 rrd to a quad-opteron 8GB
RAID10 server, in the hope to reduce the load.
I am managing 18k RRDs, each being updated randomly every ~10minutes
over ssh, which makes roughly 30 file updates/s.

I had many performance problems with the new server, which I was able to
manage (=server stays alive) using xfs instead of ext3, raid10 instead of
raid5. Underlying disk technology is the same (SCSI 10k drives atatched to
LSI 1030 controller), but box brand is different.

Graphs speak for themselves.
http://swissunix.free.fr/rrd/rrd_perf.png
Before: 294% iowait
        4.9MB/s read (hardware raid1 hidden to linux)
        1.1MB/s write
        4.6 loadavg

After:  318% iowait
        57.9MB/s read (linux raid10 -> values have to be div by 2)
        2.3MB/s write
        8.9 loadavg

tests made with rrdtool 1.2.10 and 1.2.19.

Now where the heck do those 57.9/2 = 28.95MB/s READ come from???
I do not understand how the Read IOs were multiplied by a factor of 6!!!


Reply | Threaded
Open this post in threaded view
|

Re: I/O Read overkill

Fabien Wernli
On Fri, Oct 26, 2007 at 09:47:38AM +0200, Fabien Wernli wrote:

> Before: 294% iowait
>         4.9MB/s read (hardware raid1 hidden to linux)
>         1.1MB/s write
>         4.6 loadavg
>
> After:  318% iowait
>         57.9MB/s read (linux raid10 -> values have to be div by 2)
>         2.3MB/s write
>         8.9 loadavg
>
> tests made with rrdtool 1.2.10 and 1.2.19.

I forgot to mention that old server was a 2.4 32bit kernel, new one being a
2.6 64bit.

also, I just tested rrdtool 1.3beta (w/mmap) and my problem seems to
disappear totally. I changed "use lib" in update script to use rrdtool 1.3b
at 12:25 (see graph).

http://swissunix.free.fr/rrd/rrd_perf_1.3beta.png

Now I don't understand why on earth it used to work so well on the old
server and rrdtool 1.2.10 ...


Reply | Threaded
Open this post in threaded view
|

Re: I/O Read overkill

Jason Fesler
> Now I don't understand why on earth it used to work so well on the old
> server and rrdtool 1.2.10 ...

I wonder if it is the defaults for readahead.  "man blockdev" and search
for readahead.   You might read what your defaults were on the old kernel;
and temporarilly apply them to your drives under the new kernel, and see
what happens with 1.2.10.



Reply | Threaded
Open this post in threaded view
|

Re: I/O Read overkill

Fabien Wernli
Hi,

On Fri, Oct 26, 2007 at 05:53:33AM -0700, Jason Fesler wrote:
> I wonder if it is the defaults for readahead.  "man blockdev" and search
> for readahead.   You might read what your defaults were on the old kernel;
> and temporarilly apply them to your drives under the new kernel, and see
> what happens with 1.2.10.

yeah I was planning on doing that, however now my production system is
up and running again so... ;)

Here are the values before (RRDs on sda):

# /sbin/blockdev --getra /dev/sda
120

and after (RRDs on md10 raid1+0):

# /sbin/blockdev --getra /dev/sd[b-e] /dev/md[01] /dev/md10
256
256
256
256
256
256
512

(siectorsiwe is 512 in both cases)

So I went from 120 to 512. Does that mean 4.266 times more data was being read
off the drive? This doesn't account for the difference though, as I went
from average read of 730KB/s to 36.2MB/s (sorry my previous values were
wrong I mistook Avg for Max), which is a factor of more than 50(!). So
unless the response is non-linear, I guess I have to look elsewhere...
I still suspect a 64bit issue, what do you think?

Now that I use rrdtool 1.2.99999[...] the read value dropped even further:
84KB/s, which is consistent with the announcement by the author (10x more
updates per second).


Reply | Threaded
Open this post in threaded view
|

Re: I/O Read overkill

Jason Fesler
> So I went from 120 to 512. Does that mean 4.266 times more data was being read
> off the drive?

Yes, but only when the OS decided you were doing sequential enough reads
to gamble doing so.

> Now that I use rrdtool 1.2.99999[...] the read value dropped even further:
> 84KB/s, which is consistent with the announcement by the author (10x more
> updates per second).

I haven't gone to this yet (I'm no longer responsible for the ~250 ide
disks running rrd..) but it it sure does sound promising.