Inquiring on how to ease disk io (linux).

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Inquiring on how to ease disk io (linux).

Raimund Berger

Hi

I'm currently evaluating collectd on linux for monitoring (with
rrdtool backend of course), and while the default 10s update interval
is pretty cool, as the number of monitored systems increases the
server's disks start to rattle noticably. Quite understandably so I
guess, as rrdtool isn't meant to provide (sequential disk) caching
solutions like sql stuff usually sports.

Right now, I'm testing a workaround by stack mounting a tmpfs over the
real rrd database directory via aufs, which pleasantly resolves the
disk io issue. It has drawbacks though:

* as data is written only to the tmpfs, all of it written since the
  last mount/disk copy is going to be lost on crash/power outage
* those stacking fs (aufs, unionfs) apparently provide no means to
  automatically sync back data at specified intervals.

That's at least how I now see it, so my solution is not yet completely
satisfactory.

Question hence, did anybody here already run into and maybe tackle a
similar issue in a better way? If so, I'd sure appreciate some hints.

Thanks, R.


Reply | Threaded
Open this post in threaded view
|

Re: Inquiring on how to ease disk io (linux).

Jason Fesler
> Question hence, did anybody here already run into and maybe tackle a
> similar issue in a better way? If so, I'd sure appreciate some hints.

http://www.usenix.org/event/lisa07/tech/full_papers/plonka/plonka_html/index.html

And you might look at the 1.3 series rrdtool.


Reply | Threaded
Open this post in threaded view
|

Re: Inquiring on how to ease disk io (linux).

Raimund Berger
Jason Fesler <[hidden email]> writes:

>> Question hence, did anybody here already run into and maybe tackle a
>> similar issue in a better way? If so, I'd sure appreciate some hints.
>
> http://www.usenix.org/event/lisa07/tech/full_papers/plonka/plonka_html/index.html

Very nice. I had a feeling this must have been investigated before,
albeit not necessarily to that depth.

> And you might look at the 1.3 series rrdtool.

Actually, I did partly and already noticed those 'advice's around the
mmap code. While testing I've been relying on my distro's packages
though, until I have a clear idea on a reasonable upgrade path.

So right, the collectd daemon I've been running was linked against 1.2.

Thanks a lot, looking forward to checking that stuff out,
R.


Reply | Threaded
Open this post in threaded view
|

Re: Inquiring on how to ease disk io (linux).

Fabien Wernli
In reply to this post by Raimund Berger
Hi,

On Wed, Jun 25, 2008 at 05:06:29AM +0200, Raimund Berger wrote:
> is pretty cool, as the number of monitored systems increases the
> server's disks start to rattle noticably. Quite understandably so I

How many systems do you monitor?


Reply | Threaded
Open this post in threaded view
|

Re: Inquiring on how to ease disk io (linux).

Ronan Mullally
In reply to this post by Raimund Berger
Hi Raimund,

A few suggestions that I've tried on different systems in the past:

 - Upgrading your disk system is an obvious choice - more disks
   RAID 1 (or 0 if you live dangerously) and a caching RAID controller
   will help definitely help.

 - Tweaking your underlying filesystem to take advantage of a RAID
   filesystem can make a difference - I've seen big improvements using
   the stride-size and stripe-width options to mke2fs (though that was
   on a mail spool, not RRD files).

 - Tweaking some of the kernel attributes in /proc/sys/vm will change
   when and how often data cached by the OS will be written to the disk
   (or the cache in the disk controller).  I found this helped slightly
   on a filesystem with about 5,000 RRDs being hit every 5 minutes.

 - Tweaking the disk scheduler parameters in /sys/block/<disk>/queue can
   improve things a bit, depending on your traffic profile.

I hope these are useful.  The last 2 can be tried on the fly.  The others
require downtime.


-Ronan

On Wed, 25 Jun 2008, Raimund Berger wrote:

>
> Hi
>
> I'm currently evaluating collectd on linux for monitoring (with
> rrdtool backend of course), and while the default 10s update interval
> is pretty cool, as the number of monitored systems increases the
> server's disks start to rattle noticably. Quite understandably so I
> guess, as rrdtool isn't meant to provide (sequential disk) caching
> solutions like sql stuff usually sports.
>
> Right now, I'm testing a workaround by stack mounting a tmpfs over the
> real rrd database directory via aufs, which pleasantly resolves the
> disk io issue. It has drawbacks though:
>
> * as data is written only to the tmpfs, all of it written since the
>   last mount/disk copy is going to be lost on crash/power outage
> * those stacking fs (aufs, unionfs) apparently provide no means to
>   automatically sync back data at specified intervals.
>
> That's at least how I now see it, so my solution is not yet completely
> satisfactory.
>
> Question hence, did anybody here already run into and maybe tackle a
> similar issue in a better way? If so, I'd sure appreciate some hints.
>
> Thanks, R.
>
> _______________________________________________
> rrd-users mailing list
> [hidden email]
> https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Inquiring on how to ease disk io (linux).

Raimund Berger
In reply to this post by Fabien Wernli
Fabien Wernli <[hidden email]> writes:

> Hi,
>
> On Wed, Jun 25, 2008 at 05:06:29AM +0200, Raimund Berger wrote:
>> is pretty cool, as the number of monitored systems increases the
>> server's disks start to rattle noticably. Quite understandably so I
>
> How many systems do you monitor?

Been testing with up to 100 clients.


Reply | Threaded
Open this post in threaded view
|

Re: Inquiring on how to ease disk io (linux).

Fabien Wernli
On Wed, Jun 25, 2008 at 11:59:36AM +0200, Raimund Berger wrote:
> Been testing with up to 100 clients.

then my first advice will be (I have 2500 clients) to cache locally on each
client and send the rrd info once in a while.


Reply | Threaded
Open this post in threaded view
|

Re: Inquiring on how to ease disk io (linux).

Raimund Berger
In reply to this post by Ronan Mullally
Ronan Mullally <[hidden email]> writes:

> A few suggestions that I've tried on different systems in the past:
>
>  - Upgrading your disk system is an obvious choice - more disks
>    RAID 1 (or 0 if you live dangerously) and a caching RAID controller
>    will help definitely help.
>
>  - Tweaking your underlying filesystem to take advantage of a RAID
>    filesystem can make a difference - I've seen big improvements using
>    the stride-size and stripe-width options to mke2fs (though that was
>    on a mail spool, not RRD files).
>
>  - Tweaking some of the kernel attributes in /proc/sys/vm will change
>    when and how often data cached by the OS will be written to the disk
>    (or the cache in the disk controller).  I found this helped slightly
>    on a filesystem with about 5,000 RRDs being hit every 5 minutes.
>
>  - Tweaking the disk scheduler parameters in /sys/block/<disk>/queue can
>    improve things a bit, depending on your traffic profile.
>
> I hope these are useful.  The last 2 can be tried on the fly.  The others
> require downtime.


Good hints. Up to now I've been investigating mainly application local
tweaks (like that rather promising RRDCache approach), but system wide
tweaks might be a nice alternative/addon too, albeit of course leading
to tradeoff situations.

Thanks a lot for the input,
R.


Reply | Threaded
Open this post in threaded view
|

Re: Inquiring on how to ease disk io (linux).

Raimund Berger
In reply to this post by Fabien Wernli
Fabien Wernli <[hidden email]> writes:

> On Wed, Jun 25, 2008 at 11:59:36AM +0200, Raimund Berger wrote:
>> Been testing with up to 100 clients.
>
> then my first advice will be (I have 2500 clients) to cache locally on each
> client and send the rrd info once in a while.

Hm, good point, but is there already a publicly available solution for
this approach?  Sounds to me like distributing some home grown scripts
over the network ...


Reply | Threaded
Open this post in threaded view
|

Re: Inquiring on how to ease disk io (linux).

oetiker
Administrator
In reply to this post by Ronan Mullally
Hi Raimund,

Today Ronan Mullally wrote:
[disk performance config tips]

We have just successfully integrated a ssd disk into our ext3
filesystem to house the journal. The performance gain for write
heavy applications was about 100%, but the biggy was that the
responsivness of all the system became much better even under heavy
writing load, I would assume that this could help a rrd setup as
well ...

http://insights.oetiker.ch/linux/external-journal-on-ssd.html

cheers
tobi



--
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten
http://it.oetiker.ch [hidden email] ++41 62 775 9902


Reply | Threaded
Open this post in threaded view
|

Re: Inquiring on how to ease disk io (linux).

Fabien Wernli
In reply to this post by Raimund Berger
On Wed, Jun 25, 2008 at 12:31:09PM +0200, Raimund Berger wrote:
> Hm, good point, but is there already a publicly available solution for
> this approach?  Sounds to me like distributing some home grown scripts
> over the network ...

I have some trivial shell scripts if you're interested
it's only just a matter of storing your values in memory and sending them
over as usual but every now and then


Reply | Threaded
Open this post in threaded view
|

Re: Inquiring on how to ease disk io (linux).

Bernard Li
In reply to this post by Raimund Berger
Hi Raimund:

On Tue, Jun 24, 2008 at 8:06 PM, Raimund Berger
<[hidden email]> wrote:

> Right now, I'm testing a workaround by stack mounting a tmpfs over the
> real rrd database directory via aufs, which pleasantly resolves the
> disk io issue. It has drawbacks though:
>
> * as data is written only to the tmpfs, all of it written since the
>  last mount/disk copy is going to be lost on crash/power outage
> * those stacking fs (aufs, unionfs) apparently provide no means to
>  automatically sync back data at specified intervals.

I am not too familiar with stack mounting -- how is your solution
different from putting all the rrds in tmpfs?

I was thinking it would be ideal to put all the rrds in a hybrid
tmpfs/regular fs where data is written to tmps and periodically synced
to the regular fs.  Most folks who put rrds in tmpfs have some cronjob
to do this incase of server crash, but ideally this could be dealt
with by some fs hooks.  Maybe using inotify or something like that.

You might also be interested in this thread over at rrd-developers:

http://www.mail-archive.com/rrd-developers@.../msg02284.html

Cheers,

Bernard


Reply | Threaded
Open this post in threaded view
|

Re: Inquiring on how to ease disk io (linux).

Bernard Li
In reply to this post by oetiker
Hi Tobi:

On Wed, Jun 25, 2008 at 4:14 AM, Tobias Oetiker <[hidden email]> wrote:

> We have just successfully integrated a ssd disk into our ext3
> filesystem to house the journal. The performance gain for write
> heavy applications was about 100%, but the biggy was that the
> responsivness of all the system became much better even under heavy
> writing load, I would assume that this could help a rrd setup as
> well ...
>
> http://insights.oetiker.ch/linux/external-journal-on-ssd.html

Isn't it somewhat risky to put the journal of a RAID on a single SSD?
What if the SSD fails, wouldn't that cause data corruption?

Cheers,

Bernard


Reply | Threaded
Open this post in threaded view
|

Re: Inquiring on how to ease disk io (linux).

Raimund Berger
In reply to this post by Bernard Li
"Bernard Li" <[hidden email]> writes:

> I am not too familiar with stack mounting -- how is your solution
> different from putting all the rrds in tmpfs?

Well, with tmpfs you need to populate it with RRD's on startup, and
write back to disk at specified intervals or at least on shutdown.

Stacking a tmpfs - which I do also use - over the disk takes care at
least of the first part by implementing a simple copy on write. So you
don't have to take care of tmpfs population anymore.

Unfortunately, nobody yet seems to have foreseen use cases like ours.
So the second part, syncinc back, apparently isn't yet supported.

Typically, stacking is used to mount tmpfs over ro media in
installation/diskless/(quasi)embedded situations to simulate writable
mounts, so historically they came from a different angle.

I haven't yet fully investigated the current state of things though,
so there might even be a solution out there right now resp.
corresponding work being in progress.

>
> I was thinking it would be ideal to put all the rrds in a hybrid
> tmpfs/regular fs where data is written to tmps and periodically synced
> to the regular fs.  Most folks who put rrds in tmpfs have some cronjob
> to do this incase of server crash, but ideally this could be dealt
> with by some fs hooks.  Maybe using inotify or something like that.
>
> You might also be interested in this thread over at rrd-developers:
>
> http://www.mail-archive.com/rrd-developers@.../msg02284.html

Cool, more stuff to read. Thanks a bunch.


Reply | Threaded
Open this post in threaded view
|

Re: Inquiring on how to ease disk io (linux).

Raimund Berger
In reply to this post by oetiker
Tobias Oetiker <[hidden email]> writes:

> Hi Raimund,
>
> Today Ronan Mullally wrote:
> [disk performance config tips]
>
> We have just successfully integrated a ssd disk into our ext3
> filesystem to house the journal. The performance gain for write
> heavy applications was about 100%, but the biggy was that the
> responsivness of all the system became much better even under heavy
> writing load, I would assume that this could help a rrd setup as
> well ...
>
> http://insights.oetiker.ch/linux/external-journal-on-ssd.html

Ah, ok thanks for the tip. Very stimulating with which solutions you
guys come up.

>From application layer in memory caching over tweaking OS level vm and
buffering knobs to reusing OS level performant disk caches (journal).

As a random thought along those lines, if the RRDs were housed on an
own partition, putting the journal into tmpfs might also work, even
though kind of defeating the original purpose I guess ;)

Thanks a bunch
R.


Reply | Threaded
Open this post in threaded view
|

Re: Inquiring on how to ease disk io (linux).

Raimund Berger
In reply to this post by Fabien Wernli
Fabien Wernli <[hidden email]> writes:

> On Wed, Jun 25, 2008 at 12:31:09PM +0200, Raimund Berger wrote:
>> Hm, good point, but is there already a publicly available solution for
>> this approach?  Sounds to me like distributing some home grown scripts
>> over the network ...
>
> I have some trivial shell scripts if you're interested
> it's only just a matter of storing your values in memory and sending them
> over as usual but every now and then

I guess I'll first make an attempt on server side solutions. If they
don't scale as expected, I'm sure your approach does, and I'd sure
appreciate your offer in that case.

Many thanks,
R.