Who has the most RRD files (or data sources)?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Who has the most RRD files (or data sources)?

Mark Plaksin
Who has the most RRD files (or total data sources in RRDs)?  How frequently
are they updated?  What do you use to update them?

We are using a home-grown system to update over 20,000 RRD files.  Each RRD
file has a single data source.  RRD files are using a total of 45G of disk
space.

Data collection is separate from RRD-updating.  We used to stuff data into
RRDs every 5 minutes but that just caused the disks to be hammered all the
time.  The machine was still usable but the IO bottleneck caused it to be
pretty slow.  It wasn't swapping.

We switched to stuffing more data points at a time at less frequent
intervals and things are better.  But we really want to stuff every 5
minutes.

Clearly we're doing something wrong :)  I'm looking for pointers about
making our system more efficient.  While it is possible that our machine
has a poor IO system, let's pretend it doesn't and just talk about the most
efficient way to get data into a large number of RRD files.  We're
investigating the IO system separately.

Besides the size of the RRD files, does the number of rows per RRA impact
performance?

Thanks for any assistance!

--
Unsubscribe mailto:[hidden email]?subject=unsubscribe
Help        mailto:[hidden email]?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi

Reply | Threaded
Open this post in threaded view
|

Re: Who has the most RRD files (or data sources)?

nate-4
Mark Plaksin said:
> Who has the most RRD files (or total data sources in RRDs)?  How frequently
> are they updated?  What do you use to update them?


At my last company(just left it) I made a custom RRD system that
had about 20-30k of RRD files spread accross 8 servers, each
datafile was updated at least once per minute, there was about
5k that were updated more often than that(ranging from 5 seconds
to 45 seconds). The servers were taxed quite hard, the systems
were not very responsive on the command line. system specs for
the latest round of systems was:

Dual 3Ghz Xeon 1MB cache
4GB memory
8x74GB 10k RPM Raptor disks in RAID10 (3ware raid)
7x74GB 10k RPM Raptor disks in RAID5 (3ware raid)
Debian Sarge(3.1) 2.4.31 kernel
(these servers are redundant, 2 systems monitoring the
same datapoints)

(these systems struggled with disk I/O with about 5k files updating
every minute, load was typically in the 5.0-7.0 range, didn't appear
to impact end user experience using rrdcgi but using the shell was
quite painful)

>
> We are using a home-grown system to update over 20,000 RRD files.  Each RRD
> file has a single data source.  RRD files are using a total of 45G of disk
> space.

my datafiles consumed around 1.3TB of space I think. My standard
was to keep 2 years worth of data so the typical datafile size was
around 65MB. I am sure it wasn't the best way to store the data but
I couldn't find any real clear easy to understand docs on how to
store stuff in RRAs so I just stored every data point and never
went back and re-evaluated things. I created:

RRA:(MIN|AVG|MAX|LAST):0.5:1:1051200


> Clearly we're doing something wrong :)  I'm looking for pointers about
> making our system more efficient.  While it is possible that our machine
> has a poor IO system, let's pretend it doesn't and just talk about the most
> efficient way to get data into a large number of RRD files.  We're
> investigating the IO system separately.
>
> Besides the size of the RRD files, does the number of rows per RRA impact
> performance?

I asked a similar question a few months back (9/17/2005 - "scaling RRDTool
to the sky..and beyond") and did not get any responses(that I noticed). What
sort of hardware are you running?

nate

--
Unsubscribe mailto:[hidden email]?subject=unsubscribe
Help        mailto:[hidden email]?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi

Reply | Threaded
Open this post in threaded view
|

Re: Who has the most RRD files (or data sources)?

Jason Fesler
In reply to this post by Mark Plaksin
On Wed, 15 Mar 2006, Mark Plaksin wrote:

> Who has the most RRD files (or total data sources in RRDs)?  How frequently
> are they updated?  What do you use to update them?

I can't say how many I'm doing aggregate..  but..  on a single box basis,
we're updating 2000 files with an average of 10 ds's per rrd file, fora
total of 20,000 values, once a minute, and we have reasonable query
performance for displaying data when we need it.

On a server with 8 ide disks, raid 10, 3ware, CPU is usually not the
bottleneck (except for displays perhaps).  Aveage write wait is about
3 ms (as instrumented by our collection scripts).  Linux report our
I/O utilization at 25-30%.

We've pushed it much higher in the past, but that basically made all that
valuable data.. useless.  Especially when things are on fire and you need
access to that data NOW.

Our observation  is that as we add rrd files, load goes up significantly.
Adding more DS's to existing files, much more negligable.  However, adding
DS's to files, is painful  at best.


--
Unsubscribe mailto:[hidden email]?subject=unsubscribe
Help        mailto:[hidden email]?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi

Reply | Threaded
Open this post in threaded view
|

Re: Who has the most RRD files (or data sources)?

Tobias Oetiker
In reply to this post by Mark Plaksin
Hi Mark,

I have not done serious work into performance analysis, but you may
want to make sure that you have lots of memory and that you use a
current version of rrdtool with memory mapped io support (1.2 does
that for you)

cheers
tobi



--
 ______    __   _
/_  __/_  / /  (_) Oetiker @ ISG.EE, ETL F24.2, ETH, CH-8092 Zurich
 / // _ \/ _ \/ /  System Manager, Time Lord, Coder, Designer, Coach
/_/ \.__/_.__/_/   http://people.ee.ethz.ch/oetiker +41(0)44-632-5286

--
Unsubscribe mailto:[hidden email]?subject=unsubscribe
Help        mailto:[hidden email]?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi

Reply | Threaded
Open this post in threaded view
|

Re: Who has the most RRD files (or data sources)?

Peter Valdemar Mørch
In reply to this post by Mark Plaksin
Mark Plaksin happy-at-usg.edu |Lists| wrote:
> We are using a home-grown system to update over 20,000 RRD files.  Each RRD
> file has a single data source.  RRD files are using a total of 45G of disk
> space.

We just migrated our solution from having one DS pr. file to having on
average 100 ds's pr file. About 2GB worth.

That changed the "updating time" from 13 minutes to 12 seconds (on
standard PC desktop hardware)!!!! Load on the machine went from 20 to
0.9 . Having individual DSes each in their own files is *not* the way to
go for performance!

So now instead we've implemented a way to add, remove and move DSes in
and out of individual files using a dump->parse->load cycle.... Parsing
the XML with a real perl XML parser just took forever, so instead we're
using raw perl regular expressions / string operations. Not GPLed,
though. If there is interest, I'll ask if it can be released as GPL.

Peter

--
Unsubscribe mailto:[hidden email]?subject=unsubscribe
Help        mailto:[hidden email]?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi

Reply | Threaded
Open this post in threaded view
|

Re: Who has the most RRD files (or data sources)?

Mark Plaksin
In reply to this post by Jason Fesler
Jason Fesler <[hidden email]> writes:

> On Wed, 15 Mar 2006, Mark Plaksin wrote:
>
>> Who has the most RRD files (or total data sources in RRDs)?  How frequently
>> are they updated?  What do you use to update them?
>
> I can't say how many I'm doing aggregate..  but..  on a single box basis,
> we're updating 2000 files with an average of 10 ds's per rrd file, fora
> total of 20,000 values, once a minute, and we have reasonable query
> performance for displaying data when we need it.
>
> On a server with 8 ide disks, raid 10, 3ware, CPU is usually not the
> bottleneck (except for displays perhaps).  Aveage write wait is about
> 3 ms (as instrumented by our collection scripts).  Linux report our
> I/O utilization at 25-30%.
>
> We've pushed it much higher in the past, but that basically made all that
> valuable data.. useless.  Especially when things are on fire and you need
> access to that data NOW.
>
> Our observation  is that as we add rrd files, load goes up significantly.
> Adding more DS's to existing files, much more negligable.  However, adding
> DS's to files, is painful  at best.

Do you have a feel for why having more DSes per RRD file gives better
performance?

--
Unsubscribe mailto:[hidden email]?subject=unsubscribe
Help        mailto:[hidden email]?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi

Reply | Threaded
Open this post in threaded view
|

Re: Who has the most RRD files (or data sources)?

Mark Plaksin
In reply to this post by Tobias Oetiker
Tobias Oetiker <[hidden email]> writes:

> Hi Mark,
>
> I have not done serious work into performance analysis, but you may
> want to make sure that you have lots of memory and that you use a
> current version of rrdtool with memory mapped io support (1.2 does
> that for you)

Thanks, Tobi!  We are going to switch to 1.2 soon and will use mmap.

Given the way RRDtool is written, do you have a feel for whether one DS per
RRD or many DSes per RRD should perform better?  Different people say one
or the other is better.

--
Unsubscribe mailto:[hidden email]?subject=unsubscribe
Help        mailto:[hidden email]?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi

Reply | Threaded
Open this post in threaded view
|

Re: Who has the most RRD files (or data sources)?

Mark Plaksin
In reply to this post by Peter Valdemar Mørch
Peter Valdemar Mørch <[hidden email]> writes:

> Mark Plaksin happy-at-usg.edu |Lists| wrote:
>
>> We are using a home-grown system to update over 20,000 RRD files.  Each RRD
>> file has a single data source.  RRD files are using a total of 45G of disk
>> space.
>
> We just migrated our solution from having one DS pr. file to having on
> average 100 ds's pr file. About 2GB worth.
>
> That changed the "updating time" from 13 minutes to 12 seconds (on
> standard PC desktop hardware)!!!! Load on the machine went from 20 to
> 0.9 . Having individual DSes each in their own files is *not* the way to
> go for performance!

Do you know why multiple DSes per RRD is better?  I'm willing to accept
that it is but I'm curious about what makes it better.  It's definitely
easier to manage one DS per RRD.

> So now instead we've implemented a way to add, remove and move DSes in
> and out of individual files using a dump->parse->load cycle.... Parsing
> the XML with a real perl XML parser just took forever, so instead we're
> using raw perl regular expressions / string operations. Not GPLed,
> though. If there is interest, I'll ask if it can be released as GPL.

GPL is always good.  Is the Perl script very complex?  That is, I think I
can imagine a quick Perl script which would probably do the trick but maybe
the job is more complex than I imagine.

Also, do you have a feel for why doing the operation via XML is so slow?
Is it the nature of XML?  Some specific problem with your XML library?
Does your Perl script ever fail or miss some important data?

Thanks!

--
Unsubscribe mailto:[hidden email]?subject=unsubscribe
Help        mailto:[hidden email]?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi

Reply | Threaded
Open this post in threaded view
|

Re: Who has the most RRD files (or data sources)?

Mark Plaksin
In reply to this post by Mark Plaksin
Mark Plaksin <[hidden email]> writes:

> Tobias Oetiker <[hidden email]> writes:
>
>> Hi Mark,
>>
>> I have not done serious work into performance analysis, but you may
>> want to make sure that you have lots of memory and that you use a
>> current version of rrdtool with memory mapped io support (1.2 does
>> that for you)
>
> Thanks, Tobi!  We are going to switch to 1.2 soon and will use mmap.
>
> Given the way RRDtool is written, do you have a feel for whether one DS per
> RRD or many DSes per RRD should perform better?  Different people say one
> or the other is better.

One more question along these lines--ignoring the size of the RRD file
itself, do you have a feel how much consolidation should impact
performance?  Our RRD generally keep one data point per minute for 30 days
and then one data point per hour for 10 years.  The second RRA was pretty
arbitrarily chosen because it didn't have a significant impact on the size
of the RRD files.  And, at the time, we didn't have enough data for
performance to be a concern.

--
Unsubscribe mailto:[hidden email]?subject=unsubscribe
Help        mailto:[hidden email]?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi

Reply | Threaded
Open this post in threaded view
|

Re: Who has the most RRD files (or data sources)?

Jason Fesler
In reply to this post by Mark Plaksin
> Do you have a feel for why having more DSes per RRD file gives better
> performance?

To be clear:  Why, for a given number of DS's, it is cheaper to lump them
together, instead of a DS per file...  that would be due to the number of
seeks per update involved, where I/O is mostly defined by how many RRA's
you use in the rrd - not the number of DS's.

--
Unsubscribe mailto:[hidden email]?subject=unsubscribe
Help        mailto:[hidden email]?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi

Reply | Threaded
Open this post in threaded view
|

Re: Who has the most RRD files (or data sources)?

Tobias Oetiker
In reply to this post by Mark Plaksin
Hi Mark,

opening files is an expensive operation as oposed to writing to a
file ... so having more DS in a single file makes things faster ...
unfortunately this places the whole burden of managing the DSs on
rrdtool or rather on you since rrdtool has no built-in
functionality for adding and removing DS ...

cheers
tobi

Yesterday Mark Plaksin wrote:

> Tobias Oetiker <[hidden email]> writes:
>
> > Hi Mark,
> >
> > I have not done serious work into performance analysis, but you may
> > want to make sure that you have lots of memory and that you use a
> > current version of rrdtool with memory mapped io support (1.2 does
> > that for you)
>
> Thanks, Tobi!  We are going to switch to 1.2 soon and will use mmap.
>
> Given the way RRDtool is written, do you have a feel for whether one DS per
> RRD or many DSes per RRD should perform better?  Different people say one
> or the other is better.
>
> --
> Unsubscribe mailto:[hidden email]?subject=unsubscribe
> Help        mailto:[hidden email]?subject=help
> Archive     http://lists.ee.ethz.ch/rrd-users
> WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi
>

--
 ______    __   _
/_  __/_  / /  (_) Oetiker @ ISG.EE, ETL F24.2, ETH, CH-8092 Zurich
 / // _ \/ _ \/ /  System Manager, Time Lord, Coder, Designer, Coach
/_/ \.__/_.__/_/   http://people.ee.ethz.ch/oetiker +41(0)44-632-5286

--
Unsubscribe mailto:[hidden email]?subject=unsubscribe
Help        mailto:[hidden email]?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi

Reply | Threaded
Open this post in threaded view
|

Re: Who has the most RRD files (or data sources)?

Fabien Wernli
In reply to this post by Mark Plaksin
Hi,

On Sat, Mar 18, 2006 at 09:48:07PM -0500, Mark Plaksin wrote:
>
> Also, do you have a feel for why doing the operation via XML is so slow?
> Is it the nature of XML?  Some specific problem with your XML library?
> Does your Perl script ever fail or miss some important data?
>

When you do full XML processing, you need to load the entire tree into
memory, say just for discarding some columns. Whereas if you sequentially
parse the XML is much faster and less Memory hungry.

--
Unsubscribe mailto:[hidden email]?subject=unsubscribe
Help        mailto:[hidden email]?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi

Reply | Threaded
Open this post in threaded view
|

Re: Who has the most RRD files (or data sources)?

Mark Plaksin
In reply to this post by Tobias Oetiker
Tobias Oetiker <[hidden email]> writes:

> Hi Mark,
>
> opening files is an expensive operation as oposed to writing to a
> file ... so having more DS in a single file makes things faster ...
> unfortunately this places the whole burden of managing the DSs on
> rrdtool or rather on you since rrdtool has no built-in
> functionality for adding and removing DS ...

But opening isn't *that* expensive.  I wrote a short Perl script which
opens and closes all of our 20k+ RRD files and it runs in .5 seconds.  So
it seems like something else is at play.

--
Unsubscribe mailto:[hidden email]?subject=unsubscribe
Help        mailto:[hidden email]?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi

Reply | Threaded
Open this post in threaded view
|

Re: Who has the most RRD files (or data sources)?

Mark Plaksin
In reply to this post by Jason Fesler
Jason Fesler <[hidden email]> writes:

>> Do you have a feel for why having more DSes per RRD file gives better
>> performance?
>
> To be clear:  Why, for a given number of DS's, it is cheaper to lump them
> together, instead of a DS per file...  that would be due to the number of
> seeks per update involved, where I/O is mostly defined by how many RRA's
> you use in the rrd - not the number of DS's.

Does this boil down to meaning that the parts of a single RRD file are
physically close to each other on disk?  And that multiple RRD files are
further apart?  I can see how that would make things a bit faster or slower
but it doesn't seem like it would make the huge difference that some people
seem to see when they put multiple RRAs into one RRD.

Am I just clueless? :)  Or does it make sense to you that the difference is
huge?

--
Unsubscribe mailto:[hidden email]?subject=unsubscribe
Help        mailto:[hidden email]?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi

Reply | Threaded
Open this post in threaded view
|

Re: Who has the most RRD files (or data sources)?

Mark Plaksin
In reply to this post by Fabien Wernli
Fabien Wernli <[hidden email]> writes:

> Hi,
>
> On Sat, Mar 18, 2006 at 09:48:07PM -0500, Mark Plaksin wrote:
>>
>> Also, do you have a feel for why doing the operation via XML is so slow?
>> Is it the nature of XML?  Some specific problem with your XML library?
>> Does your Perl script ever fail or miss some important data?
>>
>
> When you do full XML processing, you need to load the entire tree into
> memory, say just for discarding some columns. Whereas if you sequentially
> parse the XML is much faster and less Memory hungry.

And there are no sequential XML parsers?  And thus XML is only good for
small amounts of data?  I know little about XML but find this hard to
believe :)

--
Unsubscribe mailto:[hidden email]?subject=unsubscribe
Help        mailto:[hidden email]?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi

Reply | Threaded
Open this post in threaded view
|

Re: Who has the most RRD files (or data sources)?

Steve Friedl-3
On Mon, Mar 20, 2006 at 12:35:30PM -0500, Mark Plaksin wrote:
> And there are no sequential XML parsers?  And thus XML is only good for
> small amounts of data?  I know little about XML but find this hard to
> believe :)

You'd be right: there are two broad categories of XML parsers:

DOM - load the whole thing in to a tree
SAX - stream parser with callbacks

DOM is useful if you need to manipulate everything as a whole, but it
sucks everything into memory at once. Not really great for huge trees.

SAX gives callbacks at the various points in the process, so if you
need to just pick out one part of the tree, it can be done with no
more memory than just the intersting parts.

Steve

--
Stephen J Friedl | Security Consultant |  UNIX Wizard  |   +1 714 544-6561
www.unixwiz.net  | Tustin, Calif. USA  | Microsoft MVP | [hidden email]

--
Unsubscribe mailto:[hidden email]?subject=unsubscribe
Help        mailto:[hidden email]?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi

Reply | Threaded
Open this post in threaded view
|

Re: Who has the most RRD files (or data sources)?

Peter Valdemar Mørch
In reply to this post by Mark Plaksin
Mark Plaksin happy-at-usg.edu |Lists| wrote:
> Am I just clueless? :)  Or does it make sense to you that the difference is
> huge?

I have no idea whether you are clueless! :-D

Personally, I don't understand it either. I was *very* surprised to see
a factor 60 improvement in our system by consolidating into fewer RRD files.

So I'm following your line of questions with much interest too!

... Maybe we're both clueless! ;D

Peter
--
Peter Valdemar Mørch
http://www.morch.com

--
Unsubscribe mailto:[hidden email]?subject=unsubscribe
Help        mailto:[hidden email]?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi

Reply | Threaded
Open this post in threaded view
|

Re: Who has the most RRD files (or data sources)?

Peter Valdemar Mørch
In reply to this post by Steve Friedl-3
Steve Friedl steve+rrd-users-at-unixwiz.net |Lists| wrote:

> On Mon, Mar 20, 2006 at 12:35:30PM -0500, Mark Plaksin wrote:
>
>>And there are no sequential XML parsers?  And thus XML is only good for
>>small amounts of data?  I know little about XML but find this hard to
>>believe :)
>
>
> You'd be right: there are two broad categories of XML parsers:
>
> DOM - load the whole thing in to a tree
> SAX - stream parser with callbacks
>
> DOM is useful if you need to manipulate everything as a whole, but it
> sucks everything into memory at once. Not really great for huge trees.
>
> SAX gives callbacks at the various points in the process, so if you
> need to just pick out one part of the tree, it can be done with no
> more memory than just the intersting parts.
>

I didn't even bother considering a DOM parser for the huge dump files. I
tried running the dump files through perl's XML::Parser which behind the
scenes is James Clark's expat library.

Memory consumption was not high, but IIRC parsing one of our "typicial
rrd files" took 30 secs with XML::Parser, without doing *anything* with
the parsed data.

Parsing and extracting the structure only took 6 seconds with regexps -
so I dropped the XML parser. But yes, if Tobi decides to change the
output of rrdtool dump while still adhering to the same DTD, we'll be
vulnerable to that.

Peter

--
Unsubscribe mailto:[hidden email]?subject=unsubscribe
Help        mailto:[hidden email]?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi

Reply | Threaded
Open this post in threaded view
|

Re: Who has the most RRD files (or data sources)?

Tobias Oetiker
In reply to this post by Mark Plaksin
Mark,

well it is opening and accessing/seeking ...

cheers
tobi
Today Mark Plaksin wrote:

> Tobias Oetiker <[hidden email]> writes:
>
> > Hi Mark,
> >
> > opening files is an expensive operation as oposed to writing to a
> > file ... so having more DS in a single file makes things faster ...
> > unfortunately this places the whole burden of managing the DSs on
> > rrdtool or rather on you since rrdtool has no built-in
> > functionality for adding and removing DS ...
>
> But opening isn't *that* expensive.  I wrote a short Perl script which
> opens and closes all of our 20k+ RRD files and it runs in .5 seconds.  So
> it seems like something else is at play.
>
> --
> Unsubscribe mailto:[hidden email]?subject=unsubscribe
> Help        mailto:[hidden email]?subject=help
> Archive     http://lists.ee.ethz.ch/rrd-users
> WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi
>

--
 ______    __   _
/_  __/_  / /  (_) Oetiker @ ISG.EE, ETL F24.2, ETH, CH-8092 Zurich
 / // _ \/ _ \/ /  System Manager, Time Lord, Coder, Designer, Coach
/_/ \.__/_.__/_/   http://people.ee.ethz.ch/oetiker +41(0)44-632-5286

--
Unsubscribe mailto:[hidden email]?subject=unsubscribe
Help        mailto:[hidden email]?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi

Reply | Threaded
Open this post in threaded view
|

Re: Who has the most RRD files (or data sources)?

Jason Fesler
>> But opening isn't *that* expensive.  I wrote a short Perl script which
>> opens and closes all of our 20k+ RRD files and it runs in .5 seconds.  So
>> it seems like something else is at play.

Open.
Seek.
Read a chunk for the headers.
Seek.
Write a chunk for the headers.
For each RRA (regardless of DS count):
   Seek.
   Write.

--
Unsubscribe mailto:[hidden email]?subject=unsubscribe
Help        mailto:[hidden email]?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi

12