max DS per rrd file

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

max DS per rrd file

mikel
Hi all,

What is a healthy number of data-sources for a given RRD file?

I was considering to have 50 per file, but I have many cases where logically it would make sense to store hundreds.

I was looking for a standard-way to scale this numbers but I could not find a definitive guide.

I am only looking for a one-year lifespan and I am happy for data to 'blur'.

How good/bad is it for a standard rrd file to make queries/add data with so many data sources?

Any hint to a good document is much appreciated since I could not find much information.

I know there are ways to programatically create more RRD files but I am interested to know under which values the RRD would normally start misbehaving and becoming unusable (too long to respond to a query).

Thanks in advance,
m
Reply | Threaded
Open this post in threaded view
|

Re: [unsure] max DS per rrd file

Alex van den Bogaerdt-5

----- Original Message -----
From: "mikel" <[hidden email]>
To: <[hidden email]>
Sent: Saturday, April 20, 2013 11:08 AM
Subject: [unsure] [rrd-users] max DS per rrd file


> Hi all,
>
> What is a healthy number of data-sources for a given RRD file?
>
> I was considering to have 50 per file, but I have many cases where
> logically
> it would make sense to store hundreds.

This begs for a return question...

Please explain?

> I was looking for a standard-way to scale this numbers but I could not
> find
> a definitive guide.

That's probably because it is depending on too many variables. Your hardware
amongst them.

> I am only looking for a one-year lifespan and I am happy for data to
> 'blur'.
>
> How good/bad is it for a standard rrd file to make queries/add data with
> so
> many data sources?

It's just more work, using more resources. More CPU, more memory, more disk.

 > Any hint to a good document is much appreciated since I could not find
much
> information.
>
> I know there are ways to programatically create more RRD files but I am
> interested to know under which values the RRD would normally start
> misbehaving and becoming unusable (too long to respond to a query).

If you divide your data into more than one RRD, but keep updating all DSes,
I think there will be slightly more work, not less.

I have to admit, there are people on this list with more experience with
large scale setups. Let's hope they chime in. Also look through the
archives, but I cannot give hints on which keywords to use.

_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
Reply | Threaded
Open this post in threaded view
|

Re: [unsure] max DS per rrd file

mikel
Thanks for your quick response.

A bit more background information follows,

-Some hosts have 50-60 metrics (DSs)
-Other hosts have up to 200 metrics (DSs)
-All metrics are different

-Some metrics update fresh data every 5 mins, but not all of them at the same time
-Updates are local to each host, same with queries, that is to say, RRD file lives in each of the monitored machines local disk, without being shared with anything else.

-Some metrics may get queried eventually, but this is far less common than updates. For example queries would be around the number of ten per week. However the query would need to scan all metrics in a given RRD file.

-I can not run rrdcached agent on each of these hosts.

I know this is very uncommon setup.

As you can see this is a monitoring setup for a largely distributed cluster, so the only way I can see to have a long historical record of too many metrics is to keep them distributed.

Obviously I am building a mechanism to query the nodes, however this is not the main problem.

What I really would like to hear about is:

-even when using the highest possible "data-loss" for a given RRD file, does anybody have experience querying/updating something between fifty to 200 Datasources inside one given RRD file ?

I am open to hear for any comments, no matter how crazy they are.

Thanks again,
m
Reply | Threaded
Open this post in threaded view
|

Re: [unsure] max DS per rrd file

Alex van den Bogaerdt-5

----- Original Message -----
From: "mikel" <[hidden email]>
To: <[hidden email]>
Sent: Saturday, April 20, 2013 11:42 AM
Subject: Re: [rrd-users] [unsure] max DS per rrd file


> Thanks for your quick response.
>
> A bit more background information follows,
>
> -Some hosts have 50-60 metrics (DSs)
> -Other hosts have up to 200 metrics (DSs)
> -All metrics are different
>
> -Some metrics update fresh data every 5 mins, but not all of them at the
> same time

That would be a good reason to spread them over multiple RRDs, so you can
spread out the load.

> -Updates are local to each host, same with queries, that is to say, RRD
> file
> lives in each of the monitored machines local disk, without being shared
> with anything else.
>
> -Some metrics may get queried eventually, but this is far less common than
> updates. For example queries would be around the number of ten per week.
> However the query would need to scan all metrics in a given RRD file.

Maybe I don't understand what you say here. Some metrics, or all metrics are
queried? Both statements cannot be true at the same time?

Anyway, if you query only once in a while, maybe you should think about
reducing the number of RRAs in each RRD, and just let it consolidate at
graph time. Yes, this will mean you will have to wait longer for your graph
to be made, but you save processing time at every update.

No, I cannot give numbers ;)

Clipped the rest; others can do better I'm sure.

_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
Reply | Threaded
Open this post in threaded view
|

Re: [unsure] max DS per rrd file

mikel

Thanks for your fast reply again.

>Maybe I don't understand what you say here. Some metrics, or all metrics are
>queried? Both statements cannot be true at the same time?

Yes it is a tricky case. Apologies I was not clear enough.

In most cases all metrics are queried at the same time, because we want to know what value they had at a given time. And classify them.

Very randomly we would query for just one metric.

>Anyway, if you query only once in a while, maybe you should think about
>reducing the number of RRAs in each RRD, and just let it consolidate at
>graph time. Yes, this will mean you will have to wait longer for your graph
>to be made, but you save processing time at every update.

This is interesting I did not think about that. Thanks for the hint.

Thanks for your help again.
m
Reply | Threaded
Open this post in threaded view
|

Re: [unsure] max DS per rrd file

David Thornton
I try to group the collection in logical units: 15 sql vars per host -> 1 file. 25 smtp variables per host -> 1 file. 1500 hosts? 3000 files. Each host writes two files. It spreads well.

Later I move those files centrally and start "looking" at them. Separating collection from reporting is big. There is a trade off, making the collection nice and light , my reporting is has more latency in it, both in the time before I get data, and the complexity and time to compose the report. google "olap versus oltp".

David Thornton


On Sat, Apr 20, 2013 at 7:48 AM, mikel <[hidden email]> wrote:

Thanks for your fast reply again.

>Maybe I don't understand what you say here. Some metrics, or all metrics
are
>queried? Both statements cannot be true at the same time?

Yes it is a tricky case. Apologies I was not clear enough.

In most cases all metrics are queried at the same time, because we want to
know what value they had at a given time. And classify them.

Very randomly we would query for just one metric.

>Anyway, if you query only once in a while, maybe you should think about
>reducing the number of RRAs in each RRD, and just let it consolidate at
>graph time. Yes, this will mean you will have to wait longer for your graph
>to be made, but you save processing time at every update.

This is interesting I did not think about that. Thanks for the hint.

Thanks for your help again.
m



--
View this message in context: http://rrd-mailinglists.937164.n2.nabble.com/max-DS-per-rrd-file-tp7580966p7580971.html
Sent from the RRDtool Users Mailinglist mailing list archive at Nabble.com.

_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users


_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
Reply | Threaded
Open this post in threaded view
|

Re: [unsure] max DS per rrd file

mikel
Thank you David, that is interesting input. cheers, m


2013/4/20 David Thornton [via RRD Mailinglists] <[hidden email]>
I try to group the collection in logical units: 15 sql vars per host -> 1 file. 25 smtp variables per host -> 1 file. 1500 hosts? 3000 files. Each host writes two files. It spreads well.

Later I move those files centrally and start "looking" at them. Separating collection from reporting is big. There is a trade off, making the collection nice and light , my reporting is has more latency in it, both in the time before I get data, and the complexity and time to compose the report. google "olap versus oltp".

David Thornton


On Sat, Apr 20, 2013 at 7:48 AM, mikel <[hidden email]> wrote:

Thanks for your fast reply again.

>Maybe I don't understand what you say here. Some metrics, or all metrics
are
>queried? Both statements cannot be true at the same time?

Yes it is a tricky case. Apologies I was not clear enough.

In most cases all metrics are queried at the same time, because we want to
know what value they had at a given time. And classify them.

Very randomly we would query for just one metric.

>Anyway, if you query only once in a while, maybe you should think about
>reducing the number of RRAs in each RRD, and just let it consolidate at
>graph time. Yes, this will mean you will have to wait longer for your graph
>to be made, but you save processing time at every update.

This is interesting I did not think about that. Thanks for the hint.

Thanks for your help again.
m



--
View this message in context: http://rrd-mailinglists.937164.n2.nabble.com/max-DS-per-rrd-file-tp7580966p7580971.html

Sent from the RRDtool Users Mailinglist mailing list archive at Nabble.com.

_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users


_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users



If you reply to this email, your message will be added to the discussion below:
http://rrd-mailinglists.937164.n2.nabble.com/max-DS-per-rrd-file-tp7580966p7580973.html
To unsubscribe from max DS per rrd file, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: [unsure] max DS per rrd file

Ryan Kubica
In reply to this post by mikel

Hi Mikel,

I've personally never found a good reason to store more than one datasource per RRD datafile; and run -very large- rrdtool data servers ( multi-millions per server - many servers .)

There are far too many edge-cases, latency issues and join overhead in trying to consolidate datasources into a single datafile.  Yes, rrdtool itself is more efficient with an insert like that but: 1) what if the datapoints are collected at different times?  2) what if they are different steps?  3) what if you want to add a datasource? 4) what if you simply have too many datasources to try and order/consolidate from a queue to the datafile?  There is also non-trivial complexity, overhead and index'ing into an rrd datafile for specific datasources. 

Linux is extremely efficient at block updates, caching, open/closes, etc ... rrdtool on a low-end ( 4 cpu ) server with limited memory can easily store 160 thousand datasources per minute - on a better server, a whole lot more than that.

'Distributed Cluster' isn't a good reason to not send all your time-series data to one server or small set of servers.  The latency/request-time incurred in having to fetch data from those servers is usually not worth the trade off.

Graphs of many hundreds of datasources computed for multi-day/week time-ranges in the result set are generated in 10s of milliseconds; not seconds ... rrdtool is quite capable of producing on-demand graphs of hundreds of graphs per second from one server.

I suggest you write a little test-script to write out rrd data to individual rrd datafiles to see 'how quick' your servers are at it.  There is some OS tuning and rrdtool RRA sizing that will help; especially don't keep hour or daily rollups ... the server has to hold onto those blocks to make the consolidation quick and not incur a read from disk.

rrdtool scales rather simply ( and without rrdcached -- as I don't use that either. )

HTH
-Ryan


From: mikel <[hidden email]>
To: [hidden email]
Sent: Saturday, April 20, 2013 4:48 AM
Subject: Re: [rrd-users] [unsure] max DS per rrd file


Thanks for your fast reply again.

>Maybe I don't understand what you say here. Some metrics, or all metrics
are
>queried? Both statements cannot be true at the same time?

Yes it is a tricky case. Apologies I was not clear enough.

In most cases all metrics are queried at the same time, because we want to
know what value they had at a given time. And classify them.

Very randomly we would query for just one metric.

>Anyway, if you query only once in a while, maybe you should think about
>reducing the number of RRAs in each RRD, and just let it consolidate at
>graph time. Yes, this will mean you will have to wait longer for your graph
>to be made, but you save processing time at every update.

This is interesting I did not think about that. Thanks for the hint.

Thanks for your help again.
m



--
View this message in context: http://rrd-mailinglists.937164.n2.nabble.com/max-DS-per-rrd-file-tp7580966p7580971.html
Sent from the RRDtool Users Mailinglist mailing list archive at Nabble.com.

_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users



_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users