Using Granular RRA Automatically

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Using Granular RRA Automatically

Stephen R Guglielmo
Hello,

I'm new to rrdtool. I'm trying to create a database that will record
and graph data from an application. I have been using the rrdtool
documentation and the RRD Wizard website to assist me. I'd like to
keep track of "total sites," "deleted sites," "outdated sites, and
"total users" (users will be a separate graph).

rrdtool create filename.rrd \
  --step '86400' \
  'DS:TotalSites:GAUGE:172800:0:U' \
  'DS:DeletedSites:GAUGE:172800:0:U' \
  'DS:OutdatedSites:GAUGE:172800:0:U' \
  'DS:TotalUsers:GAUGE:172800:0:U' \
  'RRA:MIN:0.3:7:520' \
  'RRA:MAX:0.3:7:520' \
  'RRA:AVERAGE:0.3:7:520' \
  'RRA:LAST:0.3:7:520' \
  'RRA:MIN:0.3:1:35' \
  'RRA:MAX:0.3:1:35' \
  'RRA:AVERAGE:0.3:1:35' \
  'RRA:LAST:0.3:1:35'

With the above database, I update it once a day with four data points.
I record MIN, MAX, AVERAGE, and LAST on a weekly basis for 10 years. I
also record MIN, MAX, AVERAGE, and LAST on a daily basis for 35 days.

My question is: Since the RRAs are not named, if I create 1 month, 1
year, and 2 year graphs, will the 1 month graph automatically use the
more granular RRAs (the "daily" 35-day ones as opposed to the "weekly"
10-year ones)?

Here is an example `graph` command that I received from the RRD Wizard:

rrdtool graph 'sites-1month.png' \
--title 'Sites, 1 Month' \
--vertical-label 'Number of Sites' \
--width '800' \
--height '300' \
--full-size-mode \
--start end-1m \
'DEF:SitesMin=sites-new.rrd:TotalSites:MIN' \
'DEF:SitesMax=sites-new.rrd:TotalSites:MAX' \
'DEF:SitesAvg=sites-new.rrd:TotalSites:AVERAGE' \
'DEF:SitesLast=sites-new.rrd:TotalSites:LAST' \
'DEF:DeletedMin=sites-new.rrd:DeletedSites:MIN' \
'DEF:DeletedMax=sites-new.rrd:DeletedSites:MAX' \
'DEF:DeletedAvg=sites-new.rrd:DeletedSites:AVERAGE' \
'DEF:DeletedLast=sites-new.rrd:DeletedSites:LAST' \
'DEF:OutdatedMin=sites-new.rrd:OutdatedSites:MIN' \
'DEF:OutdatedMax=sites-new.rrd:OutdatedSites:MAX' \
'DEF:OutdatedAvg=sites-new.rrd:OutdatedSites:AVERAGE' \
'DEF:OutdatedLast=sites-new.rrd:OutdatedSites:LAST' \
'VDEF:SitesMinMin=SitesMin,MINIMUM' \
'VDEF:SitesMaxMax=SitesMax,MAXIMUM' \
'VDEF:SitesAvgAvg=SitesAvg,AVERAGE' \
'VDEF:DeletedMinMin=DeletedMin,MINIMUM' \
'VDEF:DeletedMaxMax=DeletedMax,MAXIMUM' \
'VDEF:OudatedMinMin=OutdatedMin,MINIMUM' \
'VDEF:OutdatedMaxMax=OutdatedMax,MAXIMUM' \
'VDEF:OutdatedAvgAvg=OutdatedAvg,AVERAGE' \
'VDEF:DeletedAvgAvg=DeletedAvg,AVERAGE' \
'LINE1:SitesLast#00FF00:Total Sites' \
'LINE1:DeletedLast#FF0000:Deleted Sites' \
'LINE1:OutdatedLast#FFFF00:Outdated Sites (>=2 Years)' \
'PRINT:SitesMinMin:%.1lf Sites Min' \
'PRINT:SitesMaxMax:%.1lf Sites Max' \
'PRINT:SitesAvgAvg:%.1lf Sites Avg' \
'PRINT:DeletedMinMin:%.1lf Deleted Min' \
'PRINT:DeletedMaxMax:%.1lf Deleted Max' \
'PRINT:DeletedAvgAvg:%.1lf Deleted Avg' \
'PRINT:OudatedMinMin:%.1lf Outdated Min' \
'PRINT:OutdatedMaxMax:%.1lf Outdated Max' \
'PRINT:OutdatedAvgAvg:%.1lf Outdated Avg'


Thank you!
Steve

_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
Reply | Threaded
Open this post in threaded view
|

Re: Using Granular RRA Automatically

Simon Hobson
Stephen R Guglielmo <[hidden email]> wrote:

> With the above database, I update it once a day with four data points.
> I record MIN, MAX, AVERAGE, and LAST on a weekly basis for 10 years. I
> also record MIN, MAX, AVERAGE, and LAST on a daily basis for 35 days.
>
> My question is: Since the RRAs are not named, if I create 1 month, 1
> year, and 2 year graphs, will the 1 month graph automatically use the
> more granular RRAs (the "daily" 35-day ones as opposed to the "weekly"
> 10-year ones)?

Yes, it'll automatically use the "best fitting" RRA - which in most cases will be the right thing. You can also specify the resolution in your graph command.


> Here is an example `graph` command that I received from the RRD Wizard:
>
> rrdtool graph 'sites-1month.png' \
> --title 'Sites, 1 Month' \
> --vertical-label 'Number of Sites' \
> --width '800' \
> --height '300' \
> --full-size-mode \
> --start end-1m \
> ...


It is recommended to explicitly specify the end time, and to round it to an integer multiple of the RRA resolution. So, for an RRA with a resolution of 1 day (86400 seconds), specify the end time as an integer multiple of 86400.
And specify the start time as end-(integer multiple of resolution), and ideally the multiple will be the number of pixels in the graph which avoids rescaling the data to fit.

Longer explanation, RRD will use the RRA which best fits - specifically it'll pick the RRA(s) providing the most data within the specified timescale, and if more than one, the one with the highest resolution. So in your case, if you plot a graph over 36 days then the 1 day RRA won't fill it and it'll switch to a coarser RRA. for 35 days and below, it'll pick the 1 day RRA as having the higher resolution.
But there are cases where a coarser resolution may provide more data - though I don't think they are very common.

Consider if you aggregated at 2 day and 5 day resolution - contrived but illustrates the point. There will be times where the 5 day RRA covers a day later than the 2 day RRA (basically where it's a multiple of 5 days since unix epoch) - and so the "use the RRA with the most data in the desired period" method will switch to the 5 day RRA.
For a case like this, you'd need to explicitly specify the RRA to be used, and your data would stop a day earlier while showing more detail.
_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
Reply | Threaded
Open this post in threaded view
|

Re: Using Granular RRA Automatically

Stephen R Guglielmo
On Mon, Jan 23, 2017 at 6:06 PM, Simon Hobson <[hidden email]> wrote:

> Stephen R Guglielmo <[hidden email]> wrote:
>
>> With the above database, I update it once a day with four data points.
>> I record MIN, MAX, AVERAGE, and LAST on a weekly basis for 10 years. I
>> also record MIN, MAX, AVERAGE, and LAST on a daily basis for 35 days.
>>
>> My question is: Since the RRAs are not named, if I create 1 month, 1
>> year, and 2 year graphs, will the 1 month graph automatically use the
>> more granular RRAs (the "daily" 35-day ones as opposed to the "weekly"
>> 10-year ones)?
>
> Yes, it'll automatically use the "best fitting" RRA - which in most cases will be the right thing. You can also specify the resolution in your graph command.
>
>
>> Here is an example `graph` command that I received from the RRD Wizard:
>>
>> rrdtool graph 'sites-1month.png' \
>> --title 'Sites, 1 Month' \
>> --vertical-label 'Number of Sites' \
>> --width '800' \
>> --height '300' \
>> --full-size-mode \
>> --start end-1m \
>> ...
>
>
> It is recommended to explicitly specify the end time, and to round it to an integer multiple of the RRA resolution. So, for an RRA with a resolution of 1 day (86400 seconds), specify the end time as an integer multiple of 86400.
> And specify the start time as end-(integer multiple of resolution), and ideally the multiple will be the number of pixels in the graph which avoids rescaling the data to fit.
>
> Longer explanation, RRD will use the RRA which best fits - specifically it'll pick the RRA(s) providing the most data within the specified timescale, and if more than one, the one with the highest resolution. So in your case, if you plot a graph over 36 days then the 1 day RRA won't fill it and it'll switch to a coarser RRA. for 35 days and below, it'll pick the 1 day RRA as having the higher resolution.
> But there are cases where a coarser resolution may provide more data - though I don't think they are very common.
>
> Consider if you aggregated at 2 day and 5 day resolution - contrived but illustrates the point. There will be times where the 5 day RRA covers a day later than the 2 day RRA (basically where it's a multiple of 5 days since unix epoch) - and so the "use the RRA with the most data in the desired period" method will switch to the 5 day RRA.
> For a case like this, you'd need to explicitly specify the RRA to be used, and your data would stop a day earlier while showing more detail.


Simon,

Thank you for the information. It's helpful. If I were to do something
like the following:

rrdtool create filename.rrd \
  --step '86400' \
  'DS:Sites:GAUGE:172800:0:U' \
  'DS:DeletedSites:GAUGE:172800:0:U' \
  'DS:OutdatedSites:GAUGE:172800:0:U' \
  'DS:Users:GAUGE:172800:0:U' \
  'RRA:LAST:0.3:1:3660'

This will basically keep every single value entered for 3660 days (~10
years), correct? It won't consolidate any values? I understand this
will have increased disk usage in the .rrd file, but are there any
other downsides to doing this?

Thank you!
Steve

_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
Reply | Threaded
Open this post in threaded view
|

Re: Using Granular RRA Automatically

Simon Hobson
Stephen R Guglielmo <[hidden email]> wrote:

> If I were to do something
> like the following:
>
> rrdtool create filename.rrd \
>  --step '86400' \
>  'DS:Sites:GAUGE:172800:0:U' \
>  'DS:DeletedSites:GAUGE:172800:0:U' \
>  'DS:OutdatedSites:GAUGE:172800:0:U' \
>  'DS:Users:GAUGE:172800:0:U' \
>  'RRA:LAST:0.3:1:3660'
>
> This will basically keep every single value entered for 3660 days (~10
> years), correct? It won't consolidate any values?

Correct.

> I understand this
> will have increased disk usage in the .rrd file, but are there any
> other downsides to doing this?

For this application, I don't think so.
In general, it's a trade-off between the amount of data retained (ie how far back you can go vs how detailed it is), disk usage, and resources needed to draw graphs.
So if you were to keep (say) traffic for every 5 seconds for the last 10 years with no consolidation, then drawing graphs would be very I/O and processor intensive as it would have to do a lot of consolidation to generate the graph (as well as taking a lot of disk space). In your case, even keeping unconsolidated data for 10 years, it's not a large amount of data, and unless youre system is very resource constrained, I doubt it's going to make any difference.

_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users