wrong rra used

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

wrong rra used

spock
Hi,

I am using this database to collect temperature data for a swimming pool.
I want to keep the detailed data for 36 months ("103680 * 900sec") but also want to have a daily average temp (96 values = 24h).

# rrdtool create temp_pool.rrd --step 900 --start "20140101 00:00" \
# DS:air:GAUGE:1200:-40:50 \
# DS:flow:GAUGE:1200:-40:50 \
# DS:return:GAUGE:1200:-40:50 \
# DS:delta:COMPUTE:return,flow,- \
# DS:pump_flag:GAUGE:1200:0:1 \
# DS:heat_flag:GAUGE:1200:0:1 \
# DS:device:GAUGE:1200:0:90
# RRA:AVERAGE:0.5:1:103680 \
# RRA:MIN:0.5:96:3650 \
# RRA:MAX:0.5:96:3650 \
# RRA:AVERAGE:0.5:96:3650

When I create a graph with
       rrdtool graph temp7d.png \
        -t "Pooltemp (Status $(date "+%d.%m.%Y %H:%M"))"  \
        -s $graph_start_7d -e $timestamp_rounded_unix \
        -w 640 -h 400 -D -a PNG -T 15\
        DEF:airmax2=temp_pool.rrd:air:MAX \
        DEF:airmin2=temp_pool.rrd:air:MIN \
        DEF:airavg2=temp_pool.rrd:air:AVERAGE \
        CDEF:airrange=airmax2,airmin2,- \
        LINE2:airmin2#00FF00:"Luft Min" \
        AREA:airrange#8dadf588::STACK \
        LINE2:airmax2#00FF00:"Luft Min" \
        LINE2:airavg2#00FF00:Aussen

I get for airavg2 the datailed values - but not the daily average.
Airmin2 and airmax2 work as  expected - there is no other RRA Archive to choose from.

Question (1)
How do I convince the tool to use the consolidated data instead of the detailed?

Question (2)
When drawing the graph, the line for max/min values are not precisely aligned above a day. It looks like it starts about 900 secs after 0:00 each day.
How does rrdtool determine, which 96 values to consolidate? My data point are stored with rounded timestamps (e.g. 13.06.2014 0:00, 0:15, 0:30, etc.)

Question (3)
The ssf factor is 0.5 for the RRAs. However, there is no min/max value for certain days with missing data, although there are only a few values missing. I am sure, is is not more than 50%. How can I check this.

Reply | Threaded
Open this post in threaded view
|

Re: wrong rra used

Simon Hobson
spock <[hidden email]> wrote:

> I am using this database to collect temperature data for a swimming pool.
> I want to keep the detailed data for 36 months ("103680 * 900sec") but also
> want to have a daily average temp (96 values = 24h).
>
> # rrdtool create temp_pool.rrd --step 900 --start "20140101 00:00" \
> # DS:air:GAUGE:1200:-40:50 \
> # DS:flow:GAUGE:1200:-40:50 \
> # DS:return:GAUGE:1200:-40:50 \
> # DS:delta:COMPUTE:return,flow,- \
> # DS:pump_flag:GAUGE:1200:0:1 \
> # DS:heat_flag:GAUGE:1200:0:1 \
> # DS:device:GAUGE:1200:0:90
> # RRA:AVERAGE:0.5:1:103680 \
> # RRA:MIN:0.5:96:3650 \
> # RRA:MAX:0.5:96:3650 \
> # RRA:AVERAGE:0.5:96:3650
>
> When I create a graph with
>       rrdtool graph temp7d.png \
>        -t "Pooltemp (Status $(date "+%d.%m.%Y %H:%M"))"  \
>        -s $graph_start_7d -e $timestamp_rounded_unix \
>        -w 640 -h 400 -D -a PNG -T 15\
>        DEF:airmax2=temp_pool.rrd:air:MAX \
>        DEF:airmin2=temp_pool.rrd:air:MIN \
>        DEF:airavg2=temp_pool.rrd:air:AVERAGE \
>        CDEF:airrange=airmax2,airmin2,- \
>        LINE2:airmin2#00FF00:"Luft Min" \
>        AREA:airrange#8dadf588::STACK \
>        LINE2:airmax2#00FF00:"Luft Min" \
>        LINE2:airavg2#00FF00:Aussen
>
> I get for airavg2 the datailed values - but not the daily average.
> Airmin2 and airmax2 work as  expected - there is no other RRA Archive to
> choose from.
>
> Question (1)
> How do I convince the tool to use the consolidated data instead of the
> detailed?

It does the best it can to use the most detailed data. Looking at the docs, I don't see any option to override that.


> Question (2)
> When drawing the graph, the line for max/min values are not precisely
> aligned above a day. It looks like it starts about 900 secs after 0:00 each
> day.

Looks more like an hour to me, are you on +1 timezone by any chance ?

> How does rrdtool determine, which 96 values to consolidate? My data point
> are stored with rounded timestamps (e.g. 13.06.2014 0:00, 0:15, 0:30, etc.)

*ALL* periods are aligned to an integer multiple of step (or step * steps/CDP) since unix epoch (midnight 1-1-1970 UTC). So if you consolidate to a day then a "day" will always start at 00:00:00 UTC. If your timezone isn't UTC, then the data will appear to be off by your timezone offset.


> Question (3)
> The ssf factor is 0.5 for the RRAs. However, there is no min/max value for
> certain days with missing data, although there are only a few values
> missing. I am sure, is is not more than 50%. How can I check this.
>
> <http://rrd-mailinglists.937164.n2.nabble.com/file/n7582214/temp7d.png>

Are you certain you are updating frequently enough ? I can see some gaps in the data, if you have more than 1200s between updates then you'll have gaps.


For more on how consolidation is done, see this tutorial
http://www.vandenbogaerdt.nl/rrdtool/process.php

_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
Reply | Threaded
Open this post in threaded view
|

Re: wrong rra used

spock
Hello Simon,

thanks for your comments...

> It does the best it can to use the most detailed data. Looking at the docs, I don't see any
> option to override that.

I saw a similar setup on Linkand there are also a 2 RRA; the example graph showed the correct average line: average of all values of a single day.
Maybe I try to contact him if there is any "trick" involved.

Do you see any trick to create a single data point per day with the average value (temperature)?


> Looks more like an hour to me, are you on +1 timezone by any chance ?
Yes, I am in UTC+1 timezone...
So data consolidation is always done based on UTC+0 timezone.

I am using the following unix statements to get the timestamp:

        set $(date "+%Y %m %d %H %M")
        m=$((100+15*(${5#0}/15)))
        timestamp_rounded="$1$2$3 $4:${m#1}"
        timestamp_rounded_unix=$(date --date "$1$2$3 $4:${m#1}" +%s)
        timestamp_real=$(date "+%Y%m%d %H:%M")
        timestamp_real_unix=$(date --date "$timestamp_real" +%s)

If I check my own logfiles, i can see:
I106 rrdtool update /testtemp/temp_pool.rrd 1403702100:23.5:22.6:29.8:23.5:23.9:1:1:50.8
I103 20140625 15:15 --> this is my time, UTC+1 CEST

1403702100 translates into 25.06.2014 - 15:15:00

So the algorithm from a rrdtool perspective uses my time as UTC+0 time.

I still cannot explain this 1h offset in the graph...



> Are you certain you are updating frequently enough ?

The reason here was a downtime due to maintenance work.
Well, as I understand it, the RRA function tolerates with SSF 0.5, that 50% of data points may be missing. As you can see in the graph, there are a few values missing, but not 50% (48 data points out of 96).  Or does the max function return no values, if the data vector contains unknowns?
Reply | Threaded
Open this post in threaded view
|

Re: wrong rra used

Simon Hobson
spock <[hidden email]> wrote:

> Do you see any trick to create a single data point per day with the average
> value (temperature)?

You could create two RRDs - one with and one without a high resolution consolidation. Then you can use the two RRDs in the one graph.
Either modify your scripts to update both RRDs, or have another script that updates the second with data from the first. I'd recommend the former, the latter can be messy.



_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
Reply | Threaded
Open this post in threaded view
|

Re: wrong rra used

spock
Hello Simon,

I now found the option to specify the resolution of a "DEF" statement.

if I enter in the rrdtool graph command:

        DEF:airavg2=temp_pool.rrd:air:AVERAGE:step=86400 \

The step=86400 option averages the available data for 24h = 86400 secs. This way I get the desired result (a max temperature, a min temperature and the average temperature per day).

I am still struggling with the 1h offset, probably cause by the fact, that I am living in UTC+1 timezone. I will post the solution, if I find it myseld.

Regards,
Spock
Reply | Threaded
Open this post in threaded view
|

Re: wrong rra used

S Shipway
As far as the 1hr offset (due to all summaries being done in UTC)  goes, there is no in-RRDtool solution.  We're in New Zealand so the offset is 12h or 13h, so it really shows!  If I remember correctly, Tobi mentioned about possibly addressing this issue in RRDTool 1.5 or later; it would probably require a new RRD format as there would be a timezone component to each RRA...

If you really need to work around it, you might be able to extract per-hour data and summarise in your own code, but that still doesn't help for when you want a graph of one CDP per day.

Steve


Steve Shipway
University of Auckland ITS
UNIX Systems Design Lead
[hidden email]
Ph: +64 9 373 7599 ext 86487


_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
Reply | Threaded
Open this post in threaded view
|

Re: wrong rra used

Alex van den Bogaerdt-5
> As far as the 1hr offset (due to all summaries being done in UTC)  goes,
> there is no in-RRDtool solution.  We're in New Zealand so the offset is
> 12h or 13h, so it really shows!  If I remember correctly, Tobi mentioned
> about possibly addressing this issue in RRDTool 1.5 or later; it would
> probably require a new RRD format as there would be a timezone component
> to each RRA...
>
> If you really need to work around it, you might be able to extract
> per-hour data and summarise in your own code, but that still doesn't help
> for when you want a graph of one CDP per day.

The obvious workaround is to let the computer clock be equal to wall clock
time. Set the computer's timezone to UTC, disable NTP, and change it's
time to match your wall clock.

A similar effect can be achieved by massaging the timestamps for rrdtool
update, and running rrdtool graph (or rrdtool fetch) with TZ=utc (or
whatever is needed).

The problem is: what are you going to do when daylight saving starts or
ends. Not an easy task to tackle, but now you still have time to think and
prepare.



_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users