Adding updates but fetch shows NaNs

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Adding updates but fetch shows NaNs

Issaac B
I was hoping someone could help me understand how values are computed in RRD.
I got a temp database (create below). I got a loop that adds an update ever ~2 minutes.

rrdtool info says the DS got known values for the last two columns (ext_temperature, ext_humidity). 
Yet when I do a graph/fetch, I'm getting NaNs for the last two columns.

Thanks,
===============

Create:
rrdtool create $HOME/.xdg/data/nest_thermostat.rrd \
--step '200' \
'DS:temperature:GAUGE:600:0:100' \
'DS:humidity:GAUGE:600:0:100' \
'DS:target:GAUGE:600:0:100' \
'DS:mode:GAUGE:600:0:10' \
'DS:state:GAUGE:600:0:10' \
'DS:fan:GAUGE:600:0:1' \
\
'DS:ext_temperature:GAUGE:1500:0:100' \
'DS:ext_humidity:GAUGE:1500:0:100' \
\
'RRA:AVERAGE:0.5:1:2016' \
'RRA:AVERAGE:0.5:1:4032' \
'RRA:AVERAGE:0.5:6:1440' \
'RRA:AVERAGE:0.5:6:4416' \
'RRA:AVERAGE:0.5:12:4392' \
'RRA:AVERAGE:0.5:24:4380' \
'RRA:AVERAGE:0.5:48:4380'

Update logs:
2018-01-22 16:54:40.787444: Updating RRD with N:67:50:65:2:0:0:U:U
2018-01-22 16:54:40.788156: Updating RRD with N:U:U:U:U:U:U:56.5:65
2018-01-22 16:56:11.899996: Updating RRD with N:67:50:65:2:0:0:U:U
2018-01-22 16:56:11.900661: Updating RRD with N:U:U:U:U:U:U:56.5:65

rrdtool info:
$  rrdtool info nest_thermostat.rrd  | grep ext
ds[ext_temperature].index = 6
ds[ext_temperature].type = "GAUGE"
ds[ext_temperature].minimal_heartbeat = 1500
ds[ext_temperature].min = 0.0000000000e+00
ds[ext_temperature].max = 1.0000000000e+02
ds[ext_temperature].last_ds = "56.5"
ds[ext_temperature].value = 3.6216500000e-02
ds[ext_temperature].unknown_sec = 62
ds[ext_humidity].index = 7
ds[ext_humidity].type = "GAUGE"
ds[ext_humidity].minimal_heartbeat = 1500
ds[ext_humidity].min = 0.0000000000e+00
ds[ext_humidity].max = 1.0000000000e+02
ds[ext_humidity].last_ds = "65"
ds[ext_humidity].value = 4.1665000000e-02
ds[ext_humidity].unknown_sec = 62

Fetch:
$  rrdtool fetch nest_thermostat.rrd AVERAGE | tail -n5
1516668600: 6.6830290005e+01 4.9873350750e+01 6.4835355975e+01 1.9949340300e+00 0.0000000000e+00 0.0000000000e+00 -nan -nan
1516668800: 6.6834396100e+01 4.9876415000e+01 6.4839339500e+01 1.9950566000e+00 0.0000000000e+00 0.0000000000e+00 -nan -nan
1516669000: 6.6999548755e+01 4.9999663250e+01 6.4999562225e+01 1.9999865300e+00 0.0000000000e+00 0.0000000000e+00 -nan -nan
1516669200: 6.6910302075e+01 4.9933061250e+01 6.4912979625e+01 1.9973224500e+00 0.0000000000e+00 0.0000000000e+00 -nan -nan
1516669400: -nan -nan -nan -nan -nan -nan -nan -nan

_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
Reply | Threaded
Open this post in threaded view
|

Re: [unsure] Adding updates but fetch shows NaNs

Alex van den Bogaerdt-5
> I was hoping someone could help me understand how values are computed in
> RRD.
> I got a temp database (create below). I got a loop that adds an update
> ever
> ~2 minutes.

120 seconds.

And your interval (database step) is 200 seconds.

> rrdtool info says the DS got known values for the last two
> columns (ext_temperature, ext_humidity).

and also reports part of the interval is unknown.

Guessing here, I did not do all the exercises: my guess is that you always
have part of an interval set to unknown data.

Suggestion to debug, and maybe also to make permanent changes to your
database setup:

1: set your database to step==1
2: the RRAs you already have, collect data 200x the current step size
(e.g. you now have RRA:AVERAGE:0.5:1:2016 this becomes
RRA:AVERAGE:0.5:200:2016 )
3: an extra RRA expects data every 1 step. It won't get it that often, but
that does not matter. Just make sure heartbeat is set high enough.
4: also keep MIN and MAX if so desired

This new, extra RRA aids in debugging. Just make it long enough to look
back an hour or so (3600 seconds). Longer if you wish to keep and use the
data. Disk space is cheap.

Your previous setup: one complete interval was 200 seconds and you needed
at least 2 updates for it to fill. The new proposed setup completes more
than one interval of 1 second each time you update.

If you see weird rates in the first interval of such ranges (rates being
pulled to zero), then your version of RRDtool suffers from the
sub-second-precision bug. Ignore, or make sure you update with integer
timestamps instead of "N" for now.


Last but not least:  if you receive your data as one batch of numbers,
then never mind but if you receive your numbers separate from each other,
then do not combine them in one RRD database.

E.g. update x:1:2:3:4:5:6:U:U followed by update x+1:U:U:U:U:U:U:1:2
messes things up. It will NOT result in only updating the first 6 DSes and
then only the last 2.

HTH
Alex


_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
Reply | Threaded
Open this post in threaded view
|

Re: [unsure] Adding updates but fetch shows NaNs

Issaac B
Thanks, Alex! The last but about the double update did it for me.
I split my Python script into two objects which each did their own rrdupdate and set half the values to U.
By combining the data into a single update, my values are now working just fine. 
I thought that the U would be the same as no-update but apparently not.

On Mon, Jan 22, 2018 at 11:18 PM, Alex van den Bogaerdt <[hidden email]> wrote:
> I was hoping someone could help me understand how values are computed in
> RRD.
> I got a temp database (create below). I got a loop that adds an update
> ever
> ~2 minutes.

120 seconds.

And your interval (database step) is 200 seconds.

> rrdtool info says the DS got known values for the last two
> columns (ext_temperature, ext_humidity).

and also reports part of the interval is unknown.

Guessing here, I did not do all the exercises: my guess is that you always
have part of an interval set to unknown data.

Suggestion to debug, and maybe also to make permanent changes to your
database setup:

1: set your database to step==1
2: the RRAs you already have, collect data 200x the current step size
(e.g. you now have RRA:AVERAGE:0.5:1:2016 this becomes
RRA:AVERAGE:0.5:200:2016 )
3: an extra RRA expects data every 1 step. It won't get it that often, but
that does not matter. Just make sure heartbeat is set high enough.
4: also keep MIN and MAX if so desired

This new, extra RRA aids in debugging. Just make it long enough to look
back an hour or so (3600 seconds). Longer if you wish to keep and use the
data. Disk space is cheap.

Your previous setup: one complete interval was 200 seconds and you needed
at least 2 updates for it to fill. The new proposed setup completes more
than one interval of 1 second each time you update.

If you see weird rates in the first interval of such ranges (rates being
pulled to zero), then your version of RRDtool suffers from the
sub-second-precision bug. Ignore, or make sure you update with integer
timestamps instead of "N" for now.


Last but not least:  if you receive your data as one batch of numbers,
then never mind but if you receive your numbers separate from each other,
then do not combine them in one RRD database.

E.g. update x:1:2:3:4:5:6:U:U followed by update x+1:U:U:U:U:U:U:1:2
messes things up. It will NOT result in only updating the first 6 DSes and
then only the last 2.

HTH
Alex




_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users