Intermittent data causing stepped graphs

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Intermittent data causing stepped graphs

jamiedg
I've been using RRD for sometime to chart temp sensor data and have found it to be one of the best charting tools , the charts generate in the blink of an eye and use only minimal resources.
Recently though I've hit a problem that maybe someone can help with;
My temp sensors would normally save data at least once a min which worked fine, but to extend battery life the newer sensors are only transmitting data when the temp changes or at most every hour. This causes intermittent data causing the charts to sometimes have a stepped output.

RRD temp over 6H

When I post the same data to the cloud service EmonCMS I get a more smoothed output.

EmonCMS Temp over 6H

Because the data could change every 30s, I have the step at 30 but the heartbeat at 3600 and the xff at 20%

See the below rrd create script:

rrdtool create sen3step30.rrd \
--step 30 \
DS:Sen3:GAUGE:3600:U:U \
DS:Temp3:GAUGE:3600:U:U \
RRA:AVERAGE:0.2:1:1440 \
RRA:AVERAGE:0.2:6:1440 \
RRA:AVERAGE:0.2:36:1440 \
RRA:AVERAGE:0.2:144:1440 \
RRA:AVERAGE:0.2:1008:1440

I use the below chart script:
rrdtool graph /var/www/graphs/graph03t.png --start end-6h --height 150 --width 700 --end now --slope-mode --title "Temp for last 6h" --vertical-label Temp --alt-autoscale-max    \
DEF:sen3=/home/pi/cc/sen3step30.rrd:Temp3:AVERAGE    CDEF:cdefsen3=sen3,UN,PREV,sen3,1,*,IF    LINE3:cdefsen3#9A2EFE:'Conservatory Temp'  \

If the data is Undefined I use the Prev value (sen3,UN,PREV,sen3,1,*,IF)

Can anyone please offer any suggestions or help to smooth the stepped graph output.

rrdtool -v = RRDtool 1.4.7  

thanks and regards,
jamie
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Intermittent data causing stepped graphs

Simon Hobson-2
jamiedg <[hidden email]> wrote:

> Recently though I've hit a problem that maybe someone can help with;
> My temp sensors would normally save data at least once a min which worked
> fine, but to extend battery life the newer sensors are only transmitting
> data when the temp changes or at most every hour. This causes intermittent
> data causing the charts to sometimes have a stepped output.
>
> <http://rrd-mailinglists.937164.n2.nabble.com/file/n7583586/Temp_over_6h_RRD.jpg>
>
> When I post the same data to the cloud service EmonCMS I get a more smoothed
> output.
>
> <http://rrd-mailinglists.937164.n2.nabble.com/file/n7583586/Temp_over_6h_EmonCMS.jpg>

The obvious observation is that the latter is wrong FOR THAT DATA. The data you are feeding in doesn't have the data to show a sloping line, so drawing a straight line between the points is strictly speaking incorrect FOR THAT DATA.

Your real problem is that the data itself is stepped. When you say that the sensor only sends data when the temperature changes, it presumably has some quantisation - it appears to be reporting to a resolution of 0.1˚. I suspect that the older sensors are reporting to a higher resolution - eg if it reports to 0.01˚ then you'd not see the steps as they'd be smaller than the pixels on the graph, both in the Y axis due to the resolution, and in the X axis due to the frequent reports.

Do you still have any of the old sensors ? If so then I'd suggest comparing the raw data. My guess is that the old ones report to a higher resolution, which combined with frequent updates will give you a smooth graph. The new ones are almost certainly quantising to a much lower resolution (otherwise there'd be little or no battery saving) which is what's giving you your problem.


> If the data is Undefined I use the Prev value (sen3,UN,PREV,sen3,1,*,IF)

That is probably wrong as well. You are explicitly telling RRD to draw a flat line there.

> Can anyone please offer any suggestions or help to smooth the stepped graph
> output.

Fix the input ?

_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Intermittent data causing stepped graphs

jamiedg
In reply to this post by jamiedg
Thanks for the reply Simon.
I've checked and the old and new sensors and they have the same 0.1 resolution but as mentioned the new sensors only send data intermittently which can be as long as every hour.
I removed the (IF Undefined use Prev value) code but still have same problem.
See below recent example with log data, the second image has an added black line with my expected plot.



As mentioned this problem only happens because the data is so intermittent.

thanks again for your feedback
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Intermittent data causing stepped graphs

jamiedg
In reply to this post by jamiedg
It's not fully accurate but it looks much better:



I used the TREND function over 30mins (1800) which has removed the stepped data.

CDEF:cdefsen3=sen3,1800,TREND  
Loading...