How to graph slow changing counters

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

How to graph slow changing counters

Thorsten Erdmann

Hi,

 

I want to graph the data of my energy meter. But it is measuring only in 1kWh raster. So with normal power usage I have every 2..4 hours a step of 1kWh. The meter displays it’s data as kWh, so it gives e.g 600 for the absolute counter value of 600kWh.

I sample the absolute energy every 5 minutes. So I took this as step and 600s as heartbeat.

 

What I get is short peaks with a height which is displayed as 330u over a period of 24h. With longer periods like a week the values get smaller and smaller and there are weird stepping curves. So how can I measure/graph such low resolution measures in a senseful way?

 

This is how I defined the data:

 

                ret = rrdtool.create(rrdname + '.rrd', '--step', '300', '--start', str(starttime),

                    'DS:Gauge:GAUGE:600:U:U',

                    'DS:Counter:COUNTER:600:U:U',

                    'RRA:AVERAGE:0.5:5m:24h',

                    'RRA:AVERAGE:0.5:2h:31d',

                    'RRA:AVERAGE:0.5:6h:31d',

                    'RRA:AVERAGE:0.5:1d:1y')

 

And this is how I graph them

 

    ret = rrdtool.graph(rrdname + '24h.png', '--start', 'end-24h', '--end', 'now',

        '--upper-limit', '650', '--lower-limit', '600', '--rigid', '--right-axis', '0.00001:-0.006',

        'DEF:gauge=./' + rrdname + '.rrd:Gauge:AVERAGE',

        'DEF:counter=./' + rrdname + '.rrd:Counter:AVERAGE',

        'CDEF:cnt1000=600,counter,10000,*,+',

        'LINE1:gauge#00ff00:Gauge',

        'LINE1:cnt1000#ff0000:Counter')

 

The gauge line looks good, but the counter line looks weird.

 

Thanks

Thorsten

 


_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
Reply | Threaded
Open this post in threaded view
|

Re: How to graph slow changing counters

Simon Hobson
Thorsten Erdmann <[hidden email]> wrote:

> I want to graph the data of my energy meter. But it is measuring only in 1kWh raster. So with normal power usage I have every 2..4 hours a step of 1kWh. The meter displays it’s data as kWh, so it gives e.g 600 for the absolute counter value of 600kWh.
> I sample the absolute energy every 5 minutes. So I took this as step and 600s as heartbeat.
>  
> What I get is short peaks with a height which is displayed as 330u over a period of 24h. With longer periods like a week the values get smaller and smaller and there are weird stepping curves. So how can I measure/graph such low resolution measures in a senseful way?

Realistically, there's not a lot you can do with such low resolution data. You'll get a spike each time a unit clocks up, then as you zoom out for a longer scale view, it will flatten out to (as you've found) a rather low value. If you are using a unit every (say) 3 hours (taking a mean between your 2 and 4 hours), then that equates to an average of 333W - or if you are just storing kWH, 0.333kW. Again depending on how you store and calculate, this then equates to around 93µJ/s - at least, if my brain is working right this afternoon !


You don't say how you are interrogating the meter - is it via a data source it provides ?
If the meter is providing you with a counter value when asked, then this will give you the best long term accuracy - your counter will always match what is in the meter.

As an alternative, many meters provide a pulsed output of some sort (even if it's only an LED flashing). A bit like the old Ferraris disk meters with their "X revolutions/kWH", the meter pulses will give you an indication when a much lower quantity of energy has been used - it should say on the face of the meter how many pulses/kWH. The downside to counting these pulses is that if your counter isn't running or otherwise misses them, then you lose sight of energy consumed and your RRD calculations will show a lower figure than the meter.

You could, of course, collect both sources - use the pulses for short term indication (ie better graphing up to a few days duration) and the register count for longer term.
_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
Reply | Threaded
Open this post in threaded view
|

Re: How to graph slow changing counters

Thorsten Erdmann
Hi Simon,

you are right (and I know) that with such a low resolution it is difficult to draw sensefull data. But I want to understand what RRD does. I think the aggregation is wrong:

If I sample the data every 5 minutes (1/12h) and get a single 1kWh step in one of these 5 minute intervals then I get a peak about 0,3kW in the dayly graph. But why? I get headache with these calculation. I would say the formular is 1kWh / (1/12)h but that gives 12kW, which is definitely wrong.

Another weird thing is what happens on the weekly graph. I get some "nice" step lines up and down. Shouldn't it be simply a flattened out curve just as you said. I cannot give you a picture right now, because my server is down for implementing the S0 photosensor, just as you suggested. I will implent both. So I can adjust the S0 pulse counter once a day or so to the real counter value.

Thanks
Thorsten

> -----Ursprüngliche Nachricht-----
> Von: rrd-users [mailto:rrd-users-
> bounces+thorsten=[hidden email]] Im Auftrag von Simon
> Hobson
> Gesendet: Montag, 30. März 2020 14:09
> An: [hidden email] rrdtool
> Betreff: Re: [rrd-users] How to graph slow changing counters
>
> Thorsten Erdmann <[hidden email]> wrote:
>
> > I want to graph the data of my energy meter. But it is measuring only in
> 1kWh raster. So with normal power usage I have every 2..4 hours a step of
> 1kWh. The meter displays it’s data as kWh, so it gives e.g 600 for the absolute
> counter value of 600kWh.
> > I sample the absolute energy every 5 minutes. So I took this as step and
> 600s as heartbeat.
> >
> > What I get is short peaks with a height which is displayed as 330u over a
> period of 24h. With longer periods like a week the values get smaller and
> smaller and there are weird stepping curves. So how can I measure/graph
> such low resolution measures in a senseful way?
>
> Realistically, there's not a lot you can do with such low resolution data. You'll
> get a spike each time a unit clocks up, then as you zoom out for a longer scale
> view, it will flatten out to (as you've found) a rather low value. If you are
> using a unit every (say) 3 hours (taking a mean between your 2 and 4 hours),
> then that equates to an average of 333W - or if you are just storing kWH,
> 0.333kW. Again depending on how you store and calculate, this then equates
> to around 93µJ/s - at least, if my brain is working right this afternoon !
>
>
> You don't say how you are interrogating the meter - is it via a data source it
> provides ?
> If the meter is providing you with a counter value when asked, then this will
> give you the best long term accuracy - your counter will always match what is
> in the meter.
>
> As an alternative, many meters provide a pulsed output of some sort (even if
> it's only an LED flashing). A bit like the old Ferraris disk meters with their "X
> revolutions/kWH", the meter pulses will give you an indication when a much
> lower quantity of energy has been used - it should say on the face of the
> meter how many pulses/kWH. The downside to counting these pulses is that
> if your counter isn't running or otherwise misses them, then you lose sight of
> energy consumed and your RRD calculations will show a lower figure than the
> meter.
>
> You could, of course, collect both sources - use the pulses for short term
> indication (ie better graphing up to a few days duration) and the register
> count for longer term.
> _______________________________________________
> rrd-users mailing list
> [hidden email]
> https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
Reply | Threaded
Open this post in threaded view
|

Re: How to graph slow changing counters

Alex van den Bogaerdt-5
Thorsten Erdmann schreef op 2020-03-30 15:34:

> you are right (and I know) that with such a low resolution it is
> difficult to draw sensefull data. But I want to understand what RRD
> does. I think the aggregation is wrong:
>
> If I sample the data every 5 minutes (1/12h) and get a single 1kWh
> step in one of these 5 minute intervals then I get a peak about 0,3kW
> in the dayly graph. But why? I get headache with these calculation. I
> would say the formular is 1kWh / (1/12)h but that gives 12kW, which is
> definitely wrong.

You feed 'something' to RRDtool and it will process it into 'something
per second'. That's how it works if you feed COUNTER values.

If you get an ever increasing value, as is usually the case from
electricity meters, then COUNTER is indeed the type to use. GAUGE may
provide a pretty (prettier) picture, but it is wrong. Gauge is when you
already have 'something per second'.

Make sure you understand kWh. It is >>>not<<< kilowatt _per_ hour. It is
a kilowatt _during_ an hour. In other words, it is 1000 Joule per
second, during 3600 seconds, thus 3,6 MJ.  If you multiply by 3600000
during graph time (use a CDEF) then you will display J/s aka W.

If you have one 5-minute slot filled with 1 kWh, it is 1/300 kWh/s so I
expect to see a number close to 0,003333333... or 3m3333.... It may be a
bit different due to normalization. You also get a lot of slots with 0
or near 0. Over time this should average out. 12 slots, one having a
rate of 0,003333... and the rest 0, would average out to a rate of
0,00027777... valid during an hour. Yes, this is smaller, but that is to
be expected. After all, you did not use a kWh during 5 minutes, you did
it during an hour (or even longer amount of time). Dividing that same
kWh by 1 or even 2 hours does correctly result in a smaller average
rate.

If RRDtool is doing its job properly, and if your estimate of one 1 kWh
every few hours is correct, then you would end up with a few kWh a day,
still not many a week. You would get weird combinations such as
"mkWh/s". Sounds wrong, but it is, in a way, correct.  Again: multiply
by 3,6M to get J/s, or Watt. Perhaps this makes more sense.

You start with 'weird jumps' because you have 1 interval filled with a
huge rate and then many intervals with zero rate. Over time this should
average out.

Example:

{1/300 0 0 0 0 0 0 0 0 0 0 0} could average out to 1/3600. The next
consolidated interval may look lke this:
{1/300 0 0 0 0 0 0 0 0 0 0 1/300} and averaged out it is 2/3600. Weird
peak? Not really, it makes sense.

Your real world will of course look much more complex than this
simplified example.

In the month graph, looking at days at a time, I expect to see less
jumping around. If not, try another version of RRDtool.

HTH
Alex

_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
Reply | Threaded
Open this post in threaded view
|

Re: How to graph slow changing counters

Simon Hobson
In reply to this post by Thorsten Erdmann
Thorsten Erdmann <[hidden email]> wrote:

> you are right (and I know) that with such a low resolution it is difficult to draw sensefull data. But I want to understand what RRD does. I think the aggregation is wrong:
>
> If I sample the data every 5 minutes (1/12h) and get a single 1kWh step in one of these 5 minute intervals then I get a peak about 0,3kW in the dayly graph. But why? I get headache with these calculation. I would say the formular is 1kWh / (1/12)h but that gives 12kW, which is definitely wrong.
>
> Another weird thing is what happens on the weekly graph. I get some "nice" step lines up and down. Shouldn't it be simply a flattened out curve just as you said. I cannot give you a picture right now, because my server is down for implementing the S0 photosensor, just as you suggested. I will implent both. So I can adjust the S0 pulse counter once a day or so to the real counter value.



> This is how I defined the data:
>  
>                 ret = rrdtool.create(rrdname + '.rrd', '--step', '300', '--start', str(starttime),
>                     'DS:Gauge:GAUGE:600:U:U',
>                     'DS:Counter:COUNTER:600:U:U',
>                     'RRA:AVERAGE:0.5:5m:24h',
>                     'RRA:AVERAGE:0.5:2h:31d',
>                     'RRA:AVERAGE:0.5:6h:31d',
>                     'RRA:AVERAGE:0.5:1d:1y')
>  
> And this is how I graph them
>  
>     ret = rrdtool.graph(rrdname + '24h.png', '--start', 'end-24h', '--end', 'now',
>         '--upper-limit', '650', '--lower-limit', '600', '--rigid', '--right-axis', '0.00001:-0.006',
>         'DEF:gauge=./' + rrdname + '.rrd:Gauge:AVERAGE',
>         'DEF:counter=./' + rrdname + '.rrd:Counter:AVERAGE',
>         'CDEF:cnt1000=600,counter,10000,*,+',
>         'LINE1:gauge#00ff00:Gauge',
>         'LINE1:cnt1000#ff0000:Counter')
>  
> The gauge line looks good, but the counter line looks weird.


Looking again I have some questions and observations ...

You have a GAUGE and a COUNTER in your rrd - what are you collecting, and how ? Counter would be correct for the meter, what are you putting in the gauge ?
Knowing this may affect the advice offered.

I'd recommend you store a bit more data than you are (e.g. 25h for daily graphs) - and that links in with graphing.
As you've written them here, I think you'll find the wrong aggregate data being used. If you do "now-24h" to "now" then you'll find that the first data point in that period is just outside your 24 hours @ 5 minute - so instead you'll find that the 2 hour aggregation is used (see more below). For that reason I've always calculated the exact end time of the last period - i.e. in bash, something like :

step=300
durn=87600
etime=`date +%s`
etime=$((${etime}/${step}*${step}))
stime=$((${etime}-${durn}))
... rrdtool ... "--start ${stime} --end ${etime} --step ${step} ...

That way, your strt and end times are aligned to the bucket boundaries of the RRD file, and you've explicitly defined the data resolution to be used. This gives more reliable selection of data !
I tend to set step and durn in a case statement, along the lines of :
case period in
  daily ( step=300;durn=87600 );;
  weekly ( step-7200; ...
and then the graphing command is common to all resolutions.


A bit more on how RRDtool selects data. Bear in mind this is from memory which may have faded a bit ...
If you don't tell it otherwise, it will select the highest resolution data that supplies the entire graphing period selected. So selecting "now" as the end point, you will be selecting an incomplete bucket at the end of the graph. From memory, I think this also means that "now - 24h" goes part of a bucket past the first bucket in the RRA - because when the current incomplete bucket is started, it takes the place of the one from 24h ago.
As this goes beyond the 300s stored data, the next resolution is selected - hence you day graph that should show 300s buckets, is actually drawn with 2hr buckets - so coarser data AND potentially a gap at the end (e.g. if "now" is 11:55 UTC, you'll only see data up to 10:00 UTC because the 10:00 to 12:00 UTC bucket is not yet complete).


Simon

_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users