Image optimizing

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Image optimizing

Jordy Goldstein
I have about 5000 rrd files that I update every 10 minutes. I don't really have
a problem updating the files in that amount of time, but I have to make 12
graphs/file - one set for each of 3 variables showing the data over the last
day, week, month, and year. This obviously is an extremely time and memory
consuming process and I was wondering if there was any way to cut down on the
amount of time needed for this.
 

Basic algorithm for the script [it's in perl] is:

 

1)       Open the CSV file where I get my data from to figure out the name of
each file

2)       Open a file that just contains a number in it that I use to track how
many times the script has been run

3)       Update all the last-24-hour graphs because they should be updated every
time new data is collected (10 min)

4)       Every third script (ideally 30 min) update the weekly graphs, every
12th script [2 hrs] update the monthly graphs, and once a day [every 72nd
script] update the yearly graphs

5)       Increment the number in the file

 

Obviously, the script does not run anywhere near 10 minutes. I'm open to all
suggestions - I'm not sure if maybe the font change, or the interlaced option,
or anything else makes the graphs take  longer to generate. At this point I
don't care much about the quality.

 

 

 

The code:

 

 

[code]

 

print time()."\n";    

$rrd = 'C:\RRDTool\RRDTool\Release\rrdtool.exe';

 

open(INFO, "<Y:\obscurrent.csv") || die('cannot open file: ' . $!);

@lines = <INFO>;

close(INFO);

 

open(INFO, "<numUpdated.txt") || die('cannot open file: ' . $!);

@lines2 = <INFO>;

close(INFO);

 

$updatedLast = $lines2[0];

 

foreach $station (@lines){

 

@updated = split(/,/,$station);

 

$cmd= $rrd.' graph images/'.$updated[0].'tempLAST24.png --interlaced -n
DEFAULT:0:C:/WINDOWS/Fonts/ARIAL.TTF -a PNG -X 0 -v Degrees
DEF:theLine=rrd\\'.$updated[0].'.rrd:temp:AVERAGE
LINE1:theLine#FF0000:"Temperature - Last 24 Hours" --start '.(time() - 86400).'
--end N -A';

`$cmd`;

$cmd= $rrd.' graph images/'.$updated[0].'windLAST24.png --interlaced -n
DEFAULT:0:C:/WINDOWS/Fonts/ARIAL.TTF -a PNG -X 0 -v MPH
DEF:theLine=rrd\\'.$updated[0].'.rrd:wind:AVERAGE
LINE1:theLine#FF0000:"Windspeed - Last 24 Hours" --start '.(time() - 86400).'
--end N -A';

`$cmd`;

$cmd= $rrd.' graph images/'.$updated[0].'humidLAST24.png --interlaced -n
DEFAULT:0:C:/WINDOWS/Fonts/ARIAL.TTF -a PNG -X 0 -v Percent
DEF:theLine=rrd\\'.$updated[0].'.rrd:humid:AVERAGE
LINE1:theLine#FF0000:"Humidity - Last 24 Hours" --start '.(time() - 86400).'
--end N -A';

`$cmd`;

 

 

#*******************************************************************************
*************************

#                      UPDATE WEEKLY EVERY 3 TIMES

#*******************************************************************************
*************************

 

if ($updatedLast% 3 == 0){

            $cmd= $rrd.' graph images/'.$updated[0].'tempLASTWEEK.png
--interlaced -n DEFAULT:0:C:/WINDOWS/Fonts/ARIAL.TTF -a PNG -X 0 -v Degrees
DEF:theLine=rrd\\'.$updated[0].'.rrd:temp:AVERAGE
LINE1:theLine#FF0000:"Temperature - Last Week" --start '.(time() - 604800).'
--end N -A';

            `$cmd`;

            $cmd= $rrd.' graph images/'.$updated[0].'windLASTWEEK.png
--interlaced -n DEFAULT:0:C:/WINDOWS/Fonts/ARIAL.TTF -a PNG -X 0 -v MPH
DEF:theLine=rrd\\'.$updated[0].'.rrd:wind:AVERAGE
LINE1:theLine#FF0000:"Windspeed - Last Week" --start '.(time() - 604800).' --end
N -A';

            `$cmd`;

            $cmd= $rrd.' graph images/'.$updated[0].'humidLASTWEEK.png
--interlaced -n DEFAULT:0:C:/WINDOWS/Fonts/ARIAL.TTF -a PNG -X 0 -v Percent
DEF:theLine=rrd\\'.$updated[0].'.rrd:humid:AVERAGE
LINE1:theLine#FF0000:"Humidity - Last Week" --start '.(time() - 604800).' --end
N -A';

            `$cmd`;

 

}

 

#*******************************************************************************
*************************

#                      UPDATE MONTHLY EVERY 12 TIMES

#*******************************************************************************
*************************

 

           

if ($updatedLast% 12 == 0){

            $cmd= $rrd.' graph images/'.$updated[0].'tempLASTmonth.png
--interlaced -n DEFAULT:0:C:/WINDOWS/Fonts/ARIAL.TTF -a PNG -X 0 -v Degrees
DEF:theLine=rrd\\'.$updated[0].'.rrd:temp:AVERAGE
LINE1:theLine#FF0000:"Temperature - Last Month" --start '.(time() - 2419200).'
--end N -A';

            `$cmd`;

            $cmd= $rrd.' graph images/'.$updated[0].'windLASTmonth.png
--interlaced -n DEFAULT:0:C:/WINDOWS/Fonts/ARIAL.TTF -a PNG -X 0 -v MPH
DEF:theLine=rrd\\'.$updated[0].'.rrd:wind:AVERAGE
LINE1:theLine#FF0000:"Windspeed - Last Month" --start '.(time() - 2419200).'
--end N -A';

            `$cmd`;

            $cmd= $rrd.' graph images/'.$updated[0].'humidLASTmonth.png
--interlaced -n DEFAULT:0:C:/WINDOWS/Fonts/ARIAL.TTF -a PNG -X 0 -v Percent
DEF:theLine=rrd\\'.$updated[0].'.rrd:humid:AVERAGE
LINE1:theLine#FF0000:"Humidity - Last Month" --start '.(time() - 2419200).'
--end N -A';

            `$cmd`;

 

}

 

#*******************************************************************************
*************************

#                      UPDATE YEARLY EVERY 72 TIMES

#*******************************************************************************
*************************

           

if ($updatedLast% 72 == 0){

            $cmd= $rrd.' graph images/'.$updated[0].'tempLASTyear.png
--interlaced -n DEFAULT:0:C:/WINDOWS/Fonts/ARIAL.TTF -a PNG -X 0 -v Degrees
DEF:theLine=rrd\\'.$updated[0].'.rrd:temp:AVERAGE
LINE1:theLine#FF0000:"Temperature - Last Year" --start '.(time() - 31536000).'
--end N -A';

            `$cmd`;

            $cmd= $rrd.' graph images/'.$updated[0].'windLASTyear.png
--interlaced -n DEFAULT:0:C:/WINDOWS/Fonts/ARIAL.TTF -a PNG -X 0 -v MPH
DEF:theLine=rrd\\'.$updated[0].'.rrd:wind:AVERAGE
LINE1:theLine#FF0000:"Windspeed - Last Year" --start '.(time() - 31536000).'
--end N -A';

            `$cmd`;

            $cmd= $rrd.' graph images/'.$updated[0].'humidLASTyear.png
--interlaced -n DEFAULT:0:C:/WINDOWS/Fonts/ARIAL.TTF -a PNG -X 0 -v Percent
DEF:theLine=rrd\\'.$updated[0].'.rrd:humid:AVERAGE
LINE1:theLine#FF0000:"Humidity - Last Year" --start '.(time() - 31536000).'
--end N -A';

            `$cmd`;

 

}

 

}

 

$file = 'numUpdated.txt';            

open(INFO, ">$file");                  

print INFO ($updated+1);                        

close(INFO);

 

print time();

 

 

 

 

[/code]

This message (including any attachments) is a confidential and privileged communication of
AWS Convergence Technologies, Inc. and intended only for the addressee. Any unauthorized use,
distribution or copying of this message (or any attachment) is prohibited. If you are not the
addressee or a person authorized to receive messages for the addressee, you have received this
message in error. In that case, please delete this message and call us at 301-250-4000 so that we
can correct our records in order to avoid this mistake in the future. Thank you.

--
Unsubscribe mailto:[hidden email]?subject=unsubscribe
Help        mailto:[hidden email]?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi

Reply | Threaded
Open this post in threaded view
|

Re: Image optimizing

Haroon Rafique
On Today at 10:57am, JG=>Jordy Goldstein <[hidden email]> wrote:

JG> I have about 5000 rrd files that I update every 10 minutes. I don't
JG> really have a problem updating the files in that amount of time, but I
JG> have to make 12 graphs/file - one set for each of 3 variables showing
JG> the data over the last day, week, month, and year. This obviously is
JG> an extremely time and memory consuming process and I was wondering if
JG> there was any way to cut down on the amount of time needed for this.
JG>  

Use an on-demand grapher. I wrote one at:

http://haroon.sis.utoronto.ca/rrd/scripts/

There should be plenty other at RRD World:

http://oss.oetiker.ch/rrdtool/rrdworld/index.en.html

JG>
JG> [..snip..]
JG>

Regards,
--
Haroon Rafique
<[hidden email]>

--
Unsubscribe mailto:[hidden email]?subject=unsubscribe
Help        mailto:[hidden email]?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi

Reply | Threaded
Open this post in threaded view
|

Re: Image optimizing

Christian, Ed
In reply to this post by Jordy Goldstein
Two comments/suggestions:

>
> 4)       Every third script (ideally 30 min) update the
> weekly graphs, every
> 12th script [2 hrs] update the monthly graphs, and once a day
> [every 72nd
> script] update the yearly graphs
>


Why do all this work/math and not use the --lazy option built into
RRDtool?

    [-z|--lazy]

    Only generate the graph if the current graph is out of date or not
existent.



> $cmd= $rrd.' graph images/'.$updated[0].'tempLAST24.png
> --interlaced -n
> DEFAULT:0:C:/WINDOWS/Fonts/ARIAL.TTF -a PNG -X 0 -v Degrees
> DEF:theLine=rrd\\'.$updated[0].'.rrd:temp:AVERAGE
> LINE1:theLine#FF0000:"Temperature - Last 24 Hours" --start
> '.(time() - 86400).'
> --end N -A';
>
> `$cmd`;
>

Why use backticks to shell out of your Perl code instead of using the
RRDs.pm perl module?

use RRDs;
RRDs::graph(
        'images/'.$updated[0].'tempLAST24.png',
        '--interlaced',
        '--font=DEFAULT:0:C:/WINDOWS/Fonts/ARIAL.TTF',
        :
        :
        :
);


The suggestions may not save you much, but over ~12k iterations you may
find it'll save you enough not to have to write anything fancier.

--
Unsubscribe mailto:[hidden email]?subject=unsubscribe
Help        mailto:[hidden email]?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi

Reply | Threaded
Open this post in threaded view
|

Re: Image optimizing

Jason Fesler
In reply to this post by Jordy Goldstein
> I have about 5000 rrd files that I update every 10 minutes. I don't really have
> a problem updating the files in that amount of time, but I have to make 12
> graphs/file - one set for each of 3 variables showing the data over the last
> day, week, month, and year. This obviously is an extremely time and memory
> consuming process and I was wondering if there was any way to cut down on the
> amount of time needed for this.

Consider generating them on demand only.  The CPU and I/O savings will
probaly make up for the havig to wait for the graph to generate :-).  I'm
doing that with hundreds of thousands of RRD files now.  I also have a 60
second cache, so if multiple parties are hitting the same graph parameters
at the same time (responding to some event) the servers doesn't fall over
on their arses.

--
Unsubscribe mailto:[hidden email]?subject=unsubscribe
Help        mailto:[hidden email]?subject=help
Archive     http://lists.ee.ethz.ch/rrd-users
WebAdmin    http://lists.ee.ethz.ch/lsg2.cgi