beginner questions on multiple series

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

beginner questions on multiple series

SecretCode
Hi ... I have done some basic work with rrdtool but I'm still a beginner.

I'm trying to report on traffic usage through a firewall for multiple IP
addresses. Each address will have in & out values, but I don't know in
advance how many addresses there will be - and new addresses could
theoretically be added during any collection interval.

Am I right that each IP address's traffic would be a separate series and
would have to be a separate RRD file (because each RRD has a sized fixed
up-front)? Is this going to be manageable with rrdtool?

thanks
Joe


Reply | Threaded
Open this post in threaded view
|

Re: beginner questions on multiple series

Jason Fesler
> Am I right that each IP address's traffic would be a separate series and
> would have to be a separate RRD file (because each RRD has a sized fixed
> up-front)? Is this going to be manageable with rrdtool?

Yep, and maybe.  "managable" is relative to the resources you have at
hand, and how many you plan on tracking.


Reply | Threaded
Open this post in threaded view
|

Re: beginner questions on multiple series

Alex van den Bogaerdt-2
In reply to this post by SecretCode
On Fri, Jul 04, 2008 at 12:19:48AM +0200, SecretCode wrote:

> Hi ... I have done some basic work with rrdtool but I'm still a beginner.
>
> I'm trying to report on traffic usage through a firewall for multiple IP
> addresses. Each address will have in & out values, but I don't know in
> advance how many addresses there will be - and new addresses could
> theoretically be added during any collection interval.
>
> Am I right that each IP address's traffic would be a separate series and
> would have to be a separate RRD file (because each RRD has a sized fixed
> up-front)? Is this going to be manageable with rrdtool?

Theoretically you could have multiple IP addresses in one RRD. There
are two viable scenario's here:

-1- you could have one RRD per network (e.g. per class C)
-2- you could have a number (any number) of 'buckets' per RRD, and
    keep track of which IP address goes in which bucket of which RRD

But be sure to understand that this means you need to update all
IP addresses in the same RRD file at the same time!

That's why generally speaking each IP address should get its own RRD:
it makes life easier.


Does RRDtool manage any of this?  No, not at all. You need a front-end
to handle that kind of stuff, or you build something yourself.
Same for collecting the data. RRDtool won't do that either.

Tobi's site has links to front-ends.

HTH
--
Alex van den Bogaerdt
http://www.vandenbogaerdt.nl/rrdtool/


Reply | Threaded
Open this post in threaded view
|

Re: beginner questions on multiple series

Raimund Berger
In reply to this post by SecretCode
SecretCode <[hidden email]> writes:

> Hi ... I have done some basic work with rrdtool but I'm still a beginner.
>
> I'm trying to report on traffic usage through a firewall for multiple IP
> addresses. Each address will have in & out values, but I don't know in
> advance how many addresses there will be - and new addresses could
> theoretically be added during any collection interval.
>
> Am I right that each IP address's traffic would be a separate series and
> would have to be a separate RRD file (because each RRD has a sized fixed
> up-front)? Is this going to be manageable with rrdtool?

The requirement of
* tracking each ip individually and
* allowing new ip's to be added at run time
already implies that each ip has to have it's own rrd, because you
can't just add a new "value" field to an exisiting rrd.

I'd say this sure can be handled by rrdtool, but depending on what
data you want to collect it might put some load on your system,
especially the disk(s) as has been previously discussed on this list.

More specifically, you didn't say if you want to track e.g.
* tcp connection requests (syn packets) and/or
* total bandwidth consumption of an ip and/or
* each single packet going to and fro an ip
where the workload would of course increase noticably in each case.

Also, you'll gonna need a tool to *collect* the data, and which will
do other stuff for you like creating a new rrd for new ip's
automagically at run time.

Assuming we're not talking an enterprise class environment with
hundreds/thousands of ip's sitting inside your network, I guess I can
recommend collectd for these kinds of purposes. You'll find it to be
very much "plug&play", albeit not including a graphing solution yet. A
deficiency which can be worked around though.

Regards,
R.


Reply | Threaded
Open this post in threaded view
|

Re: beginner questions on multiple series

SecretCode
Thanks Raimund, Alex and Jason (off list)!

> SecretCode <[hidden email]> writes:
>> i ... I have done some basic work with rrdtool but I'm still a beginner.
>>
>> I'm trying to report on traffic usage through a firewall for multiple IP
>> addresses. Each address will have in & out values, but I don't know in
>> advance how many addresses there will be - and new addresses could
>> theoretically be added during any collection interval.
>>
>> Am I right that each IP address's traffic would be a separate series and
>> would have to be a separate RRD file (because each RRD has a sized fixed
>> up-front)? Is this going to be manageable with rrdtool?
>>    
To expand on the need: I want to store total bytes per hour (or minute)
per IP address. This collection is already in place, and I'm logging it
to a text file, but of course that leaves me with handling all the
aggregation and truncation of history that rrdtool already solves.

So I will need to manage the allocation of IP addresses to rrds in my
front end, and creating the rrd for a new ip address.

I initially want this for my home network, a mere handful of addresses,
which shouldn't be a size or processing problem (obviously I can't keep
per-minute data for a year). But I'd like to extend it to the office
environment which is much larger. But not enterprise-class, no.

Alex: you mentioned having several buckets in one RRD. Since I'll need
to allow for multiple rrds anyway, is there any advantage in having say
3 rrds with 10 addresses in each rather than 30 rrds? (Which would be
easier to manage.)

thanks again
Joe


Reply | Threaded
Open this post in threaded view
|

Re: beginner questions on multiple series

Alex van den Bogaerdt-2
On Fri, Jul 04, 2008 at 11:28:30AM +0200, SecretCode wrote:
> Thanks Raimund, Alex and Jason (off list)!

I received it on-list and reply on-list.

> I initially want this for my home network, a mere handful of addresses,
> which shouldn't be a size or processing problem (obviously I can't keep
> per-minute data for a year).

Why not?  365 days times 24 hours times 60 minutes = 525600 rows.
8 bytes per DS per row, plus some bytes in the header. That's
a small file of just over 8MB per address if it has two DSes and
525600 rows:  365*24*60*8*2=8409600.

Am I missing something, did I screw up the computation or isn't
it as obvious as you state?

> per-minute data for a year). But I'd like to extend it to the office
> environment which is much larger. But not enterprise-class, no.
>
> Alex: you mentioned having several buckets in one RRD. Since I'll need
> to allow for multiple rrds anyway, is there any advantage in having say
> 3 rrds with 10 addresses in each rather than 30 rrds? (Which would be
> easier to manage.)

You asked: "... have to be a separate RRD file". The answer is: "no."
But that doesn't mean it isn't the most logical thing to do.

Unless you really need to worry about I/O, memory and/or CPU, just go
for the easy method of having one RRD per address.

Only a careful design could benefit from minimizing overhead by
keeping ip addresses together.  And such a design could backfire, for
instance if you only need to have data from one of the DSes, RRDtool
would need to read all 256*2 DSes to present just that one to you.

Most of the time it will be cheaper to add hardware instead of brains.

This said, a bad design cannot be fixed by adding more hardware, so
do think before you act.

--
Alex van den Bogaerdt
http://www.vandenbogaerdt.nl/rrdtool/