racluster memory utilization

[Jason]

Correction: -i should be -u


On Thu, May 22, 2014 at 7:46 PM, Jason <dn1nj4 at gmail.com> wrote:

> (Changing the subject to be relevant to the current conversation)
>
> Carter,
>
> Based on your suggestions below, with 3.0.7.28, I conducted 2 tests
> against 50GB of flow files with the following:
>
> racluster -r <filelist> -i -nn -c"," -m srcid saddr daddr proto dport -Zb
> -s stime saddr daddr proto sport dport sbytes runtime dbytes trans state
>
> On a beefy server, I let this run for 70 minutes before killing it.  In
> that time it consumed 33GB of RAM.
>
> Next, I added the "-f racluster.conf" option with the following
> configuration:
>
> filter="udp and port domain" model="saddr daddr proto sport dport"
> status=0 idle=10
> filter="udp" model="saddr daddr proto sport dport" status=0 idle=60
> filter="" model="saddr daddr proto sport dport" status=0 idle=600
>
> This version (which I was expecting to consume less memory based on
> previous list threads) I killed after 53 minutes with it consuming 39GB of
> RAM (read: less time, more RAM).
>
> So even with your suggested changes, the amount of RAM utilization still
> seems really high.  Are there changes I should make to the racluster.conf
> file to reduce the memory footprint further?  Do you have any kind of
> statistics correlating volume of flow data to volume of memory utilization?
>
> I know you mentioned rasqlinsert, but my performance testing for trying to
> process another large batch of files files indicated the processing
> probably would not finish before the next batch needed to be processed.  So
> I'm thinking that's not really a viable option.
>
> Appreciate all the help.
> Jason
>
> On Thu, May 22, 2014 at 2:33 PM, Carter Bullard <carter at qosient.com>wrote:
>
>> Hey Jason,
>> So you want to do service based tracking on an IP address basis,
>> but you want to track client and server oriented stats.
>>
>> Once you say that you want to track directionality, then
>> the “-M rmon” option is not the correct tool.
>>
>> Tracking single IP addresses and all the ports that they offer
>> is a great way to go, and “-M rmon” is a good way to do that.
>>
>> What do you get with "racluster -m srcid smac saddr sport” ?
>> You get the ethernet, IP address pairings, and any port that
>> is used on that IP address.  This information can answer your
>> server questions.  If the port of interest is in the output,
>> it was used on the that IP address, if there are lots of connections,
>> with traffic, then you may be able to infer that it is a server
>> for that port, but it is not definitive.
>>
>> You should use straight racluster() with a filter that
>> assures that your port operations are valid.
>>
>>    racluster -m srcid saddr daddr proto dport -r file - \(syn or synack\)
>>
>> This will give you TCP flow records where the dport is the service port.
>> You will end up with a list of records that are:
>>
>>    client -> server.serverPort metrics
>>
>>
>> You should get yourself a good racluster.conf file and do a decent job
>> on defining a cluster scheme that really works.
>>
>> Carter
>>
>>
>> On May 22, 2014, at 12:58 PM, Jason <dn1nj4 at gmail.com> wrote:
>>
>> Let me clarify and provide a bit more context...  I expect the following
>> flows:
>>
>> 1.2.3.4:23456 -> 5.6.7.8:34567
>> 1.2.3.4:45678 -> 6.7.8.9:34567
>>
>> To result in the following output data:
>>
>> 1.2.3.4 23456 34567
>> 1.2.3.4 45678 34567
>> 5.6.7.8 34567 23456
>> 6.7.8.9 34567 45678
>>
>> ((in addition to various other stats aggregated with the
>> saddr,sport,dport fields as the key))
>>
>> I'm then taking the above data and doing simplistic port groupings, such
>> as "34567 is (typically) part of the app1 port group" (think 80, 8000, 8080
>> as typically "web").  Then I generate a report that says:
>>
>> 1.2.3.4, client to the app1 port group, X bytes from this client, Y bytes
>> to this client, Z connections from this client
>>
>> 5.6.7.8, server for the app1 port group, X bytes from this server, Y
>> bytes to this server, Z connections to this server
>>
>> 6.7.8.9, server for the app1 port group, X bytes from this server, Y
>> bytes to this server,Z connections to this server
>>
>> This is a gross oversimplification, but is there a better way to do the
>> above?
>>
>> Thanks!
>> Jason
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20140522/48f01219/attachment.html>