Quick question or two about Argus

Tue Jul 31 18:45:40 EDT 2012

OK, just commenting on your racluster() against a set of rasplit() data.

When you aggregate with a modified key ( -m .... ) the logic changes and operations
like directional correction don't occur.  You should consider running racluster() without
a modified key as a first pass, then pipe that into a racluster () that then modifes the key.
This should have a chance of providing similar results.

racluster -R /space/argus/rasplit/07/29 -t 1343587314-1343587914 -w - | racluster  -m saddr daddr dport -w  /space/argus/testing/clusterrecursive.raw - src net 10.0.0.0/8 and dst net not 10.0.0.0/8 or 172.16.0.0/15 and dst port not 53

racluster -r /space/argus/radium/radium.out  -t 1343587314-1343587914 -w - | racluster  -m saddr daddr dport /space/argus/testing/clusterbigfile.raw - src net 10.0.0.0/8 and dst net not 10.0.0.0/8 or 172.16.0.0/15 and dst port not 53

How do those output files compare?  Are you using argus-clients-3.0.6.2 ?

Carter

On Jul 31, 2012, at 3:42 PM, Craig Merchant <cmerchant at responsys.com> wrote:

> I ran the following command against a directory with 10 minute raw files produced by rasplit:
>  
> racluster -R /space/argus/rasplit/07/29 -m saddr daddr dport -t 1343587314-1343587914 -w /space/argus/testing/clusterrecursive.raw - src net 10.0.0.0/8 and dst net not 10.0.0.0/8 or 172.16.0.0/15 and dst port not 53
>  
> The output file was 72597624.
>  
> I ran this search against a single file spanning several days:
>  
> racluster -r /space/argus/radium/radium.out -m saddr daddr dport -t 1343587314-1343587914 -w /space/argus/testing/clusterbigfile.raw - src net 10.0.0.0/8 and dst net not 10.0.0.0/8 or 172.16.0.0/15 and dst port not 53
>  
> The output file was 7068656, almost ten times smaller.
>  
> I’m guessing that we’ve got a lot of flows that rasplit is segmenting, so more records appear?  If I try to run rahisto against a day’s worth of 10 minute files, I get a segmentation fault.  If I specify “-t 1343587314-1343587914”, it doesn’t produce any results.
>  
> I also tried to sort the output from the single big file by avgdur to see how long our flows tend to be:
>  
> rasort -r clusterbigfile.raw -m avgdur
> 12:41:30.279031 sort syntax error. 'avgdur' not supported
>  
> Thx.
>  
> C
>  
> From: Carter Bullard [mailto:carter at qosient.com] 
> Sent: Tuesday, July 31, 2012 6:44 AM
> To: Craig Merchant
> Cc: Argus; Steve Slater; Vishnu Shankar; Paul DeBone
> Subject: Re: Quick question or two about Argus
>  
> Hey Craig,
> My bad, I wasn't thinking last night.  rabins() is not the tool for your situation.  rabins() is a time series tool when you bin based on time, and netflow streams are pretty with regard to the out of order nature of the data and massive hold times ( unless of course you have a lot of time and a lot of memory ).
>  
> The -B option is incredibly important to rabins(), without it, rabins will process data until EOF, so you won't get any output until the file is over or you exit.  -B defines the hold buffer for bin formation,  data whose timestamps are before the ( current time + the hold buffer time ) are rejected when reading real time data.
> 
> 
> As a result, you generally can't use rabins on netflow data, because of the huge hold times and out of order nature of the output.  
> 
> 
> Better to write netflow into a file using rasplit(), and process the netflow data later, after all the data for a given time period has arrived.
> 
> 
> With regard to filenames and rasplit().  Your debug information is saying you have toooooo many ' % ' in your specification, and putting addresses in the output filename may not scale for you.  The total number of output files is limited by the kernel, not by the application.  The error is saying you saw more than x addresses during he last 60 seconds.  I see > 1000 per minute all the time.  Your kernel may be tuned as low as 64 open files.
>  
> Carter
> 
> On Jul 30, 2012, at 10:48 PM, Craig Merchant <cmerchant at responsys.com> wrote:
>> Thanks for the quick response!
>>  
>> rabins
>> I just tried the following test with rabins:
>>  
>> rabins -S localhost -M time 1m -w argus%M &
>>  
>> No bins appeared after several minutes, though when I killed the process, argus%M was created.  It didn’t translate the %M into the decimal minute though.  When connected to a stream, when does rabins write out a bin?  I see the rabins process steadily consuming more memory, but nothing seems to get written to disk. 
>>  
>> I tried the –B 15 switch, but files aren’t being written out either.  I should say that we’re using radium to listen for incoming Cisco netflow data, so maybe rabins isn’t seeing the keep alives that argusd sends out?
>>  
>> ratop
>> Installing ncurses-devel solved the compilation problem.
>>  
>> rasplit
>> I get the following error when running rasplit and using the $saddr field in the path or filename:  ArgusWriteNewLogfile(/space/argus/rasplit/tcp/19/argus.%%T.0.000000.10.raw, 0x2e8ea5e0) fopen Too many open files
>>  
>> $proto gets translated correctly, but not $saddr or $daddr
>>  
>> The command was:
>> rasplit -S localhost -M time 1m -w /space/argus/rasplit/\$proto/%H/argus.\$saddr.%M.raw &
>>  
>> What’s the limit on the amount of open files that rasplit can handle?
>>  
>> A couple more questions…
>>  
>> Does argus use any kind of indexing of the data files so that when client applications search recursively, they know they don’t have to scan an entire file because a particular saddr or dport doesn’t exist in the data?
>>  
>> Is it possible to make rasplit output data by netblock or a label?
>>  
>> Thanks!
>>  
>> Craig
>>  
>> From: Carter Bullard [mailto:carter at qosient.com] 
>> Sent: Monday, July 30, 2012 5:32 PM
>> To: Craig Merchant
>> Cc: Argus
>> Subject: Re: Quick question or two about Argus
>>  
>> Hey Craig,
>> Answers in line.
>> Carter
>>  
>> On Jul 30, 2012, at 7:38 PM, Craig Merchant <cmerchant at responsys.com> wrote:
>> 
>> 
>> 
>> Hi.  My name is Craig Merchant.  I’m in charge of implementing a security monitoring solution using Argus.  I’ve got a few questions for you if you’ve got a second…
>>  
>> 1.       The documentation says rabins supports time prefixes in the output file name (%Y,%m, etc.).  In my experience on CentOS, it writes those as a literal string.
>>  
>> Not sure how you're calling rabins, such that the strftime() directives in the output file are not being done.
>> Just in case, you do need to indicate the time period, so that the logic knows to do the substitution.
>>  
>>    rabins -M time 1d -w argus_%Y_%m_%d
>>  
>> 2.       Is there any benefit to having rabins forward the data from the sensor to radium in a multi-sensor environment?
>>  
>> The preferred strategy would be to do the rabins() at the end of the data flow.  So it could be
>> argus -> radium -> .... -> rabins
>> 
>> 
>> 
>> 3.       Ratop complains that it wasn’t compiled with ncurses support (ncurses-5.5-24.20060715 is installed).  –M nocurses works, but I get the impression that’s not very efficient…  I tried the “make clobber” step you recommended, but no luck.
>>  
>> When you run ./configure, the output will state if curses was found in the search paths.  You can search
>> the file ./include/argus-config.h to see what its saying about CURSES.  If its defined, then you won't get
>> the error message.   ratop without the curses window isn't really very gratifying, so try to figure out why
>> ./configure isn't finding ncurses.
>> 
>> 
>> 
>> 4.       What other ra commands/fields/switches does ratop support?
>>  
>> ratop is an aggregator, so it does everything that racluster() can do, it can sort, so it does all that rasort() does, and it can print every field that ra* programs support, and a few others.  Type ' :h ' and you'll get a help screen.  If that doesn't do it for you, send some more email, and we'll add to that screen.
>> 
>> 
>> 
>> 5.       When I tried following the video for ratop, I see you specify “-M rmon”.  When I tried that, I get:  ArgusClientInit: ArgusNewAggregator error
>>  
>> When you report a bug, we need at least how it was called, to see what the issues might be.
>> 
>> 
>> 
>> 6.       Using rasplit and specifying $saddr  in the output file path ends up with a directory named %T.00000.  Same with $daddr
>>  
>>  
>> This error is sometimes caused by the shell, which wants to interpret the ' $ ' as special.  Escape the ' $ ' using ' \$ '.
>> 
>> 
>> 
>> Thanks!
>>  
>> Hopefully this is helpful!!
>>  
>> 
>> 
>> 
>> Craig
>>  
>> Carter
>>  

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20120731/115c8e73/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2589 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20120731/115c8e73/attachment.bin>