argus processing pcaps

[Carter Bullard]

Hey Dave,
You are assigning a different srcid for each file, and that may not be appropriate.
If the packets are just timed packet capture files from the same wire, then you
wouldn't want to change the srcid.  But if the files come from different places,
then you are absolutely correct to assign different srcid's.

If you use the -X, then you will need to set the options you want on the command
line to change from the default behavior, or you will want to provide an argus.conf
file.  I sometimes design perl scripts that create the argus.conf file I want on the
fly, and have my calls to argus use those temporary argus.conf files, so the
command lines aren't that long (convenience, ease of use and debugging).

Carter

On Apr 22, 2013, at 8:28 AM, "Dave Edelman" <dedelman at iname.com> wrote:

> Carter,
> 
> I've been doing quite a bit of converting packet captures to Argus flow
> records and I wonder if I may be over thinking the solution. If I omit the
> for loop, I use something like this:
> ($a = the pcap file name, $b = an integer that starts at 1 and increments
> for each file pcap file processed)
> 
> /usr/local/sbin/argus -X -A -Z -R -J -C -m  -e $b -U 2048 -r $a -w
> $inputDir/$a.argus;
> 
> I always use -X to make sure that my argus.conf file doesn't have some
> setting that confuses things. Do I need the AZRJCm flags or are they just
> going to get in the way?
> 
> --Dave
> 
> 
>> -----Original Message-----
>> From: argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu
>> [mailto:argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu]
>> On Behalf Of Carter Bullard
>> Sent: Friday, April 19, 2013 9:08 AM
>> To: Michal Purzynski
>> Cc: argus-info at lists.andrew.cmu.edu
>> Subject: Re: [ARGUS] argus processing pcaps
>> 
>> Hey Michal,
>> The simplest way, to keep it sane, would be, assuming you use a linux
>> machine, are in the directory where the files are,  and bash as your
> shell:
>> 
>>   % for i in *; do echo $i; argus -r $i -w argus.$i; done
>> 
>> This will generate an argus file for each packet file that you have.
>> This generates argus primitive data, and then when you want to generate
>> a report, you can process the primitive file to create the metric you're
>> interested in.
>> 
>> Once your done, read a few of the files using ra(), from the argus-clients
>> package, to see if the data looks reasonable.
>> 
>> If the packet files are all from the " observation domain ", then merging
> the
>> data to create a single stream of argus files may make sense.  Argus
>> will try to use the filename to sort on time, so if your timestamp is
> sortable,
>> then you can do this:
>> 
>>   % argus -r snort.log* -w - argus.out
>> 
>> Getting the packets into argus in time order is important, so its possible
> that
>> the command above will generate warning messages, but hopefully it will
>> do what you want.  If this doesn't work, no problem, you can use other
>> programs to merge the multiple argus files from the first strategy.
>> 
>>   % rasort -m stime -r argus.snort.log* -w argus.out
>> 
>> If you want to then organize the data, you can use rasplit() to process
> the
>> argus output to generate a time oriented data repository.
>> 
>>   % rasplit -M time 5m -r argus.out -w
>> archive/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S
>> 
>> Now you have flow data that is in time series chunks that make time series
>> graphs useful, these can be indexed and processed in mysql, can be graphed
>> quickly, and be searched etc somewhat efficiently.....
>> 
>> Hopefully that will get you started !!!
>> 
>> Carter
>> 
>> On Apr 19, 2013, at 7:29 AM, Michal Purzynski <michal at rsbac.org> wrote:
>> 
>>> On 4/19/13 1:25 PM, Michal Purzynski wrote:
>>>> Hi,
>>>> 
>>>> I've got a bunch of pcap files and would like to generate argus data
> from
>> them. It's important to keep things sane, so the flows are merged, etc.
>>>> 
>>>> How can i achieve this? I guess it's some combination of the argus
>> command in a for loop, feeding data to rasplit, but don't really know
> where
>> to start.
>>> Forgot something:
>>> 
>>> the pcap files are in a single dir, named snort.log.<unix timestamp>
> each
>> about 150MB in size.
>>> 
>>> And there's a lot of them, a few TB.
>>> 
> 
>