argus processing pcaps

[Dave Edelman]

Carter,

I've been doing quite a bit of converting packet captures to Argus flow
records and I wonder if I may be over thinking the solution. If I omit the
for loop, I use something like this:
 ($a = the pcap file name, $b = an integer that starts at 1 and increments
for each file pcap file processed)

/usr/local/sbin/argus -X -A -Z -R -J -C -m  -e $b -U 2048 -r $a -w
$inputDir/$a.argus;

I always use -X to make sure that my argus.conf file doesn't have some
setting that confuses things. Do I need the AZRJCm flags or are they just
going to get in the way?

--Dave


> -----Original Message-----
> From: argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu
> [mailto:argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu]
> On Behalf Of Carter Bullard
> Sent: Friday, April 19, 2013 9:08 AM
> To: Michal Purzynski
> Cc: argus-info at lists.andrew.cmu.edu
> Subject: Re: [ARGUS] argus processing pcaps
> 
> Hey Michal,
> The simplest way, to keep it sane, would be, assuming you use a linux
> machine, are in the directory where the files are,  and bash as your
shell:
> 
>    % for i in *; do echo $i; argus -r $i -w argus.$i; done
> 
> This will generate an argus file for each packet file that you have.
> This generates argus primitive data, and then when you want to generate
> a report, you can process the primitive file to create the metric you're
> interested in.
> 
> Once your done, read a few of the files using ra(), from the argus-clients
> package, to see if the data looks reasonable.
> 
> If the packet files are all from the " observation domain ", then merging
the
> data to create a single stream of argus files may make sense.  Argus
> will try to use the filename to sort on time, so if your timestamp is
sortable,
> then you can do this:
> 
>    % argus -r snort.log* -w - argus.out
> 
> Getting the packets into argus in time order is important, so its possible
that
> the command above will generate warning messages, but hopefully it will
> do what you want.  If this doesn't work, no problem, you can use other
> programs to merge the multiple argus files from the first strategy.
> 
>    % rasort -m stime -r argus.snort.log* -w argus.out
> 
> If you want to then organize the data, you can use rasplit() to process
the
> argus output to generate a time oriented data repository.
> 
>    % rasplit -M time 5m -r argus.out -w
> archive/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S
> 
> Now you have flow data that is in time series chunks that make time series
> graphs useful, these can be indexed and processed in mysql, can be graphed
> quickly, and be searched etc somewhat efficiently.....
> 
> Hopefully that will get you started !!!
> 
> Carter
> 
> On Apr 19, 2013, at 7:29 AM, Michal Purzynski <michal at rsbac.org> wrote:
> 
> > On 4/19/13 1:25 PM, Michal Purzynski wrote:
> >> Hi,
> >>
> >> I've got a bunch of pcap files and would like to generate argus data
from
> them. It's important to keep things sane, so the flows are merged, etc.
> >>
> >> How can i achieve this? I guess it's some combination of the argus
> command in a for loop, feeding data to rasplit, but don't really know
where
> to start.
> > Forgot something:
> >
> > the pcap files are in a single dir, named snort.log.<unix timestamp>
each
> about 150MB in size.
> >
> > And there's a lot of them, a few TB.
> >