argus processing pcaps

[Carter Bullard]

Hey Michal,
The simplest way, to keep it sane, would be, assuming you use a linux
machine, are in the directory where the files are,  and bash as your shell:

   % for i in *; do echo $i; argus -r $i -w argus.$i; done

This will generate an argus file for each packet file that you have.
This generates argus primitive data, and then when you want to generate
a report, you can process the primitive file to create the metric you're
interested in.

Once your done, read a few of the files using ra(), from the argus-clients
package, to see if the data looks reasonable.  

If the packet files are all from the " observation domain ", then merging the
data to create a single stream of argus files may make sense.  Argus
will try to use the filename to sort on time, so if your timestamp is sortable,
then you can do this:

   % argus -r snort.log* -w - argus.out

Getting the packets into argus in time order is important, so its possible that
the command above will generate warning messages, but hopefully it will
do what you want.  If this doesn't work, no problem, you can use other
programs to merge the multiple argus files from the first strategy.

   % rasort -m stime -r argus.snort.log* -w argus.out

If you want to then organize the data, you can use rasplit() to process the
argus output to generate a time oriented data repository.

   % rasplit -M time 5m -r argus.out -w archive/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S

Now you have flow data that is in time series chunks that make time series
graphs useful, these can be indexed and processed in mysql, can be graphed
quickly, and be searched etc somewhat efficiently.....

Hopefully that will get you started !!!

Carter

On Apr 19, 2013, at 7:29 AM, Michal Purzynski <michal at rsbac.org> wrote:

> On 4/19/13 1:25 PM, Michal Purzynski wrote:
>> Hi,
>> 
>> I've got a bunch of pcap files and would like to generate argus data from them. It's important to keep things sane, so the flows are merged, etc.
>> 
>> How can i achieve this? I guess it's some combination of the argus command in a for loop, feeding data to rasplit, but don't really know where to start.
> Forgot something:
> 
> the pcap files are in a single dir, named snort.log.<unix timestamp> each about 150MB in size.
> 
> And there's a lot of them, a few TB.
>