argus processing pcaps

[Dave Edelman]

Carter,

In each instance the files are from different places eg: a wide -area load
balanced website would have two or more sources of packet capture. I think
that the reason that I started to use the -X was occasional trouble with the
-n parameter. By using -X I knew the starting point for the -n so I could be
sure of what was being translated and what would appear untranslated. Is
there a solution looking for a problem similar to allowing an optional value
for -n to fix the setting regardless of the .rarc file?

--Dave

> -----Original Message-----
> From: Carter Bullard [mailto:carter at qosient.com]
> Sent: Monday, April 22, 2013 11:16 AM
> To: Dave Edelman
> Cc: 'Michal Purzynski'; argus-info at lists.andrew.cmu.edu
> Subject: Re: [ARGUS] argus processing pcaps
> 
> Hey Dave,
> You are assigning a different srcid for each file, and that may not be
> appropriate.
> If the packets are just timed packet capture files from the same wire,
then
> you
> wouldn't want to change the srcid.  But if the files come from different
> places,
> then you are absolutely correct to assign different srcid's.
> 
> If you use the -X, then you will need to set the options you want on the
> command
> line to change from the default behavior, or you will want to provide an
> argus.conf
> file.  I sometimes design perl scripts that create the argus.conf file I
want on
> the
> fly, and have my calls to argus use those temporary argus.conf files, so
the
> command lines aren't that long (convenience, ease of use and debugging).
> 
> Carter
> 
> On Apr 22, 2013, at 8:28 AM, "Dave Edelman" <dedelman at iname.com>
> wrote:
> 
> > Carter,
> >
> > I've been doing quite a bit of converting packet captures to Argus flow
> > records and I wonder if I may be over thinking the solution. If I omit
the
> > for loop, I use something like this:
> > ($a = the pcap file name, $b = an integer that starts at 1 and
increments
> > for each file pcap file processed)
> >
> > /usr/local/sbin/argus -X -A -Z -R -J -C -m  -e $b -U 2048 -r $a -w
> > $inputDir/$a.argus;
> >
> > I always use -X to make sure that my argus.conf file doesn't have some
> > setting that confuses things. Do I need the AZRJCm flags or are they
just
> > going to get in the way?
> >
> > --Dave
> >
> >
> >> -----Original Message-----
> >> From: argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu
> >> [mailto:argus-info-
> bounces+dedelman=iname.com at lists.andrew.cmu.edu]
> >> On Behalf Of Carter Bullard
> >> Sent: Friday, April 19, 2013 9:08 AM
> >> To: Michal Purzynski
> >> Cc: argus-info at lists.andrew.cmu.edu
> >> Subject: Re: [ARGUS] argus processing pcaps
> >>
> >> Hey Michal,
> >> The simplest way, to keep it sane, would be, assuming you use a linux
> >> machine, are in the directory where the files are,  and bash as your
> > shell:
> >>
> >>   % for i in *; do echo $i; argus -r $i -w argus.$i; done
> >>
> >> This will generate an argus file for each packet file that you have.
> >> This generates argus primitive data, and then when you want to generate
> >> a report, you can process the primitive file to create the metric
you're
> >> interested in.
> >>
> >> Once your done, read a few of the files using ra(), from the
argus-clients
> >> package, to see if the data looks reasonable.
> >>
> >> If the packet files are all from the " observation domain ", then
merging
> > the
> >> data to create a single stream of argus files may make sense.  Argus
> >> will try to use the filename to sort on time, so if your timestamp is
> > sortable,
> >> then you can do this:
> >>
> >>   % argus -r snort.log* -w - argus.out
> >>
> >> Getting the packets into argus in time order is important, so its
possible
> > that
> >> the command above will generate warning messages, but hopefully it will
> >> do what you want.  If this doesn't work, no problem, you can use other
> >> programs to merge the multiple argus files from the first strategy.
> >>
> >>   % rasort -m stime -r argus.snort.log* -w argus.out
> >>
> >> If you want to then organize the data, you can use rasplit() to process
> > the
> >> argus output to generate a time oriented data repository.
> >>
> >>   % rasplit -M time 5m -r argus.out -w
> >> archive/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S
> >>
> >> Now you have flow data that is in time series chunks that make time
series
> >> graphs useful, these can be indexed and processed in mysql, can be
> graphed
> >> quickly, and be searched etc somewhat efficiently.....
> >>
> >> Hopefully that will get you started !!!
> >>
> >> Carter
> >>
> >> On Apr 19, 2013, at 7:29 AM, Michal Purzynski <michal at rsbac.org> wrote:
> >>
> >>> On 4/19/13 1:25 PM, Michal Purzynski wrote:
> >>>> Hi,
> >>>>
> >>>> I've got a bunch of pcap files and would like to generate argus data
> > from
> >> them. It's important to keep things sane, so the flows are merged, etc.
> >>>>
> >>>> How can i achieve this? I guess it's some combination of the argus
> >> command in a for loop, feeding data to rasplit, but don't really know
> > where
> >> to start.
> >>> Forgot something:
> >>>
> >>> the pcap files are in a single dir, named snort.log.<unix timestamp>
> > each
> >> about 150MB in size.
> >>>
> >>> And there's a lot of them, a few TB.
> >>>
> >
> >