argus processing pcaps

[Carter Bullard]

Hey Dave,
Well, the idea of having a command line option that cycles through options,
rather than setting specific options maybe unfamiliar to some (or most).

The principal idea for argus clients is that you use the system rarc files
and  your own personal rarc files to set variables and options.  And then use
the command line options to override your custom settings.  If you've
set your rarc files, then the -n option is easy to work with.  If you don't have
any rarc files, then the default behavior is well defined, and the -n is easy
to work with, so I'm not thinking that there is a problem.

I think you should have rarc files specific for these specific commands,
and not worry about the command line, but the command line is faster.

A preferred solution for your scenario would be to add a " -M names=value "
like option that will set the value to a specific value, like the rarc files do.
If you put a " -n " after that, then it will cycle past your set value.
I can add that pretty quick.

Carter


On Apr 28, 2013, at 10:24 PM, "Dave Edelman" <dedelman at iname.com> wrote:

> Carter,
> 
> In each instance the files are from different places eg: a wide -area load
> balanced website would have two or more sources of packet capture. I think
> that the reason that I started to use the -X was occasional trouble with the
> -n parameter. By using -X I knew the starting point for the -n so I could be
> sure of what was being translated and what would appear untranslated. Is
> there a solution looking for a problem similar to allowing an optional value
> for -n to fix the setting regardless of the .rarc file?
> 
> --Dave
> 
>> -----Original Message-----
>> From: Carter Bullard [mailto:carter at qosient.com]
>> Sent: Monday, April 22, 2013 11:16 AM
>> To: Dave Edelman
>> Cc: 'Michal Purzynski'; argus-info at lists.andrew.cmu.edu
>> Subject: Re: [ARGUS] argus processing pcaps
>> 
>> Hey Dave,
>> You are assigning a different srcid for each file, and that may not be
>> appropriate.
>> If the packets are just timed packet capture files from the same wire,
> then
>> you
>> wouldn't want to change the srcid.  But if the files come from different
>> places,
>> then you are absolutely correct to assign different srcid's.
>> 
>> If you use the -X, then you will need to set the options you want on the
>> command
>> line to change from the default behavior, or you will want to provide an
>> argus.conf
>> file.  I sometimes design perl scripts that create the argus.conf file I
> want on
>> the
>> fly, and have my calls to argus use those temporary argus.conf files, so
> the
>> command lines aren't that long (convenience, ease of use and debugging).
>> 
>> Carter
>> 
>> On Apr 22, 2013, at 8:28 AM, "Dave Edelman" <dedelman at iname.com>
>> wrote:
>> 
>>> Carter,
>>> 
>>> I've been doing quite a bit of converting packet captures to Argus flow
>>> records and I wonder if I may be over thinking the solution. If I omit
> the
>>> for loop, I use something like this:
>>> ($a = the pcap file name, $b = an integer that starts at 1 and
> increments
>>> for each file pcap file processed)
>>> 
>>> /usr/local/sbin/argus -X -A -Z -R -J -C -m  -e $b -U 2048 -r $a -w
>>> $inputDir/$a.argus;
>>> 
>>> I always use -X to make sure that my argus.conf file doesn't have some
>>> setting that confuses things. Do I need the AZRJCm flags or are they
> just
>>> going to get in the way?
>>> 
>>> --Dave
>>> 
>>> 
>>>> -----Original Message-----
>>>> From: argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu
>>>> [mailto:argus-info-
>> bounces+dedelman=iname.com at lists.andrew.cmu.edu]
>>>> On Behalf Of Carter Bullard
>>>> Sent: Friday, April 19, 2013 9:08 AM
>>>> To: Michal Purzynski
>>>> Cc: argus-info at lists.andrew.cmu.edu
>>>> Subject: Re: [ARGUS] argus processing pcaps
>>>> 
>>>> Hey Michal,
>>>> The simplest way, to keep it sane, would be, assuming you use a linux
>>>> machine, are in the directory where the files are,  and bash as your
>>> shell:
>>>> 
>>>>  % for i in *; do echo $i; argus -r $i -w argus.$i; done
>>>> 
>>>> This will generate an argus file for each packet file that you have.
>>>> This generates argus primitive data, and then when you want to generate
>>>> a report, you can process the primitive file to create the metric
> you're
>>>> interested in.
>>>> 
>>>> Once your done, read a few of the files using ra(), from the
> argus-clients
>>>> package, to see if the data looks reasonable.
>>>> 
>>>> If the packet files are all from the " observation domain ", then
> merging
>>> the
>>>> data to create a single stream of argus files may make sense.  Argus
>>>> will try to use the filename to sort on time, so if your timestamp is
>>> sortable,
>>>> then you can do this:
>>>> 
>>>>  % argus -r snort.log* -w - argus.out
>>>> 
>>>> Getting the packets into argus in time order is important, so its
> possible
>>> that
>>>> the command above will generate warning messages, but hopefully it will
>>>> do what you want.  If this doesn't work, no problem, you can use other
>>>> programs to merge the multiple argus files from the first strategy.
>>>> 
>>>>  % rasort -m stime -r argus.snort.log* -w argus.out
>>>> 
>>>> If you want to then organize the data, you can use rasplit() to process
>>> the
>>>> argus output to generate a time oriented data repository.
>>>> 
>>>>  % rasplit -M time 5m -r argus.out -w
>>>> archive/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S
>>>> 
>>>> Now you have flow data that is in time series chunks that make time
> series
>>>> graphs useful, these can be indexed and processed in mysql, can be
>> graphed
>>>> quickly, and be searched etc somewhat efficiently.....
>>>> 
>>>> Hopefully that will get you started !!!
>>>> 
>>>> Carter
>>>> 
>>>> On Apr 19, 2013, at 7:29 AM, Michal Purzynski <michal at rsbac.org> wrote:
>>>> 
>>>>> On 4/19/13 1:25 PM, Michal Purzynski wrote:
>>>>>> Hi,
>>>>>> 
>>>>>> I've got a bunch of pcap files and would like to generate argus data
>>> from
>>>> them. It's important to keep things sane, so the flows are merged, etc.
>>>>>> 
>>>>>> How can i achieve this? I guess it's some combination of the argus
>>>> command in a for loop, feeding data to rasplit, but don't really know
>>> where
>>>> to start.
>>>>> Forgot something:
>>>>> 
>>>>> the pcap files are in a single dir, named snort.log.<unix timestamp>
>>> each
>>>> about 150MB in size.
>>>>> 
>>>>> And there's a lot of them, a few TB.
>>>>> 
>>> 
>>> 
> 
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4399 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130429/a5bbb921/attachment.bin>