Order when reading pcap files

[Carter Bullard]

Hey David,
Argus doesn't require packets to be in order to do its flow tracking,
but it is best for the flow cache flushing logic to see the packets in  
some
form of order.   Since you are running argus on each file (assuming
each file has packets in some order), it won't be a problem.

The way you are running argus, though, you will want to sort the
resulting flow data file, to get the flow records in order, so I would
add this after your "for" command:

    for file in *; do argus -r $file -w MyData.argus; done
    rasort -M replace -r MyData.argus

This works pretty well until MyData.argus gets bigger than memory,
then it will perform very poorly.   If you run into this problem,
you can use rasplit() to get the data into a series of time ordered  
files,
which then can be sorted independently.   Try this, if your interested:

    argus -r * -w -  |    rasplit -M time 5m \
         -w ./data/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S

This will result in the argus records being stored in a series of
argus files that are 5 minutes long, organized by year, month
and day.    There is no guarantee that within each file, the records
will be in order, but the data is now organized in a file system that
is "grossly" sorted.

All the ra* programs can recursively descend these types of archive
directories, and process the data in date order, based on the filename.
This allows you to sort the entire set, with a single command.

    rasort -R ./data -M replace

This will cause rasort() to sort each file in place, which hopefully  
will
be doable on your machine.    Then, to get all of the flow data into a
single sorted file, you would do:

    ra -R ./dir -w MyData.argus


No IRC channel that I am aware of.

Hopefully this is helpful, if it raises more questions, just send more  
email!!!!

Carter

On Aug 15, 2008, at 9:17 AM, David wrote:

> I read in a whole bunch of pcap files using argus, like so:
>
> $ for file in *; do argus -r $file -w MyData.argus; done
>
> However, these aren't guaranteed to be in date order.  Will that  
> screw up argus at all?  If so, I can get an ordered list and read in  
> properly, just wondering.
>
> Also, is there an IRC channel for argus?
>
> Regards,
>
> David
>
> ----------------------------------------------------------------
> This message was sent using IMP, the Internet Messaging Program.
>
>