Order when reading pcap files

[Nick Diel]

David,

I wanted to throw out another idea.  Now Carter's solution is quite elegant
and pure Argus; I wanted to throw out another possibility that might be
beneficial if you are running into memory problems (else I personally prefer
Carter's solution).

Use mergecap (part of the Whireshark family) to combine the files together
in correct order and feed it to argus.

mergecap -w - /pcapDir/* | argus -r - -w MyData.argus

Since you would be using this solution when you are running into memory
problems (the pcaps would be quite large compared to system memory),
mergecap will be slow.  Though I think the tradeoff a more simple approach
by you and let the system do the work.

For me, Argus and it's clients can be a little difficult when you are short
on memory.  I just feed Argus as much memory as I can get (I know have a box
with 32gb of RAM).  But I collect terabytes of pcaps.

Nick

PS  I am constantly feeding groups of pcaps to Argus, though I just make
sure they are named so lexicographical = chronological order.  You might not
have that luxury.

On Fri, Aug 15, 2008 at 8:28 AM, Carter Bullard <carter at qosient.com> wrote:

> Hey David,
> Argus doesn't require packets to be in order to do its flow tracking,
> but it is best for the flow cache flushing logic to see the packets in some
> form of order.   Since you are running argus on each file (assuming
> each file has packets in some order), it won't be a problem.
>
> The way you are running argus, though, you will want to sort the
> resulting flow data file, to get the flow records in order, so I would
> add this after your "for" command:
>
>   for file in *; do argus -r $file -w MyData.argus; done
>   rasort -M replace -r MyData.argus
>
> This works pretty well until MyData.argus gets bigger than memory,
> then it will perform very poorly.   If you run into this problem,
> you can use rasplit() to get the data into a series of time ordered files,
> which then can be sorted independently.   Try this, if your interested:
>
>   argus -r * -w -  |    rasplit -M time 5m \
>        -w ./data/%Y/%m/%d/argus.%Y.%m.%d.%H.%M.%S
>
> This will result in the argus records being stored in a series of
> argus files that are 5 minutes long, organized by year, month
> and day.    There is no guarantee that within each file, the records
> will be in order, but the data is now organized in a file system that
> is "grossly" sorted.
>
> All the ra* programs can recursively descend these types of archive
> directories, and process the data in date order, based on the filename.
> This allows you to sort the entire set, with a single command.
>
>   rasort -R ./data -M replace
>
> This will cause rasort() to sort each file in place, which hopefully will
> be doable on your machine.    Then, to get all of the flow data into a
> single sorted file, you would do:
>
>   ra -R ./dir -w MyData.argus
>
>
> No IRC channel that I am aware of.
>
> Hopefully this is helpful, if it raises more questions, just send more
> email!!!!
>
> Carter
>
>
> On Aug 15, 2008, at 9:17 AM, David wrote:
>
>  I read in a whole bunch of pcap files using argus, like so:
>>
>> $ for file in *; do argus -r $file -w MyData.argus; done
>>
>> However, these aren't guaranteed to be in date order.  Will that screw up
>> argus at all?  If so, I can get an ordered list and read in properly, just
>> wondering.
>>
>> Also, is there an IRC channel for argus?
>>
>> Regards,
>>
>> David
>>
>> ----------------------------------------------------------------
>> This message was sent using IMP, the Internet Messaging Program.
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20080815/fa2fd39a/attachment.html>