racount status and its definition
Carter Bullard
carter at qosient.com
Wed Jun 21 09:59:30 EDT 2006
At least argus-3.0 racluster and racount agree, as that is the behavior
that i've seen. Any malformed record is suspect, of course.
Hmmm, can you share the file, so I can attempt to correct the problem?
If its customer data, I can understand any sensitivities.
I have a program that I was going to release, after all the dust
settles,
that fixes/recovers corrupt argus files, but it is argus-3.0 format only
and is really experimental and so it won't work here, but I can attempt
to do the same thing with argus-2.0 data if there is demand.
Carter
On Jun 21, 2006, at 9:38 AM, Robin Gruyters wrote:
> Hi Carter,
>
> Here is the output with argus-3 commands (racount() and racluster
> ()). I have just pointed to the file which gives errors with
> argus-2.0.6 (racount()).
>
> [...]
> $ racount -r /data2/argus/05/21/*; \
> racluster -R /data2/argus/05/21 -m srcid -s trans pkts
> racount records total_pkts src_pkts dst_pkts
> total_bytes src_bytes dst_bytes
> sum 13951 771479 284510 486969
> 303901576 42836384 261065192
> 13927 771479
> [...]
>
> Here is the argus-2.0.6 output:
>
> [...]
> $ racount -r /data2/argus/archive/2006/05/21/*
> ArgusWarning: racount[48517]: ArgusReadSocketStream: malformed
> argus record len 17793
>
> racount records total_pkts src_pkts
> dst_pkts total_bytes src_bytes dst_bytes
> sum 149250 1143266 566499
> 576767 355491645 72812173 282679472
> [...]
>
> Regards,
>
> Robin
>
>
> Quoting Carter Bullard <carter at qosient.com>:
>
>> Hey Robin,
>> You can try this to see any primary discrepancies. If you find
>> a file that does generate count problems, we can zoom in on
>> a few things quickly and get at the root of the problem. I modified
>> the test script to give you direct comparisons, and its now doing it
>> a directory at a time:
>>
>> %for i in /data2/argus/05/*; do echo $i; racount -r $i/*;
>> racluster -R
>> $i -m srcid -s trans pkts
>>
>> The record values in racluster() are not going to be the same
>> as racount(), because racount() includes the management records
>> in the counting and racluster does not (since they are not merged).
>>
>> Carter
>>
>> On Jun 21, 2006, at 8:32 AM, Carter Bullard wrote:
>>
>>> Hmmm, are you a bash shell user or a csh?
>>> Do me a favor and try something like this (assuming bash)
>>>
>>> % for i in /data2/argus/05/*/* ; do echo $i; racluster -r $i -
>>> m proto -s proto trans; done
>>>
>>> and see if something doesn't look strange.
>>>
>>> Carter
>>>
>>> On Jun 21, 2006, at 4:44 AM, Robin Gruyters wrote:
>>>
>>>> Hello Carter,
>>>>
>>>> I have just ran racluster() with "-r" option and I get the same
>>>> output.
>>>> (without any errors)
>>>>
>>>> [racluster]
>>>> # racluster -r /data2/argus/05/*/* -m proto \
>>>> -s proto trans pkts spkts dpkts bytes sbytes dbytes - net
>>>> 82.148.219.32/28
>>>> esp 1 1 1 0 310
>>>> 310 0
>>>> gre 40440 16142557 6955364 9187193 4000449819
>>>> 1192848094 2807601725
>>>> udp 283037 554632 296948 257684 86633957
>>>> 35435077 51198880
>>>> tcp 144666 69282162 28369630 40912532 3994126059
>>>> 2934170533 1059955526
>>>> icmp 36644 50347 50270 77 4126254
>>>> 4121768 4486
>>>> [end racluster]
>>>>
>>>> Regards,
>>>>
>>>> Robin
>>>>
>>>> On Wed, Jun 21, 2006 at 03:32:49AM -0400, Carter Bullard wrote:
>>>>> Hey Robin,
>>>>> Looks like somethings up. Try calling racluster with the -r /
>>>>> data2/argus/05/*/*.
>>>>> I suspect that its getting some kind of error on one of the files,
>>>>> and is stopping,
>>>>> especially since your getting 1/10th the number of tcp
>>>>> transactions
>>>>> (144666 vs 1443778).
>>>>> I remember that you had a bug report with toooo many filenames, if
>>>>> you're
>>>>> still getting that problem, we'll have to try to figure out
>>>>> what is
>>>>> causing racluster()
>>>>> issues.
>>>>>
>>>>> Sorry for the problems,
>>>>>
>>>>> Carter
>>>>>
>>>>>
>>>>> On Jun 21, 2006, at 3:21 AM, Robin Gruyters wrote:
>>>>>
>>>>>> Hi Carter,
>>>>>>
>>>>>> At the moment we use the output of racount() for our monthly
>>>>>> report to
>>>>>> customers. To show them how much data they have used. (by
>>>>>> proto and
>>>>>> total)
>>>>>>
>>>>>> For me it doesn't matter if this is possible with racount() or
>>>>>> racluster(),
>>>>>> if I just get the output done.
>>>>>> If this is possible with racount() and with the "-M addr" option,
>>>>>> great!
>>>>>>
>>>>>> The other options, like counts on ports etc, that would also be
>>>>>> nice to
>>>>>> have.
>>>>>>
>>>>>> Altough you say it is also possible with racluster(), but the
>>>>>> outcome is
>>>>>> totally different what I get back from racount(). (I mean the
>>>>>> numbers)
>>>>>>
>>>>>> [racount]
>>>>>> # racount -ar /data2/argus/05/*/* - net 82.148.219.XXX/28
>>>>>> racount records total_pkts src_pkts
>>>>>> dst_pkts total_bytes src_bytes dst_bytes
>>>>>> tcp 1443778 69225031 28344760
>>>>>> 40880271 55494468479 7222126408 48272342071
>>>>>> udp 280703 549026 293754
>>>>>> 255272 86044190 35139486 50904704
>>>>>> icmp 35102 47042 46966
>>>>>> 76 3503635 3499223 4412
>>>>>> ip 40441 16142558 6955365
>>>>>> 9187193 4000450129 1192848404 2807601725
>>>>>> sum 1800024 85963657 35640845
>>>>>> 50322812 59584466433 8453613521 51130852912
>>>>>> [end racount]
>>>>>>
>>>>>> [racluster]
>>>>>> # racluster -R /data2/argus/05 -m proto \
>>>>>> -s proto trans pkts spkts dpkts bytes sbytes dbytes - net
>>>>>> 82.148.219.XXX/28
>>>>>> esp 1 1 1 0 310
>>>>>> 310 0
>>>>>> gre 40440 16142557 6955364 9187193 4000449819 1192848094
>>>>>> 2807601725
>>>>>> udp 283037 554632 296948 257684 86633957
>>>>>> 35435077 51198880
>>>>>> tcp 144666 69282162 28369630 40912532 3994126059 2934170533
>>>>>> 1059955526
>>>>>> icmp 36644 50347 50270 77 4126254
>>>>>> 4121768 4486
>>>>>> [end racluster]
>>>>>>
>>>>>> If you only check the "total bytes" on TCP packets. With
>>>>>> racount()
>>>>>> I get
>>>>>> 55494468479 bytes and with racluster() 3994126059 bytes. That
>>>>>> is a
>>>>>> huge
>>>>>> difference.
>>>>>>
>>>>>> Is there an explanation for this behaviour?
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Robin
>>>>>>
>>>>>> On Tue, Jun 20, 2006 at 11:04:01AM -0400, Carter Bullard wrote:
>>>>>>> Hey Robin et al.,
>>>>>>> You have become the target of anything racount() related ;o)
>>>>>>>
>>>>>>> So, in trying to understand if the "-M proto" option is useful,
>>>>>>> I realized that all of the old racount() functions are supported
>>>>>>> by racluster(), so I don't want to duplicate features, so I may
>>>>>>> end up redefining racount(), but keeping its default behavior.
>>>>>>> What I will do for now is leave it as it is, no -A support, but
>>>>>>> with the "-M addr" option and then figure out what to do after
>>>>>>> that based on the lists opinion.
>>>>>>>
>>>>>>> I use racount as a quick and dirty way of seeing how big is
>>>>>>> an argus data file, and to check if programs like racluster()
>>>>>>> preserve the counts when it aggregates records, so the
>>>>>>> default mode is great, but we can also generate the exact same
>>>>>>> output using racluster(), you just have to type more on the
>>>>>>> command line to get the output right. Same goes for the old -a
>>>>>>> option:
>>>>>>>
>>>>>>> The older racount() functions can be done in racluster() as:
>>>>>>>
>>>>>>> racount -r file
>>>>>>> racluster -r file -m srcid -s trans pkts spkts dpkts bytes
>>>>>>> sbytes
>>>>>>> dbytes
>>>>>>>
>>>>>>> racount -ar file
>>>>>>> racluster -r file -m proto -s proto trans pkts spkts dpkts
>>>>>>> bytes
>>>>>>> sbytes dbytes
>>>>>>>
>>>>>>> Now, with the '-M addr', we have a unique counting situation,
>>>>>>> and so that seems appropriate, and I think there should be more
>>>>>>> counting things to do, like ports, mac address types (vendor
>>>>>>> ids),
>>>>>>> that kind of thing.
>>>>>>>
>>>>>>> So, opinions? If we could discuss the counting
>>>>>>> requirements, that
>>>>>>> might help define racount a bit.
>>>>>>>
>>>>>>> Carter
>>>>>>>
>>>>>>
More information about the argus
mailing list