racount status and its definition
Carter Bullard
carter at qosient.com
Wed Jun 21 10:35:00 EDT 2006
Ok, so let me go over the error reporting for the ra* programs.
I'll try to get that done today. If anyone runs into a file that we
can't
parse, for whatever reason, and you can share it, it would be great
to have it for debugging purposes!!!!
Carter
On Jun 21, 2006, at 10:10 AM, Robin Gruyters wrote:
> Hi Carter,
>
> If it was my personal data, I didn't had any problems to give it to
> you, but this customer data, so I rather not. (sorry)
>
> If this problem only occurs when using argus-2 data with argus-3
> commands, than I can live with it. (but it would be nice to get it
> fixed)
> But it would be nice to get a message that there is a problem
> occured with the current argus file.
>
> Regards,
>
> Robin
>
> Quoting Carter Bullard <carter at qosient.com>:
>
>> At least argus-3.0 racluster and racount agree, as that is the
>> behavior
>> that i've seen. Any malformed record is suspect, of course.
>>
>> Hmmm, can you share the file, so I can attempt to correct the
>> problem?
>> If its customer data, I can understand any sensitivities.
>>
>> I have a program that I was going to release, after all the dust
>> settles,
>> that fixes/recovers corrupt argus files, but it is argus-3.0
>> format only
>> and is really experimental and so it won't work here, but I can
>> attempt
>> to do the same thing with argus-2.0 data if there is demand.
>>
>> Carter
>>
>>
>> On Jun 21, 2006, at 9:38 AM, Robin Gruyters wrote:
>>
>>> Hi Carter,
>>>
>>> Here is the output with argus-3 commands (racount() and racluster
>>> ()). I have just pointed to the file which gives errors with
>>> argus-2.0.6 (racount()).
>>>
>>> [...]
>>> $ racount -r /data2/argus/05/21/*; \
>>> racluster -R /data2/argus/05/21 -m srcid -s trans pkts
>>> racount records total_pkts src_pkts
>>> dst_pkts total_bytes src_bytes dst_bytes
>>> sum 13951 771479 284510
>>> 486969 303901576 42836384 261065192
>>> 13927 771479
>>> [...]
>>>
>>> Here is the argus-2.0.6 output:
>>>
>>> [...]
>>> $ racount -r /data2/argus/archive/2006/05/21/*
>>> ArgusWarning: racount[48517]: ArgusReadSocketStream: malformed
>>> argus record len 17793
>>>
>>> racount records total_pkts src_pkts
>>> dst_pkts total_bytes src_bytes dst_bytes
>>> sum 149250 1143266 566499
>>> 576767 355491645 72812173 282679472
>>> [...]
>>>
>>> Regards,
>>>
>>> Robin
>>>
>>>
>>> Quoting Carter Bullard <carter at qosient.com>:
>>>
>>>> Hey Robin,
>>>> You can try this to see any primary discrepancies. If you find
>>>> a file that does generate count problems, we can zoom in on
>>>> a few things quickly and get at the root of the problem. I
>>>> modified
>>>> the test script to give you direct comparisons, and its now
>>>> doing it
>>>> a directory at a time:
>>>>
>>>> %for i in /data2/argus/05/*; do echo $i; racount -r $i/*;
>>>> racluster -R
>>>> $i -m srcid -s trans pkts
>>>>
>>>> The record values in racluster() are not going to be the same
>>>> as racount(), because racount() includes the management records
>>>> in the counting and racluster does not (since they are not merged).
>>>>
>>>> Carter
>>>>
>>>> On Jun 21, 2006, at 8:32 AM, Carter Bullard wrote:
>>>>
>>>>> Hmmm, are you a bash shell user or a csh?
>>>>> Do me a favor and try something like this (assuming bash)
>>>>>
>>>>> % for i in /data2/argus/05/*/* ; do echo $i; racluster -r $i -
>>>>> m proto -s proto trans; done
>>>>>
>>>>> and see if something doesn't look strange.
>>>>>
>>>>> Carter
>>>>>
>>>>> On Jun 21, 2006, at 4:44 AM, Robin Gruyters wrote:
>>>>>
>>>>>> Hello Carter,
>>>>>>
>>>>>> I have just ran racluster() with "-r" option and I get the
>>>>>> same output.
>>>>>> (without any errors)
>>>>>>
>>>>>> [racluster]
>>>>>> # racluster -r /data2/argus/05/*/* -m proto \
>>>>>> -s proto trans pkts spkts dpkts bytes sbytes dbytes - net
>>>>>> 82.148.219.32/28
>>>>>> esp 1 1 1 0 310
>>>>>> 310 0
>>>>>> gre 40440 16142557 6955364 9187193 4000449819
>>>>>> 1192848094 2807601725
>>>>>> udp 283037 554632 296948 257684 86633957
>>>>>> 35435077 51198880
>>>>>> tcp 144666 69282162 28369630 40912532 3994126059
>>>>>> 2934170533 1059955526
>>>>>> icmp 36644 50347 50270 77 4126254
>>>>>> 4121768 4486
>>>>>> [end racluster]
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Robin
>>>>>>
>>>>>> On Wed, Jun 21, 2006 at 03:32:49AM -0400, Carter Bullard wrote:
>>>>>>> Hey Robin,
>>>>>>> Looks like somethings up. Try calling racluster with the -r /
>>>>>>> data2/argus/05/*/*.
>>>>>>> I suspect that its getting some kind of error on one of the
>>>>>>> files,
>>>>>>> and is stopping,
>>>>>>> especially since your getting 1/10th the number of tcp
>>>>>>> transactions
>>>>>>> (144666 vs 1443778).
>>>>>>> I remember that you had a bug report with toooo many
>>>>>>> filenames, if
>>>>>>> you're
>>>>>>> still getting that problem, we'll have to try to figure out
>>>>>>> what is
>>>>>>> causing racluster()
>>>>>>> issues.
>>>>>>>
>>>>>>> Sorry for the problems,
>>>>>>>
>>>>>>> Carter
>>>>>>>
>>>>>>>
>>>>>>> On Jun 21, 2006, at 3:21 AM, Robin Gruyters wrote:
>>>>>>>
>>>>>>>> Hi Carter,
>>>>>>>>
>>>>>>>> At the moment we use the output of racount() for our monthly
>>>>>>>> report to
>>>>>>>> customers. To show them how much data they have used. (by
>>>>>>>> proto and
>>>>>>>> total)
>>>>>>>>
>>>>>>>> For me it doesn't matter if this is possible with racount() or
>>>>>>>> racluster(),
>>>>>>>> if I just get the output done.
>>>>>>>> If this is possible with racount() and with the "-M addr"
>>>>>>>> option,
>>>>>>>> great!
>>>>>>>>
>>>>>>>> The other options, like counts on ports etc, that would also be
>>>>>>>> nice to
>>>>>>>> have.
>>>>>>>>
>>>>>>>> Altough you say it is also possible with racluster(), but the
>>>>>>>> outcome is
>>>>>>>> totally different what I get back from racount(). (I mean
>>>>>>>> the numbers)
>>>>>>>>
>>>>>>>> [racount]
>>>>>>>> # racount -ar /data2/argus/05/*/* - net 82.148.219.XXX/28
>>>>>>>> racount records total_pkts src_pkts
>>>>>>>> dst_pkts total_bytes src_bytes dst_bytes
>>>>>>>> tcp 1443778 69225031 28344760
>>>>>>>> 40880271 55494468479 7222126408 48272342071
>>>>>>>> udp 280703 549026 293754
>>>>>>>> 255272 86044190 35139486 50904704
>>>>>>>> icmp 35102 47042 46966
>>>>>>>> 76 3503635 3499223 4412
>>>>>>>> ip 40441 16142558 6955365
>>>>>>>> 9187193 4000450129 1192848404 2807601725
>>>>>>>> sum 1800024 85963657 35640845
>>>>>>>> 50322812 59584466433 8453613521 51130852912
>>>>>>>> [end racount]
>>>>>>>>
>>>>>>>> [racluster]
>>>>>>>> # racluster -R /data2/argus/05 -m proto \
>>>>>>>> -s proto trans pkts spkts dpkts bytes sbytes dbytes - net
>>>>>>>> 82.148.219.XXX/28
>>>>>>>> esp 1 1 1 0 310
>>>>>>>> 310 0
>>>>>>>> gre 40440 16142557 6955364 9187193 4000449819 1192848094
>>>>>>>> 2807601725
>>>>>>>> udp 283037 554632 296948 257684 86633957
>>>>>>>> 35435077 51198880
>>>>>>>> tcp 144666 69282162 28369630 40912532 3994126059 2934170533
>>>>>>>> 1059955526
>>>>>>>> icmp 36644 50347 50270 77 4126254
>>>>>>>> 4121768 4486
>>>>>>>> [end racluster]
>>>>>>>>
>>>>>>>> If you only check the "total bytes" on TCP packets. With
>>>>>>>> racount()
>>>>>>>> I get
>>>>>>>> 55494468479 bytes and with racluster() 3994126059 bytes.
>>>>>>>> That is a
>>>>>>>> huge
>>>>>>>> difference.
>>>>>>>>
>>>>>>>> Is there an explanation for this behaviour?
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Robin
>>>>>>>>
>>>>>>>> On Tue, Jun 20, 2006 at 11:04:01AM -0400, Carter Bullard wrote:
>>>>>>>>> Hey Robin et al.,
>>>>>>>>> You have become the target of anything racount() related ;o)
>>>>>>>>>
>>>>>>>>> So, in trying to understand if the "-M proto" option is
>>>>>>>>> useful,
>>>>>>>>> I realized that all of the old racount() functions are
>>>>>>>>> supported
>>>>>>>>> by racluster(), so I don't want to duplicate features, so I
>>>>>>>>> may
>>>>>>>>> end up redefining racount(), but keeping its default behavior.
>>>>>>>>> What I will do for now is leave it as it is, no -A support,
>>>>>>>>> but
>>>>>>>>> with the "-M addr" option and then figure out what to do after
>>>>>>>>> that based on the lists opinion.
>>>>>>>>>
>>>>>>>>> I use racount as a quick and dirty way of seeing how big is
>>>>>>>>> an argus data file, and to check if programs like racluster()
>>>>>>>>> preserve the counts when it aggregates records, so the
>>>>>>>>> default mode is great, but we can also generate the exact same
>>>>>>>>> output using racluster(), you just have to type more on the
>>>>>>>>> command line to get the output right. Same goes for the
>>>>>>>>> old -a
>>>>>>>>> option:
>>>>>>>>>
>>>>>>>>> The older racount() functions can be done in racluster() as:
>>>>>>>>>
>>>>>>>>> racount -r file
>>>>>>>>> racluster -r file -m srcid -s trans pkts spkts dpkts bytes
>>>>>>>>> sbytes
>>>>>>>>> dbytes
>>>>>>>>>
>>>>>>>>> racount -ar file
>>>>>>>>> racluster -r file -m proto -s proto trans pkts spkts dpkts
>>>>>>>>> bytes
>>>>>>>>> sbytes dbytes
>>>>>>>>>
>>>>>>>>> Now, with the '-M addr', we have a unique counting situation,
>>>>>>>>> and so that seems appropriate, and I think there should be
>>>>>>>>> more
>>>>>>>>> counting things to do, like ports, mac address types
>>>>>>>>> (vendor ids),
>>>>>>>>> that kind of thing.
>>>>>>>>>
>>>>>>>>> So, opinions? If we could discuss the counting
>>>>>>>>> requirements, that
>>>>>>>>> might help define racount a bit.
>>>>>>>>>
>>>>>>>>> Carter
>>>>>>>>>
>>>>>>>>
>
>
More information about the argus
mailing list