[ARGUS] the packet and byte count are unreasonably high
Carter Bullard
carter at qosient.com
Mon Nov 11 10:42:39 EST 2024
Hey Ming,
Based on your earlier email … this should work to generate an argus file with about 24 records in it that would include errant flows as a well as reasonable flows for the same flow ??
% ra -w /tmp/argus.big.counter.flow.out -r argus.vsniff1.2024-10-11-22* - src host 10.61.6.12 and port 62275
If you can grab even tighter times, if you can get the specific flow between 2024-10-11.22:15:06 - 2024-10-11.22:21:15
That should catch normal -> errant -> normal for a single flow …
All ra* programs can write its output to an argus data file, so by using the filter, you can grab the flows you want and create a manageable file ...
Carter
> On Nov 11, 2024, at 10:23 AM, Ming Fu <Ming.Fu at esentire.com> wrote:
>
> Hi Carter,
>
> The problem does not happen often, so unless we search for it on purpose across a large set of archives, we may not see it. We notice the problem mostly because we hit it during a query. I can't reproduce the problem in testing environment.
>
> Is there a command to extract just the affected connection from the original archive file into a smaller archive? There are barriers other than just the size to share the full archive.
>
> Regards
> Ming
>
> -----Original Message-----
> From: Carter Bullard <carter at qosient.com>
> Sent: Monday, November 11, 2024 10:12 AM
> To: Ming Fu <Ming.Fu at esentire.com>
> Cc: Argus <argus-info at lists.andrew.cmu.edu>
> Subject: Re: [ARGUS] the packet and byte count are unreasonably high
>
> Hey Ming,
> We were working on this issue last year about this same time …. And in June/July (?) you thought we had fixed the problem …
> No problem, just wanting to know if it went away and then came back ?? Or maybe we were just lucky ??
>
> Can you share a recent binary file of a record that is tooooo big ??
>
> Carter
>
>
>> On Nov 3, 2024, at 10:52 PM, Ming Fu <Ming.Fu at esentire.com> wrote:
>>
>> Hi Carter,
>>
>> I missed one question in previous reply.
>> checking for rpc/xdr.h... yes
>> is in the configure log.
>>
>> Regards,
>> Ming
>>
>> -----Original Message-----
>> From: Ming Fu
>> Sent: Sunday, November 3, 2024 10:35 PM
>> To: Carter Bullard <carter at qosient.com>; Argus <argus-info at lists.andrew.cmu.edu>
>> Subject: RE: [ARGUS] the packet and byte count are unreasonably high
>>
>> Hi Carter,
>>
>> Thanks for pointing out how to debug this. I tried to provide the information requested. Let me know if there is anything else I can do
>>
>>
>> 1) rasort -r archive -m spkts
>> Looks there are two TCP session during the period I checked that show very high count.
>> ====
>> rasort -r argus.vsniff1.2024-10-11-22* -m spkts | head -20
>> StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport TotPkts TotBytes State
>> 10/11.22:14:06.47* e * tcp 10.61.6.12.62275 -> 10.49.40.72.micro* 9151579* 8589934594 CON
>> 10/11.22:18:52.36* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1008813* 725891895* CON
>> 10/11.22:18:22.30* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 9817848* 230584302* CON
>> 10/11.22:15:52.06* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 8124493* 470626162* CON
>> 10/11.22:16:22.10* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1974118* 346017201* CON
>> 10/11.22:16:52.13* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 8453397* 346018677* CON
>> 10/11.22:03:16.59* e * tcp 10.100.250.137.60292 -> 10.63.36.11.2051 1732512 2345402044 FIN
>> 10/11.22:05:41.40* e * tcp 10.100.250.137.60382 -> 10.63.36.11.2051 1709742 2331974985 CON
>> 10/11.22:03:41.35* e * tcp 10.100.250.137.60328 -> 10.63.36.11.2051 1258336 1655916040 FIN
>> 10/11.22:03:02.43* e * tcp 10.100.250.137.60248 -> 10.63.36.11.2051 1076466 1451990661 FIN
>> 10/11.22:09:52.61* e * tcp 10.100.250.137.33044 -> 10.63.36.11.2051 1078723 1447225723 CON
>> 10/11.22:20:33.59* e udp 10.123.1.46.6920 <-> 10.123.8.109.6905 896083 1255574452 CON
>> 10/11.22:19:32.23* e udp 10.123.1.46.6922 <-> 10.123.8.158.6905 744181 1047695626 CON
>> 10/11.22:09:52.61* e * tcp 10.100.250.137.33054 -> 10.63.36.11.2051 785679 1033276426 CON
>> 10/11.22:21:58.69* e * tcp 10.100.250.137.33136 -> 10.63.36.11.2051 747006 966669152 CON
>> 10/11.22:22:15.16* e * tcp 10.100.250.137.33146 -> 10.63.36.11.2051 700783 932664632 CON
>> 10/11.22:00:00.17* e udp 10.123.1.46.6922 <-> 10.123.8.158.6905 671896 939987186 CON
>> 10/11.22:20:44.96* e * tcp 10.100.250.137.33146 -> 10.63.36.11.2051 668494 887168440 CON
>> 10/11.21:51:35.04* e udp 10.123.1.47.6913 <-> 10.123.12.40.6905 633041 889902024 CON
>>
>> ====
>> rasort -r argus.vsniff1.2024-10-11-22* -m spkts - src host 10.61.6.12 and port 62275
>>
>> StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport TotPkts TotBytes State
>> 10/11.22:14:06.47* e * tcp 10.61.6.12.62275 -> 10.49.40.72.micro* 9151579* 8589934594 CON
>> 10/11.22:14:36.56* e * tcp 10.61.6.12.62275 -> 10.49.40.72.micro* 3166767 4022676693 CON
>> 10/11.22:13:36.46* e * tcp 10.61.6.12.62275 -> 10.49.40.72.micro* 829991 1042946594 CON
>> 10/11.22:15:06.95* e s tcp 10.61.6.12.62275 -> 10.49.40.72.micro* 9 1697 RST
>>
>> ===
>> rasort -r argus.vsniff1.2024-10-11-22* -m spkts - src host 10.61.6.12 and port 62277
>> StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport TotPkts TotBytes State
>> 10/11.22:18:52.36* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1008813* 725891895* CON
>> 10/11.22:18:22.30* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 9817848* 230584302* CON
>> 10/11.22:15:52.06* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 8124493* 470626162* CON
>> 10/11.22:16:22.10* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1974118* 346017201* CON
>> 10/11.22:16:52.13* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 8453397* 346018677* CON
>> 10/11.22:19:22.38* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 2134661 2875562333 CON
>> 10/11.22:17:22.20* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1718734* 346017201* CON
>> 10/11.22:15:22.03* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1254258 1691480734 CON
>> 10/11.22:23:10.44* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 20 5028 CON
>> 10/11.22:27:17.24* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 18 1880 CON
>> 10/11.22:24:55.15* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 8 2158 CON
>> 10/11.22:45:17.93* e d tcp 10.61.6.12.62277 <?> 10.49.40.72.micro* 8 2164 CON
>> 10/11.22:21:10.38* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 6 378 CON
>> 10/11.22:20:10.36* e s tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 5 318 CON
>> 10/11.22:22:10.41* e s tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 4 252 CON
>> 10/11.22:26:17.24* e s tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 5 318 CON
>> 10/11.22:29:32.54* e d tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 3 192 CON
>> 10/11.22:47:17.95* e d tcp 10.61.6.12.62277 <?> 10.49.40.72.micro* 4 252 CON
>> 10/11.22:23:55.13* e s tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 5 324 CON
>> 10/11.22:46:18.01* e s tcp 10.61.6.12.62277 <?> 10.49.40.72.micro* 2 126 CON
>> 10/11.22:17:52.21* e * tcp 10.61.6.12.62277 > 10.49.40.72.micro* 9267422* 751421585* CON
>> 10/11.22:22:40.43* e g tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 2 132 CON
>> 10/11.22:25:47.22* e g tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1 66 CON
>> 10/11.22:28:02.54* e g tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 3 198 CON
>>
>> 2) ra -r archive - tcp and src port 62277 and dst port 445
>> ra -r argus.vsniff1.2024-10-11-22* - tcp and src port 62277 and dst port 445
>> StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport TotPkts TotBytes State
>> 10/11.22:15:22.03* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1254258 1691480734 CON
>> 10/11.22:15:52.06* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 8124493* 470626162* CON
>> 10/11.22:16:22.10* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1974118* 346017201* CON
>> 10/11.22:16:52.13* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 8453397* 346018677* CON
>> 10/11.22:17:22.20* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1718734* 346017201* CON
>> 10/11.22:17:52.21* e * tcp 10.61.6.12.62277 > 10.49.40.72.micro* 9267422* 751421585* CON
>> 10/11.22:18:22.30* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 9817848* 230584302* CON
>> 10/11.22:18:52.36* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1008813* 725891895* CON
>> 10/11.22:19:22.38* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 2134661 2875562333 CON
>> 10/11.22:20:10.36* e s tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 5 318 CON
>> 10/11.22:21:10.38* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 6 378 CON
>> 10/11.22:22:10.41* e s tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 4 252 CON
>> 10/11.22:22:40.43* e g tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 2 132 CON
>> 10/11.22:23:10.44* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 20 5028 CON
>> 10/11.22:23:55.13* e s tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 5 324 CON
>> 10/11.22:24:55.15* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 8 2158 CON
>> 10/11.22:25:47.22* e g tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1 66 CON
>> 10/11.22:26:17.24* e s tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 5 318 CON
>> 10/11.22:27:17.24* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 18 1880 CON
>> 10/11.22:28:02.54* e g tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 3 198 CON
>> 10/11.22:29:32.54* e d tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 3 192 CON
>> 10/11.22:45:17.93* e d tcp 10.61.6.12.62277 <?> 10.49.40.72.micro* 8 2164 CON
>> 10/11.22:46:18.01* e s tcp 10.61.6.12.62277 <?> 10.49.40.72.micro* 2 126 CON
>> 10/11.22:47:17.95* e d tcp 10.61.6.12.62277 <?> 10.49.40.72.micro* 4 252 CON
>>
>> ===
>> ra -r argus.vsniff1.2024-10-11-22* - tcp and src port 62275 and dst port 445
>> StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport TotPkts TotBytes State
>> 10/11.22:13:36.46* e * tcp 10.61.6.12.62275 -> 10.49.40.72.micro* 829991 1042946594 CON
>> 10/11.22:14:06.47* e * tcp 10.61.6.12.62275 -> 10.49.40.72.micro* 9151579* 8589934594 CON
>> 10/11.22:14:36.56* e * tcp 10.61.6.12.62275 -> 10.49.40.72.micro* 3166767 4022676693 CON
>> 10/11.22:15:06.95* e s tcp 10.61.6.12.62275 -> 10.49.40.72.micro* 9 1697 RST
>>
>> 3) Add loss, retransmission etc
>> /opt/pkgs/argus-clients-e/bin/ra -L -1 -c' ' -n -s dbytes,stime,our,stcpb,dtcpb,sloss,sretran,dloss,dretran,saddr,daddr,proto,spkts,dpkts,sport,dport,sbytes -r argus.vsniff1.2024-10-11-22* - src pkts gt 4000000000
>> 0 10/11.22:14:06.476932 1438629544 2092470428 0 10.61.6.12 10.49.40.72 tcp 9151579764958429187 17255366656 62275 445 8589934594
>> 4706261611810128643 10/11.22:15:52.069615 482070462 -2147483648 0 10.61.6.12 10.49.40.72 tcp 3458764519289913606 4665729215040269099 62277 445 13950255104
>> 0 10/11.22:16:22.104071 4006887682 0 0 10.61.6.12 10.49.40.72 tcp 1974078638784983 39735787520 62277 445 3460172017553146399
>> 14753212661760 10/11.22:16:52.131629 3069967430 288079140 0 10.61.6.12 10.49.40.72 tcp 845322578559266 17187209216 62277 445 3460172017553113607
>> 2305843022098595842 10/11.22:18:22.302991 709145400 -2147483648 0 10.61.6.12 10.49.40.72 tcp 4629700418014806016 5188147880946339125 62277 445 1077936128
>> 2305843022098595842 10/11.22:18:52.361328 4226499264 0 0 10.61.6.12 10.49.40.72 tcp 4899986764475894443 5188147880946341913 62277 445 4953075936113354752
>>
>> 4) The hardware is 64 bite Intel Xeon. Silver 4214R.
>>
>> Regards,
>> Ming
>> -----Original Message-----
>> From: Carter Bullard <carter at qosient.com>
>> Sent: Sunday, November 3, 2024 9:42 AM
>> To: Argus <argus-info at lists.andrew.cmu.edu>; Ming Fu <Ming.Fu at esentire.com>
>> Subject: Re: [ARGUS] the packet and byte count are unreasonably high
>>
>> Hey Ming,
>> Sorry for the delayed response …
>> A few questions, and if you would respond to the mailing list that would be great … Looks like you’re experiencing a bug, so if you can help us debug it that would also be great …
>>
>> Do you have a sense of the percent errant records ? Looks like you have 4 bad records, out how many ???
>> You can use rasort.1 to sort the flow for you, like:
>>
>> rasort -r archive -m spkts
>>
>> Which may give you some control over seeing what the trends might be, if there are any ...
>>
>> If you notice, the errant records are all the same flow, x.62277 -> y.445 … are there other records for this flow that look correct ?? Is this a production flow, or is it from a test ??
>>
>> ra -r archive - tcp and src port 62277 and dst port 445
>>
>>
>> These are all TCP connections … TCP can you print the “stcpb” and “dtcpb” and the state when you print out these records ???
>> To help in understanding the extent of the error, can you also print out the “sloss”, “sretrans”, “dloss” and “dtretrans” ??? If these are reasonable values that will help diagnose.
>>
>> The pkt and byte counters are 64-bit ints … some 32-bit machines can be very ’strange’ with 64-bit values, is either argus or your archiver a 32-bit machine ???
>> When you configure, does your machine support the XDR library ??? (“ checking for rpc/xdr.h… yes “)
>>
>> Rather than printing the stime and ltime, if you could print the stime and our, that is an important value … Does this happen every day ? Every hour ??
>>
>> These issues are pretty easy to find, I suspect a type mismatch processing something around the metrics DSR buffer, although we’ve been very careful of this for many years … but you never know …
>>
>> And if you could share the errant flows … something like this should work …
>>
>> ra -r archive -w big.flow.problem.out - src pkts gt 4000000000
>>
>> The filter currently handles only 32-bit values, I’ll fix that early next week ...
>>
>> Carter
>>
>>
>>> On Nov 1, 2024, at 2:58 PM, Ming Fu via Argus-info <argus-info at lists.andrew.cmu.edu> wrote:
>>>
>>> Hi,
>>>
>>> Does anyone have similar problem? This happens to us from time to time. The read out from the archive has packets and bytes counter that are impossibly high.
>>> We are running the argus from 5.0.0 branch. This also happens to the 3.x argus before we upgraded to v5. I was hopping the upgrade to v5 would solve this problem, but it still happens.
>>>
>>> ra -L -1 -c' ' -n -s
>>> dbytes,stime,ltime,saddr,daddr,proto,spkts,dpkts,sport,dport,sbytes -r archive | sort -n ....
>>> Skip the lower counts
>>> ...
>>>
>>> 3999669780 10/11.22:14:36.569519 10/11.22:15:05.029679 10.61.6.12
>>> 10.49.40.72 tcp 377597 2789170 62275 445 23006913
>>> 4185639434 10/11.22:12:16.415003 10/11.22:12:44.906949 10.61.6.12
>>> 10.49.40.72 tcp 182047 2918859 62268 445 11087544
>>> 14753212661760 10/11.22:16:52.131629 10/11.22:17:22.201542 10.61.6.12
>>> 10.49.40.72 tcp 845322578559266 17187209216 62277 445
>>> 3460172017553113607
>>> 2305843022098595842 10/11.22:18:22.302991 10/11.22:18:52.361259
>>> 10.61.6.12 10.49.40.72 tcp 4629700418014806016 5188147880946339125
>>> 62277 445 1077936128
>>> 2305843022098595842 10/11.22:18:52.361328 10/11.22:19:22.380591
>>> 10.61.6.12 10.49.40.72 tcp 4899986764475894443 5188147880946341913
>>> 62277 445 4953075936113354752
>>> 4706261611810128643 10/11.22:15:52.069615 10/11.22:16:22.104070
>>> 10.61.6.12 10.49.40.72 tcp 3458764519289913606 4665729215040269099
>>> 62277 445 13950255104
>>> 7514215852137765761 10/11.22:17:52.215696 10/11.22:18:22.302887
>>> 10.61.6.12 10.49.40.72 tcp 0 926742273327104 62277 445 1728685102
>>>
>>> Regards,
>>> Ming
>>>
>>>
>>> _______________________________________________
>>> argus mailing list
>>> argus at qosient.com
>>> https://pairlist1.pair.net/mailman/listinfo/argus
>>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 1385 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20241111/d20dd442/attachment.bin>
More information about the argus
mailing list