[ARGUS] the packet and byte count are unreasonably high
Ming Fu via Argus-info
argus-info at lists.andrew.cmu.edu
Mon Nov 11 10:23:00 EST 2024
Hi Carter,
The problem does not happen often, so unless we search for it on purpose across a large set of archives, we may not see it. We notice the problem mostly because we hit it during a query. I can't reproduce the problem in testing environment.
Is there a command to extract just the affected connection from the original archive file into a smaller archive? There are barriers other than just the size to share the full archive.
Regards
Ming
-----Original Message-----
From: Carter Bullard <carter at qosient.com>
Sent: Monday, November 11, 2024 10:12 AM
To: Ming Fu <Ming.Fu at esentire.com>
Cc: Argus <argus-info at lists.andrew.cmu.edu>
Subject: Re: [ARGUS] the packet and byte count are unreasonably high
Hey Ming,
We were working on this issue last year about this same time …. And in June/July (?) you thought we had fixed the problem …
No problem, just wanting to know if it went away and then came back ?? Or maybe we were just lucky ??
Can you share a recent binary file of a record that is tooooo big ??
Carter
> On Nov 3, 2024, at 10:52 PM, Ming Fu <Ming.Fu at esentire.com> wrote:
>
> Hi Carter,
>
> I missed one question in previous reply.
> checking for rpc/xdr.h... yes
> is in the configure log.
>
> Regards,
> Ming
>
> -----Original Message-----
> From: Ming Fu
> Sent: Sunday, November 3, 2024 10:35 PM
> To: Carter Bullard <carter at qosient.com>; Argus <argus-info at lists.andrew.cmu.edu>
> Subject: RE: [ARGUS] the packet and byte count are unreasonably high
>
> Hi Carter,
>
> Thanks for pointing out how to debug this. I tried to provide the information requested. Let me know if there is anything else I can do
>
>
> 1) rasort -r archive -m spkts
> Looks there are two TCP session during the period I checked that show very high count.
> ====
> rasort -r argus.vsniff1.2024-10-11-22* -m spkts | head -20
> StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport TotPkts TotBytes State
> 10/11.22:14:06.47* e * tcp 10.61.6.12.62275 -> 10.49.40.72.micro* 9151579* 8589934594 CON
> 10/11.22:18:52.36* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1008813* 725891895* CON
> 10/11.22:18:22.30* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 9817848* 230584302* CON
> 10/11.22:15:52.06* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 8124493* 470626162* CON
> 10/11.22:16:22.10* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1974118* 346017201* CON
> 10/11.22:16:52.13* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 8453397* 346018677* CON
> 10/11.22:03:16.59* e * tcp 10.100.250.137.60292 -> 10.63.36.11.2051 1732512 2345402044 FIN
> 10/11.22:05:41.40* e * tcp 10.100.250.137.60382 -> 10.63.36.11.2051 1709742 2331974985 CON
> 10/11.22:03:41.35* e * tcp 10.100.250.137.60328 -> 10.63.36.11.2051 1258336 1655916040 FIN
> 10/11.22:03:02.43* e * tcp 10.100.250.137.60248 -> 10.63.36.11.2051 1076466 1451990661 FIN
> 10/11.22:09:52.61* e * tcp 10.100.250.137.33044 -> 10.63.36.11.2051 1078723 1447225723 CON
> 10/11.22:20:33.59* e udp 10.123.1.46.6920 <-> 10.123.8.109.6905 896083 1255574452 CON
> 10/11.22:19:32.23* e udp 10.123.1.46.6922 <-> 10.123.8.158.6905 744181 1047695626 CON
> 10/11.22:09:52.61* e * tcp 10.100.250.137.33054 -> 10.63.36.11.2051 785679 1033276426 CON
> 10/11.22:21:58.69* e * tcp 10.100.250.137.33136 -> 10.63.36.11.2051 747006 966669152 CON
> 10/11.22:22:15.16* e * tcp 10.100.250.137.33146 -> 10.63.36.11.2051 700783 932664632 CON
> 10/11.22:00:00.17* e udp 10.123.1.46.6922 <-> 10.123.8.158.6905 671896 939987186 CON
> 10/11.22:20:44.96* e * tcp 10.100.250.137.33146 -> 10.63.36.11.2051 668494 887168440 CON
> 10/11.21:51:35.04* e udp 10.123.1.47.6913 <-> 10.123.12.40.6905 633041 889902024 CON
>
> ====
> rasort -r argus.vsniff1.2024-10-11-22* -m spkts - src host 10.61.6.12 and port 62275
>
> StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport TotPkts TotBytes State
> 10/11.22:14:06.47* e * tcp 10.61.6.12.62275 -> 10.49.40.72.micro* 9151579* 8589934594 CON
> 10/11.22:14:36.56* e * tcp 10.61.6.12.62275 -> 10.49.40.72.micro* 3166767 4022676693 CON
> 10/11.22:13:36.46* e * tcp 10.61.6.12.62275 -> 10.49.40.72.micro* 829991 1042946594 CON
> 10/11.22:15:06.95* e s tcp 10.61.6.12.62275 -> 10.49.40.72.micro* 9 1697 RST
>
> ===
> rasort -r argus.vsniff1.2024-10-11-22* -m spkts - src host 10.61.6.12 and port 62277
> StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport TotPkts TotBytes State
> 10/11.22:18:52.36* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1008813* 725891895* CON
> 10/11.22:18:22.30* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 9817848* 230584302* CON
> 10/11.22:15:52.06* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 8124493* 470626162* CON
> 10/11.22:16:22.10* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1974118* 346017201* CON
> 10/11.22:16:52.13* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 8453397* 346018677* CON
> 10/11.22:19:22.38* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 2134661 2875562333 CON
> 10/11.22:17:22.20* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1718734* 346017201* CON
> 10/11.22:15:22.03* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1254258 1691480734 CON
> 10/11.22:23:10.44* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 20 5028 CON
> 10/11.22:27:17.24* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 18 1880 CON
> 10/11.22:24:55.15* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 8 2158 CON
> 10/11.22:45:17.93* e d tcp 10.61.6.12.62277 <?> 10.49.40.72.micro* 8 2164 CON
> 10/11.22:21:10.38* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 6 378 CON
> 10/11.22:20:10.36* e s tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 5 318 CON
> 10/11.22:22:10.41* e s tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 4 252 CON
> 10/11.22:26:17.24* e s tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 5 318 CON
> 10/11.22:29:32.54* e d tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 3 192 CON
> 10/11.22:47:17.95* e d tcp 10.61.6.12.62277 <?> 10.49.40.72.micro* 4 252 CON
> 10/11.22:23:55.13* e s tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 5 324 CON
> 10/11.22:46:18.01* e s tcp 10.61.6.12.62277 <?> 10.49.40.72.micro* 2 126 CON
> 10/11.22:17:52.21* e * tcp 10.61.6.12.62277 > 10.49.40.72.micro* 9267422* 751421585* CON
> 10/11.22:22:40.43* e g tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 2 132 CON
> 10/11.22:25:47.22* e g tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1 66 CON
> 10/11.22:28:02.54* e g tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 3 198 CON
>
> 2) ra -r archive - tcp and src port 62277 and dst port 445
> ra -r argus.vsniff1.2024-10-11-22* - tcp and src port 62277 and dst port 445
> StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport TotPkts TotBytes State
> 10/11.22:15:22.03* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1254258 1691480734 CON
> 10/11.22:15:52.06* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 8124493* 470626162* CON
> 10/11.22:16:22.10* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1974118* 346017201* CON
> 10/11.22:16:52.13* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 8453397* 346018677* CON
> 10/11.22:17:22.20* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1718734* 346017201* CON
> 10/11.22:17:52.21* e * tcp 10.61.6.12.62277 > 10.49.40.72.micro* 9267422* 751421585* CON
> 10/11.22:18:22.30* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 9817848* 230584302* CON
> 10/11.22:18:52.36* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1008813* 725891895* CON
> 10/11.22:19:22.38* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 2134661 2875562333 CON
> 10/11.22:20:10.36* e s tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 5 318 CON
> 10/11.22:21:10.38* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 6 378 CON
> 10/11.22:22:10.41* e s tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 4 252 CON
> 10/11.22:22:40.43* e g tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 2 132 CON
> 10/11.22:23:10.44* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 20 5028 CON
> 10/11.22:23:55.13* e s tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 5 324 CON
> 10/11.22:24:55.15* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 8 2158 CON
> 10/11.22:25:47.22* e g tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1 66 CON
> 10/11.22:26:17.24* e s tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 5 318 CON
> 10/11.22:27:17.24* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 18 1880 CON
> 10/11.22:28:02.54* e g tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 3 198 CON
> 10/11.22:29:32.54* e d tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 3 192 CON
> 10/11.22:45:17.93* e d tcp 10.61.6.12.62277 <?> 10.49.40.72.micro* 8 2164 CON
> 10/11.22:46:18.01* e s tcp 10.61.6.12.62277 <?> 10.49.40.72.micro* 2 126 CON
> 10/11.22:47:17.95* e d tcp 10.61.6.12.62277 <?> 10.49.40.72.micro* 4 252 CON
>
> ===
> ra -r argus.vsniff1.2024-10-11-22* - tcp and src port 62275 and dst port 445
> StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport TotPkts TotBytes State
> 10/11.22:13:36.46* e * tcp 10.61.6.12.62275 -> 10.49.40.72.micro* 829991 1042946594 CON
> 10/11.22:14:06.47* e * tcp 10.61.6.12.62275 -> 10.49.40.72.micro* 9151579* 8589934594 CON
> 10/11.22:14:36.56* e * tcp 10.61.6.12.62275 -> 10.49.40.72.micro* 3166767 4022676693 CON
> 10/11.22:15:06.95* e s tcp 10.61.6.12.62275 -> 10.49.40.72.micro* 9 1697 RST
>
> 3) Add loss, retransmission etc
> /opt/pkgs/argus-clients-e/bin/ra -L -1 -c' ' -n -s dbytes,stime,our,stcpb,dtcpb,sloss,sretran,dloss,dretran,saddr,daddr,proto,spkts,dpkts,sport,dport,sbytes -r argus.vsniff1.2024-10-11-22* - src pkts gt 4000000000
> 0 10/11.22:14:06.476932 1438629544 2092470428 0 10.61.6.12 10.49.40.72 tcp 9151579764958429187 17255366656 62275 445 8589934594
> 4706261611810128643 10/11.22:15:52.069615 482070462 -2147483648 0 10.61.6.12 10.49.40.72 tcp 3458764519289913606 4665729215040269099 62277 445 13950255104
> 0 10/11.22:16:22.104071 4006887682 0 0 10.61.6.12 10.49.40.72 tcp 1974078638784983 39735787520 62277 445 3460172017553146399
> 14753212661760 10/11.22:16:52.131629 3069967430 288079140 0 10.61.6.12 10.49.40.72 tcp 845322578559266 17187209216 62277 445 3460172017553113607
> 2305843022098595842 10/11.22:18:22.302991 709145400 -2147483648 0 10.61.6.12 10.49.40.72 tcp 4629700418014806016 5188147880946339125 62277 445 1077936128
> 2305843022098595842 10/11.22:18:52.361328 4226499264 0 0 10.61.6.12 10.49.40.72 tcp 4899986764475894443 5188147880946341913 62277 445 4953075936113354752
>
> 4) The hardware is 64 bite Intel Xeon. Silver 4214R.
>
> Regards,
> Ming
> -----Original Message-----
> From: Carter Bullard <carter at qosient.com>
> Sent: Sunday, November 3, 2024 9:42 AM
> To: Argus <argus-info at lists.andrew.cmu.edu>; Ming Fu <Ming.Fu at esentire.com>
> Subject: Re: [ARGUS] the packet and byte count are unreasonably high
>
> Hey Ming,
> Sorry for the delayed response …
> A few questions, and if you would respond to the mailing list that would be great … Looks like you’re experiencing a bug, so if you can help us debug it that would also be great …
>
> Do you have a sense of the percent errant records ? Looks like you have 4 bad records, out how many ???
> You can use rasort.1 to sort the flow for you, like:
>
> rasort -r archive -m spkts
>
> Which may give you some control over seeing what the trends might be, if there are any ...
>
> If you notice, the errant records are all the same flow, x.62277 -> y.445 … are there other records for this flow that look correct ?? Is this a production flow, or is it from a test ??
>
> ra -r archive - tcp and src port 62277 and dst port 445
>
>
> These are all TCP connections … TCP can you print the “stcpb” and “dtcpb” and the state when you print out these records ???
> To help in understanding the extent of the error, can you also print out the “sloss”, “sretrans”, “dloss” and “dtretrans” ??? If these are reasonable values that will help diagnose.
>
> The pkt and byte counters are 64-bit ints … some 32-bit machines can be very ’strange’ with 64-bit values, is either argus or your archiver a 32-bit machine ???
> When you configure, does your machine support the XDR library ??? (“ checking for rpc/xdr.h… yes “)
>
> Rather than printing the stime and ltime, if you could print the stime and our, that is an important value … Does this happen every day ? Every hour ??
>
> These issues are pretty easy to find, I suspect a type mismatch processing something around the metrics DSR buffer, although we’ve been very careful of this for many years … but you never know …
>
> And if you could share the errant flows … something like this should work …
>
> ra -r archive -w big.flow.problem.out - src pkts gt 4000000000
>
> The filter currently handles only 32-bit values, I’ll fix that early next week ...
>
> Carter
>
>
>> On Nov 1, 2024, at 2:58 PM, Ming Fu via Argus-info <argus-info at lists.andrew.cmu.edu> wrote:
>>
>> Hi,
>>
>> Does anyone have similar problem? This happens to us from time to time. The read out from the archive has packets and bytes counter that are impossibly high.
>> We are running the argus from 5.0.0 branch. This also happens to the 3.x argus before we upgraded to v5. I was hopping the upgrade to v5 would solve this problem, but it still happens.
>>
>> ra -L -1 -c' ' -n -s
>> dbytes,stime,ltime,saddr,daddr,proto,spkts,dpkts,sport,dport,sbytes -r archive | sort -n ....
>> Skip the lower counts
>> ...
>>
>> 3999669780 10/11.22:14:36.569519 10/11.22:15:05.029679 10.61.6.12
>> 10.49.40.72 tcp 377597 2789170 62275 445 23006913
>> 4185639434 10/11.22:12:16.415003 10/11.22:12:44.906949 10.61.6.12
>> 10.49.40.72 tcp 182047 2918859 62268 445 11087544
>> 14753212661760 10/11.22:16:52.131629 10/11.22:17:22.201542 10.61.6.12
>> 10.49.40.72 tcp 845322578559266 17187209216 62277 445
>> 3460172017553113607
>> 2305843022098595842 10/11.22:18:22.302991 10/11.22:18:52.361259
>> 10.61.6.12 10.49.40.72 tcp 4629700418014806016 5188147880946339125
>> 62277 445 1077936128
>> 2305843022098595842 10/11.22:18:52.361328 10/11.22:19:22.380591
>> 10.61.6.12 10.49.40.72 tcp 4899986764475894443 5188147880946341913
>> 62277 445 4953075936113354752
>> 4706261611810128643 10/11.22:15:52.069615 10/11.22:16:22.104070
>> 10.61.6.12 10.49.40.72 tcp 3458764519289913606 4665729215040269099
>> 62277 445 13950255104
>> 7514215852137765761 10/11.22:17:52.215696 10/11.22:18:22.302887
>> 10.61.6.12 10.49.40.72 tcp 0 926742273327104 62277 445 1728685102
>>
>> Regards,
>> Ming
>>
>>
>> _______________________________________________
>> argus mailing list
>> argus at qosient.com
>> https://pairlist1.pair.net/mailman/listinfo/argus
>
More information about the argus
mailing list