[ARGUS] the packet and byte count are unreasonably high
Ming Fu via Argus-info
argus-info at lists.andrew.cmu.edu
Sun Nov 3 22:35:09 EST 2024
Hi Carter,
Thanks for pointing out how to debug this. I tried to provide the information requested. Let me know if there is anything else I can do
1) rasort -r archive -m spkts
Looks there are two TCP session during the period I checked that show very high count.
====
rasort -r argus.vsniff1.2024-10-11-22* -m spkts | head -20
StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport TotPkts TotBytes State
10/11.22:14:06.47* e * tcp 10.61.6.12.62275 -> 10.49.40.72.micro* 9151579* 8589934594 CON
10/11.22:18:52.36* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1008813* 725891895* CON
10/11.22:18:22.30* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 9817848* 230584302* CON
10/11.22:15:52.06* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 8124493* 470626162* CON
10/11.22:16:22.10* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1974118* 346017201* CON
10/11.22:16:52.13* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 8453397* 346018677* CON
10/11.22:03:16.59* e * tcp 10.100.250.137.60292 -> 10.63.36.11.2051 1732512 2345402044 FIN
10/11.22:05:41.40* e * tcp 10.100.250.137.60382 -> 10.63.36.11.2051 1709742 2331974985 CON
10/11.22:03:41.35* e * tcp 10.100.250.137.60328 -> 10.63.36.11.2051 1258336 1655916040 FIN
10/11.22:03:02.43* e * tcp 10.100.250.137.60248 -> 10.63.36.11.2051 1076466 1451990661 FIN
10/11.22:09:52.61* e * tcp 10.100.250.137.33044 -> 10.63.36.11.2051 1078723 1447225723 CON
10/11.22:20:33.59* e udp 10.123.1.46.6920 <-> 10.123.8.109.6905 896083 1255574452 CON
10/11.22:19:32.23* e udp 10.123.1.46.6922 <-> 10.123.8.158.6905 744181 1047695626 CON
10/11.22:09:52.61* e * tcp 10.100.250.137.33054 -> 10.63.36.11.2051 785679 1033276426 CON
10/11.22:21:58.69* e * tcp 10.100.250.137.33136 -> 10.63.36.11.2051 747006 966669152 CON
10/11.22:22:15.16* e * tcp 10.100.250.137.33146 -> 10.63.36.11.2051 700783 932664632 CON
10/11.22:00:00.17* e udp 10.123.1.46.6922 <-> 10.123.8.158.6905 671896 939987186 CON
10/11.22:20:44.96* e * tcp 10.100.250.137.33146 -> 10.63.36.11.2051 668494 887168440 CON
10/11.21:51:35.04* e udp 10.123.1.47.6913 <-> 10.123.12.40.6905 633041 889902024 CON
====
rasort -r argus.vsniff1.2024-10-11-22* -m spkts - src host 10.61.6.12 and port 62275
StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport TotPkts TotBytes State
10/11.22:14:06.47* e * tcp 10.61.6.12.62275 -> 10.49.40.72.micro* 9151579* 8589934594 CON
10/11.22:14:36.56* e * tcp 10.61.6.12.62275 -> 10.49.40.72.micro* 3166767 4022676693 CON
10/11.22:13:36.46* e * tcp 10.61.6.12.62275 -> 10.49.40.72.micro* 829991 1042946594 CON
10/11.22:15:06.95* e s tcp 10.61.6.12.62275 -> 10.49.40.72.micro* 9 1697 RST
===
rasort -r argus.vsniff1.2024-10-11-22* -m spkts - src host 10.61.6.12 and port 62277
StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport TotPkts TotBytes State
10/11.22:18:52.36* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1008813* 725891895* CON
10/11.22:18:22.30* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 9817848* 230584302* CON
10/11.22:15:52.06* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 8124493* 470626162* CON
10/11.22:16:22.10* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1974118* 346017201* CON
10/11.22:16:52.13* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 8453397* 346018677* CON
10/11.22:19:22.38* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 2134661 2875562333 CON
10/11.22:17:22.20* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1718734* 346017201* CON
10/11.22:15:22.03* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1254258 1691480734 CON
10/11.22:23:10.44* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 20 5028 CON
10/11.22:27:17.24* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 18 1880 CON
10/11.22:24:55.15* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 8 2158 CON
10/11.22:45:17.93* e d tcp 10.61.6.12.62277 <?> 10.49.40.72.micro* 8 2164 CON
10/11.22:21:10.38* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 6 378 CON
10/11.22:20:10.36* e s tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 5 318 CON
10/11.22:22:10.41* e s tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 4 252 CON
10/11.22:26:17.24* e s tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 5 318 CON
10/11.22:29:32.54* e d tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 3 192 CON
10/11.22:47:17.95* e d tcp 10.61.6.12.62277 <?> 10.49.40.72.micro* 4 252 CON
10/11.22:23:55.13* e s tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 5 324 CON
10/11.22:46:18.01* e s tcp 10.61.6.12.62277 <?> 10.49.40.72.micro* 2 126 CON
10/11.22:17:52.21* e * tcp 10.61.6.12.62277 > 10.49.40.72.micro* 9267422* 751421585* CON
10/11.22:22:40.43* e g tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 2 132 CON
10/11.22:25:47.22* e g tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1 66 CON
10/11.22:28:02.54* e g tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 3 198 CON
2) ra -r archive - tcp and src port 62277 and dst port 445
ra -r argus.vsniff1.2024-10-11-22* - tcp and src port 62277 and dst port 445
StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport TotPkts TotBytes State
10/11.22:15:22.03* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1254258 1691480734 CON
10/11.22:15:52.06* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 8124493* 470626162* CON
10/11.22:16:22.10* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1974118* 346017201* CON
10/11.22:16:52.13* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 8453397* 346018677* CON
10/11.22:17:22.20* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1718734* 346017201* CON
10/11.22:17:52.21* e * tcp 10.61.6.12.62277 > 10.49.40.72.micro* 9267422* 751421585* CON
10/11.22:18:22.30* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 9817848* 230584302* CON
10/11.22:18:52.36* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1008813* 725891895* CON
10/11.22:19:22.38* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 2134661 2875562333 CON
10/11.22:20:10.36* e s tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 5 318 CON
10/11.22:21:10.38* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 6 378 CON
10/11.22:22:10.41* e s tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 4 252 CON
10/11.22:22:40.43* e g tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 2 132 CON
10/11.22:23:10.44* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 20 5028 CON
10/11.22:23:55.13* e s tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 5 324 CON
10/11.22:24:55.15* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 8 2158 CON
10/11.22:25:47.22* e g tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 1 66 CON
10/11.22:26:17.24* e s tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 5 318 CON
10/11.22:27:17.24* e * tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 18 1880 CON
10/11.22:28:02.54* e g tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 3 198 CON
10/11.22:29:32.54* e d tcp 10.61.6.12.62277 -> 10.49.40.72.micro* 3 192 CON
10/11.22:45:17.93* e d tcp 10.61.6.12.62277 <?> 10.49.40.72.micro* 8 2164 CON
10/11.22:46:18.01* e s tcp 10.61.6.12.62277 <?> 10.49.40.72.micro* 2 126 CON
10/11.22:47:17.95* e d tcp 10.61.6.12.62277 <?> 10.49.40.72.micro* 4 252 CON
===
ra -r argus.vsniff1.2024-10-11-22* - tcp and src port 62275 and dst port 445
StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport TotPkts TotBytes State
10/11.22:13:36.46* e * tcp 10.61.6.12.62275 -> 10.49.40.72.micro* 829991 1042946594 CON
10/11.22:14:06.47* e * tcp 10.61.6.12.62275 -> 10.49.40.72.micro* 9151579* 8589934594 CON
10/11.22:14:36.56* e * tcp 10.61.6.12.62275 -> 10.49.40.72.micro* 3166767 4022676693 CON
10/11.22:15:06.95* e s tcp 10.61.6.12.62275 -> 10.49.40.72.micro* 9 1697 RST
3) Add loss, retransmission etc
/opt/pkgs/argus-clients-e/bin/ra -L -1 -c' ' -n -s dbytes,stime,our,stcpb,dtcpb,sloss,sretran,dloss,dretran,saddr,daddr,proto,spkts,dpkts,sport,dport,sbytes -r argus.vsniff1.2024-10-11-22* - src pkts gt 4000000000
0 10/11.22:14:06.476932 1438629544 2092470428 0 10.61.6.12 10.49.40.72 tcp 9151579764958429187 17255366656 62275 445 8589934594
4706261611810128643 10/11.22:15:52.069615 482070462 -2147483648 0 10.61.6.12 10.49.40.72 tcp 3458764519289913606 4665729215040269099 62277 445 13950255104
0 10/11.22:16:22.104071 4006887682 0 0 10.61.6.12 10.49.40.72 tcp 1974078638784983 39735787520 62277 445 3460172017553146399
14753212661760 10/11.22:16:52.131629 3069967430 288079140 0 10.61.6.12 10.49.40.72 tcp 845322578559266 17187209216 62277 445 3460172017553113607
2305843022098595842 10/11.22:18:22.302991 709145400 -2147483648 0 10.61.6.12 10.49.40.72 tcp 4629700418014806016 5188147880946339125 62277 445 1077936128
2305843022098595842 10/11.22:18:52.361328 4226499264 0 0 10.61.6.12 10.49.40.72 tcp 4899986764475894443 5188147880946341913 62277 445 4953075936113354752
4) The hardware is 64 bite Intel Xeon. Silver 4214R.
Regards,
Ming
-----Original Message-----
From: Carter Bullard <carter at qosient.com>
Sent: Sunday, November 3, 2024 9:42 AM
To: Argus <argus-info at lists.andrew.cmu.edu>; Ming Fu <Ming.Fu at esentire.com>
Subject: Re: [ARGUS] the packet and byte count are unreasonably high
Hey Ming,
Sorry for the delayed response …
A few questions, and if you would respond to the mailing list that would be great …
Looks like you’re experiencing a bug, so if you can help us debug it that would also be great …
Do you have a sense of the percent errant records ? Looks like you have 4 bad records, out how many ???
You can use rasort.1 to sort the flow for you, like:
rasort -r archive -m spkts
Which may give you some control over seeing what the trends might be, if there are any ...
If you notice, the errant records are all the same flow, x.62277 -> y.445 … are there other records for this flow that look correct ?? Is this a production flow, or is it from a test ??
ra -r archive - tcp and src port 62277 and dst port 445
These are all TCP connections … TCP can you print the “stcpb” and “dtcpb” and the state when you print out these records ???
To help in understanding the extent of the error, can you also print out the “sloss”, “sretrans”, “dloss” and “dtretrans” ??? If these are reasonable values that will help diagnose.
The pkt and byte counters are 64-bit ints … some 32-bit machines can be very ’strange’ with 64-bit values, is either argus or your archiver a 32-bit machine ???
When you configure, does your machine support the XDR library ??? (“ checking for rpc/xdr.h… yes “)
Rather than printing the stime and ltime, if you could print the stime and our, that is an important value …
Does this happen every day ? Every hour ??
These issues are pretty easy to find, I suspect a type mismatch processing something around the metrics DSR buffer, although we’ve been very careful of this for many years … but you never know …
And if you could share the errant flows … something like this should work …
ra -r archive -w big.flow.problem.out - src pkts gt 4000000000
The filter currently handles only 32-bit values, I’ll fix that early next week ...
Carter
> On Nov 1, 2024, at 2:58 PM, Ming Fu via Argus-info <argus-info at lists.andrew.cmu.edu> wrote:
>
> Hi,
>
> Does anyone have similar problem? This happens to us from time to time. The read out from the archive has packets and bytes counter that are impossibly high.
> We are running the argus from 5.0.0 branch. This also happens to the 3.x argus before we upgraded to v5. I was hopping the upgrade to v5 would solve this problem, but it still happens.
>
> ra -L -1 -c' ' -n -s dbytes,stime,ltime,saddr,daddr,proto,spkts,dpkts,sport,dport,sbytes -r archive | sort -n
> ....
> Skip the lower counts
> ...
>
> 3999669780 10/11.22:14:36.569519 10/11.22:15:05.029679 10.61.6.12 10.49.40.72 tcp 377597 2789170 62275 445 23006913
> 4185639434 10/11.22:12:16.415003 10/11.22:12:44.906949 10.61.6.12 10.49.40.72 tcp 182047 2918859 62268 445 11087544
> 14753212661760 10/11.22:16:52.131629 10/11.22:17:22.201542 10.61.6.12 10.49.40.72 tcp 845322578559266 17187209216 62277 445 3460172017553113607
> 2305843022098595842 10/11.22:18:22.302991 10/11.22:18:52.361259 10.61.6.12 10.49.40.72 tcp 4629700418014806016 5188147880946339125 62277 445 1077936128
> 2305843022098595842 10/11.22:18:52.361328 10/11.22:19:22.380591 10.61.6.12 10.49.40.72 tcp 4899986764475894443 5188147880946341913 62277 445 4953075936113354752
> 4706261611810128643 10/11.22:15:52.069615 10/11.22:16:22.104070 10.61.6.12 10.49.40.72 tcp 3458764519289913606 4665729215040269099 62277 445 13950255104
> 7514215852137765761 10/11.22:17:52.215696 10/11.22:18:22.302887 10.61.6.12 10.49.40.72 tcp 0 926742273327104 62277 445 1728685102
>
> Regards,
> Ming
>
>
> _______________________________________________
> argus mailing list
> argus at qosient.com
> https://pairlist1.pair.net/mailman/listinfo/argus
More information about the argus
mailing list