Argus memory issues
Russell Fulton
r.fulton at auckland.ac.nz
Sun Aug 19 23:50:07 EDT 2007
my sensors are seeing around two to three million flows per hour -- not
that high. From observing the memory footprint of my argus instances
after startup I see what appears to be a classic exponential approach to
a more or less stable limit. Which fits with it being flow related not
a leak which we would expect to be linear and unbounded.
Looks like what we need to examine is the flow timeout or garbage
collection.
One thing that I find interesting is that the memory is firmly locked in
physical memory suggesting that it must be being used all the time.
Snort OTOH only manages to keep half its memory from being swapped.
Russell
Peter Van Epp wrote:
> On Sun, Aug 19, 2007 at 11:00:07PM -0400, Carter Bullard wrote:
>
>> So it looks like we could be just processing a massive amount of
>> flows. I do have a fixed memory model for argus, so that we don't
>> grow over a certain memory foot print, we'll aggressively reclaim
>> records and hold low packet flows for only the status interval,
>> (usually 5 secs), that may work very well for you.
>>
>> So what are we thinking then, are we not leaking memory and you've got
>> a lot of flows? Should I switch to thinking about how to keep your
>> argus better behaved?
>>
>> Carter
>>
>>
>
> Yes sort of. The puzzling part is that my 2.0.6 system is still chugging
> along fine on the same data (but without user data) that is currently choking
> 3.0 (at this point without threads, user data or even output to a client).
>
> argus-3.0 (chosen from an hour that didn't seem to run out of memory :-)):
>
> test4:/var/log/argus vanepp$ ra3 -r /archive/argus3/com_argus.archive/2007/08/15/com_argus.2007.08.15.14.00.00.0.gz -n man
> 07-08-15 14:00:24 man 1172637 0 2786116 2 980417 27043 2786116 2980874864 CON
> 07-08-15 14:01:24 man 1195209 0 2840347 2 966201 27590 2840347 3040374744 CON
> 07-08-15 14:02:24 man 1217026 0 2891733 2 1000500 27616 2891733 3098033784 CON
> 07-08-15 14:03:24 man 1239216 0 2944188 2 1000624 27052 2944188 3155959864 CON
> 07-08-15 14:04:24 man 1259885 0 2993361 2 1045410 26387 2993361 3210878984 CON
> 07-08-15 14:05:24 man 1286106 0 3054195 2 1035640 27954 3054195 3275422984 CON
> 07-08-15 14:06:24 man 1306951 0 3102666 2 941859 28763 3102666 3331541144 CON
> 07-08-15 14:07:24 man 1327541 0 3150287 2 976290 29432 3150287 3387613104 CON
> 07-08-15 14:08:24 man 1354025 0 3205331 2 969769 30268 3205331 3450989784 CON
> 07-08-15 14:09:24 man 1374984 0 3253937 2 970571 28929 3253937 3507411144 CON
> 07-08-15 14:10:24 man 1393511 0 3297526 2 922896 27879 3297526 3559330344 CON
> 07-08-15 14:11:24 man 1412644 0 3341769 2 911490 29793 3341769 3612703624 CON
> 07-08-15 14:12:24 man 1432303 0 3388174 2 950633 28603 3388174 3667129144 CON
> 07-08-15 14:13:24 man 1452892 0 3436351 2 952619 29755 3436351 3723621824 CON
> 07-08-15 14:14:24 man 1472367 0 3481550 2 968650 29561 3481550 3777819144 CON
> 07-08-15 14:15:24 man 1492529 0 3529187 2 1011118 28944 3529187 3833547544 CON
> 07-08-15 14:16:24 man 1513232 0 3577628 2 992951 27440 3577628 3889140944 CON
> 07-08-15 14:17:24 man 1534016 0 3626696 2 1014643 27236 3626696 3945081104 CON
> 07-08-15 14:18:24 man 1555291 0 3676958 2 1037774 26918 3676958 4001647144 CON
> 07-08-15 14:19:24 man 1576296 0 3726748 2 1032067 26140 3726748 4057477624 CON
> 07-08-15 14:20:24 man 1595717 0 3773567 2 982867 26605 3773567 4111139264 CON
> 07-08-15 14:21:24 man 1614388 0 3817237 2 1001749 25952 3817237 4162150304 CON
> 07-08-15 14:22:24 man 1633226 0 3862001 2 1043290 25980 3862001 4213899504 CON
> 07-08-15 14:23:24 man 1653651 0 3910911 2 981144 29099 3910911 4270437224 CON
> 07-08-15 14:24:24 man 1674914 0 3961760 2 951822 29301 3961760 33357568 CON
> 07-08-15 14:25:24 man 1694708 0 4007603 2 959291 28480 4007603 87145088 CON
> 07-08-15 14:26:24 man 1715179 0 4055899 2 918180 28432 4055899 142542528 CON
> 07-08-15 14:27:24 man 1735529 0 4103903 2 930887 28628 4103903 197986648 CON
> 07-08-15 14:28:24 man 1756438 0 4153460 2 964800 29293 4153460 254923688 CON
> 07-08-15 14:29:24 man 1775701 0 4198996 2 861857 28488 4198996 308770728 CON
> 07-08-15 14:30:24 man 1795511 0 4246466 2 905501 28822 4246466 364036888 CON
> 07-08-15 14:31:24 man 1815059 0 4293349 2 842655 25969 4293349 417321168 CON
> 07-08-15 14:32:24 man 1834548 0 4340388 2 805952 25792 4340388 470724008 CON
> 07-08-15 14:33:24 man 1854168 0 4387327 2 814452 25715 4387327 524127728 CON
> 07-08-15 14:34:24 man 1873533 0 4433935 2 830901 25440 4433935 576737568 CON
> 07-08-15 14:35:24 man 1892766 0 4480385 2 841707 25568 4480385 629654888 CON
> 07-08-15 14:36:24 man 1911651 0 4525593 2 922177 26136 4525593 681994888 CON
> 07-08-15 14:37:24 man 1930756 0 4571308 2 966213 26494 4571308 734984768 CON
> 07-08-15 14:38:24 man 1955339 0 4627545 2 989488 25548 4627545 795340128 CON
> 07-08-15 14:39:24 man 1974897 0 4673706 2 896919 25180 4673706 847976928 CON
> 07-08-15 14:40:24 man 1994202 0 4718619 2 792368 26060 4718619 900271048 CON
> 07-08-15 14:41:24 man 2014358 0 4766426 2 836465 28547 4766426 955643808 CON
> 07-08-15 14:42:24 man 2032739 0 4809490 2 876541 26979 4809490 1006861768 CON
> 07-08-15 14:43:24 man 2052275 0 4855722 2 853278 28120 4855722 1060911008 CON
> 07-08-15 14:44:24 man 2071728 0 4901761 2 797199 28042 4901761 1114472568 CON
> 07-08-15 14:45:24 man 2092752 0 4951286 2 889380 29089 4951286 1171663688 CON
> 07-08-15 14:46:24 man 2113794 0 4999579 2 899038 29712 4999579 1228471528 CON
> 07-08-15 14:47:24 man 2133195 0 5045435 2 817684 29277 5045435 1283055208 CON
> 07-08-15 14:48:24 man 2153557 0 5094075 2 867185 29250 5094075 1339515288 CON
> 07-08-15 14:49:24 man 2173843 0 5143886 2 886131 28150 5143886 1395881088 CON
> 07-08-15 14:50:24 man 2193392 0 5190511 2 898308 27491 5190511 1450181448 CON
> 07-08-15 14:51:24 man 2210999 0 5232842 2 854348 25653 5232842 1500051688 CON
> 07-08-15 14:52:24 man 2228308 0 5274645 2 824314 24738 5274645 1548967728 CON
> 07-08-15 14:53:24 man 2246959 0 5320603 2 877264 25841 5320603 1601377368 CON
> 07-08-15 14:54:24 man 2266623 0 5368579 2 878698 25483 5368579 1654951448 CON
> 07-08-15 14:55:24 man 2285971 0 5415552 2 926918 25879 5415552 1707955448 CON
> 07-08-15 14:56:24 man 2301960 0 5445780 2 389347 15687 5445780 1742634408 CON
> 07-08-15 14:57:24 man 2314613 0 5466755 2 147994 10038 5466755 1765850208 CON
> 07-08-15 14:58:24 man 2325099 0 5484015 2 97277 7996 5484015 1784658408 CON
> 07-08-15 14:59:24 man 2333757 0 5498041 2 68135 6704 5498041 1799888208 CON
> test4:/var/log/argus vanepp$
>
> The same hour from the 2.0.6 sensor (which is listening on the same regen
> tap as 3.0) the flows should be the same but 2.0.6 is handling it and 3.0 isn't
> (and the 2.0.6 sensor has only 1 gig of memory not 4)
>
> nepp at sniffer:/var/log/argus> ra -r /usr/local/argus/com_argus.archive/2007/08/15/com_argus.2007.08.15.14.00.00.0.gz -nn man
> 22 Jun 07 07:36:23 man 229.97.122.203 v2.0 1 0 0 0 0 0 STA
> 15 Aug 07 13:56:27 man 229.97.122.203 v2.0 2299836494 1308855001157 0 3426164216 130783 CON
> 15 Aug 07 14:01:27 man 229.97.122.203 v2.0 2300022324 1242885022304 0 3454202706 139418 CON
> 15 Aug 07 14:06:27 man 229.97.122.203 v2.0 2300205390 1201094752165 0 3171199840 135152 CON
> 15 Aug 07 14:11:27 man 229.97.122.203 v2.0 2300376928 1220234876630 0 3360631524 122151 CON
> 15 Aug 07 14:16:27 man 229.97.122.203 v2.0 2300552396 1232065068302 0 3576425899 122621 CON
> 15 Aug 07 14:21:27 man 229.97.122.203 v2.0 2300728449 1211074847986 0 3346915526 129684 CON
> 15 Aug 07 14:26:27 man 229.97.122.203 v2.0 2300904485 1200494506089 0 2933591340 127401 CON
> 15 Aug 07 14:31:27 man 229.97.122.203 v2.0 2301070396 1197084221477 0 2643569732 119551 CON
> 15 Aug 07 14:36:27 man 229.97.122.203 v2.0 2301243118 1203024476135 0 2957584557 125656 CON
> 15 Aug 07 14:41:27 man 229.97.122.203 v2.0 2301419626 1213234313786 0 2692091848 126701 CON
> 15 Aug 07 14:46:27 man 229.97.122.203 v2.0 2301595126 1227394322412 0 2660038990 122037 CON
> 15 Aug 07 14:51:27 man 229.97.122.203 v2.0 2301758903 1228524444856 0 2784876374 118980 CON
> vanepp at sniffer:/var/log/argus>
>
> the 2.0.6 sensor has a much smaller footprint on the same traffic:
>
> %!ps
> ps auxwwww | grep argus
> root 944 2.0 20.5 215068 214168 ?? Ss 22Jun07 3162:09.10 /usr/local/bin/argus_bpf -dJR -P 561 -i em2 -i em3
>
>
> I'm wondering if 3.0 is keeping more flows open for some reason and
> thus eating a lot more memory (or is failing to close flows it should be and
> running out of memory because of it). At this point it looks like the memory
> allocation is pretty clean just something is using too much of it.
>
> Peter Van Epp / Operations and Technical Support
> Simon Fraser University, Burnaby, B.C. Canada
>
More information about the argus
mailing list