Argus memory issues
Carter Bullard
carter at qosient.com
Mon Aug 20 09:24:23 EDT 2007
Gentle men,
argus-3.0 is trying to do a lot more than argus-2.0 every tried to do,
and the memory is the thing that is impacted the most.
So, two things, while I ponder what to do, does this option provide
suitable memory use for your new probes:
ARGUS_FLOW_KEY="LAYER_3_MATRIX"
it won't be as light as argus-2.0 based on the current structure, but
it lowers cpu and memory use. If it is still eating memory, then
a more aggressive timeout is indeed indicated.
Second, how is your cpu during this? If we have some cycles,
I have a strategy that will have more dynamic memory allocs, but
runs a lot smaller because of the traffic mix.
I've been doing this argus-3.0 for too long now, without releasing
something, so hopefully we can find an approach that will get this
out the door and then I can fix it right later.
Carter
On Aug 19, 2007, at 11:50 PM, Russell Fulton wrote:
> my sensors are seeing around two to three million flows per hour --
> not
> that high. From observing the memory footprint of my argus instances
> after startup I see what appears to be a classic exponential
> approach to
> a more or less stable limit. Which fits with it being flow related
> not
> a leak which we would expect to be linear and unbounded.
>
> Looks like what we need to examine is the flow timeout or garbage
> collection.
>
> One thing that I find interesting is that the memory is firmly
> locked in
> physical memory suggesting that it must be being used all the time.
> Snort OTOH only manages to keep half its memory from being swapped.
>
> Russell
>
> Peter Van Epp wrote:
>> On Sun, Aug 19, 2007 at 11:00:07PM -0400, Carter Bullard wrote:
>>
>>> So it looks like we could be just processing a massive amount of
>>> flows. I do have a fixed memory model for argus, so that we don't
>>> grow over a certain memory foot print, we'll aggressively reclaim
>>> records and hold low packet flows for only the status interval,
>>> (usually 5 secs), that may work very well for you.
>>>
>>> So what are we thinking then, are we not leaking memory and
>>> you've got
>>> a lot of flows? Should I switch to thinking about how to keep your
>>> argus better behaved?
>>>
>>> Carter
>>>
>>>
>>
>> Yes sort of. The puzzling part is that my 2.0.6 system is still
>> chugging
>> along fine on the same data (but without user data) that is
>> currently choking
>> 3.0 (at this point without threads, user data or even output to a
>> client).
>>
>> argus-3.0 (chosen from an hour that didn't seem to run out of
>> memory :-)):
>>
>> test4:/var/log/argus vanepp$ ra3 -r /archive/argus3/
>> com_argus.archive/2007/08/15/com_argus.2007.08.15.14.00.00.0.gz -n
>> man
>> 07-08-15 14:00:24 man 1172637
>> 0 2786116 2 980417 27043 2786116
>> 2980874864 CON
>> 07-08-15 14:01:24 man 1195209
>> 0 2840347 2 966201 27590 2840347
>> 3040374744 CON
>> 07-08-15 14:02:24 man 1217026
>> 0 2891733 2 1000500 27616 2891733
>> 3098033784 CON
>> 07-08-15 14:03:24 man 1239216
>> 0 2944188 2 1000624 27052 2944188
>> 3155959864 CON
>> 07-08-15 14:04:24 man 1259885
>> 0 2993361 2 1045410 26387 2993361
>> 3210878984 CON
>> 07-08-15 14:05:24 man 1286106
>> 0 3054195 2 1035640 27954 3054195
>> 3275422984 CON
>> 07-08-15 14:06:24 man 1306951
>> 0 3102666 2 941859 28763 3102666
>> 3331541144 CON
>> 07-08-15 14:07:24 man 1327541
>> 0 3150287 2 976290 29432 3150287
>> 3387613104 CON
>> 07-08-15 14:08:24 man 1354025
>> 0 3205331 2 969769 30268 3205331
>> 3450989784 CON
>> 07-08-15 14:09:24 man 1374984
>> 0 3253937 2 970571 28929 3253937
>> 3507411144 CON
>> 07-08-15 14:10:24 man 1393511
>> 0 3297526 2 922896 27879 3297526
>> 3559330344 CON
>> 07-08-15 14:11:24 man 1412644
>> 0 3341769 2 911490 29793 3341769
>> 3612703624 CON
>> 07-08-15 14:12:24 man 1432303
>> 0 3388174 2 950633 28603 3388174
>> 3667129144 CON
>> 07-08-15 14:13:24 man 1452892
>> 0 3436351 2 952619 29755 3436351
>> 3723621824 CON
>> 07-08-15 14:14:24 man 1472367
>> 0 3481550 2 968650 29561 3481550
>> 3777819144 CON
>> 07-08-15 14:15:24 man 1492529
>> 0 3529187 2 1011118 28944 3529187
>> 3833547544 CON
>> 07-08-15 14:16:24 man 1513232
>> 0 3577628 2 992951 27440 3577628
>> 3889140944 CON
>> 07-08-15 14:17:24 man 1534016
>> 0 3626696 2 1014643 27236 3626696
>> 3945081104 CON
>> 07-08-15 14:18:24 man 1555291
>> 0 3676958 2 1037774 26918 3676958
>> 4001647144 CON
>> 07-08-15 14:19:24 man 1576296
>> 0 3726748 2 1032067 26140 3726748
>> 4057477624 CON
>> 07-08-15 14:20:24 man 1595717
>> 0 3773567 2 982867 26605 3773567
>> 4111139264 CON
>> 07-08-15 14:21:24 man 1614388
>> 0 3817237 2 1001749 25952 3817237
>> 4162150304 CON
>> 07-08-15 14:22:24 man 1633226
>> 0 3862001 2 1043290 25980 3862001
>> 4213899504 CON
>> 07-08-15 14:23:24 man 1653651
>> 0 3910911 2 981144 29099 3910911
>> 4270437224 CON
>> 07-08-15 14:24:24 man 1674914
>> 0 3961760 2 951822 29301
>> 3961760 33357568 CON
>> 07-08-15 14:25:24 man 1694708
>> 0 4007603 2 959291 28480
>> 4007603 87145088 CON
>> 07-08-15 14:26:24 man 1715179
>> 0 4055899 2 918180 28432
>> 4055899 142542528 CON
>> 07-08-15 14:27:24 man 1735529
>> 0 4103903 2 930887 28628
>> 4103903 197986648 CON
>> 07-08-15 14:28:24 man 1756438
>> 0 4153460 2 964800 29293
>> 4153460 254923688 CON
>> 07-08-15 14:29:24 man 1775701
>> 0 4198996 2 861857 28488
>> 4198996 308770728 CON
>> 07-08-15 14:30:24 man 1795511
>> 0 4246466 2 905501 28822
>> 4246466 364036888 CON
>> 07-08-15 14:31:24 man 1815059
>> 0 4293349 2 842655 25969
>> 4293349 417321168 CON
>> 07-08-15 14:32:24 man 1834548
>> 0 4340388 2 805952 25792
>> 4340388 470724008 CON
>> 07-08-15 14:33:24 man 1854168
>> 0 4387327 2 814452 25715
>> 4387327 524127728 CON
>> 07-08-15 14:34:24 man 1873533
>> 0 4433935 2 830901 25440
>> 4433935 576737568 CON
>> 07-08-15 14:35:24 man 1892766
>> 0 4480385 2 841707 25568
>> 4480385 629654888 CON
>> 07-08-15 14:36:24 man 1911651
>> 0 4525593 2 922177 26136
>> 4525593 681994888 CON
>> 07-08-15 14:37:24 man 1930756
>> 0 4571308 2 966213 26494
>> 4571308 734984768 CON
>> 07-08-15 14:38:24 man 1955339
>> 0 4627545 2 989488 25548
>> 4627545 795340128 CON
>> 07-08-15 14:39:24 man 1974897
>> 0 4673706 2 896919 25180
>> 4673706 847976928 CON
>> 07-08-15 14:40:24 man 1994202
>> 0 4718619 2 792368 26060
>> 4718619 900271048 CON
>> 07-08-15 14:41:24 man 2014358
>> 0 4766426 2 836465 28547
>> 4766426 955643808 CON
>> 07-08-15 14:42:24 man 2032739
>> 0 4809490 2 876541 26979 4809490
>> 1006861768 CON
>> 07-08-15 14:43:24 man 2052275
>> 0 4855722 2 853278 28120 4855722
>> 1060911008 CON
>> 07-08-15 14:44:24 man 2071728
>> 0 4901761 2 797199 28042 4901761
>> 1114472568 CON
>> 07-08-15 14:45:24 man 2092752
>> 0 4951286 2 889380 29089 4951286
>> 1171663688 CON
>> 07-08-15 14:46:24 man 2113794
>> 0 4999579 2 899038 29712 4999579
>> 1228471528 CON
>> 07-08-15 14:47:24 man 2133195
>> 0 5045435 2 817684 29277 5045435
>> 1283055208 CON
>> 07-08-15 14:48:24 man 2153557
>> 0 5094075 2 867185 29250 5094075
>> 1339515288 CON
>> 07-08-15 14:49:24 man 2173843
>> 0 5143886 2 886131 28150 5143886
>> 1395881088 CON
>> 07-08-15 14:50:24 man 2193392
>> 0 5190511 2 898308 27491 5190511
>> 1450181448 CON
>> 07-08-15 14:51:24 man 2210999
>> 0 5232842 2 854348 25653 5232842
>> 1500051688 CON
>> 07-08-15 14:52:24 man 2228308
>> 0 5274645 2 824314 24738 5274645
>> 1548967728 CON
>> 07-08-15 14:53:24 man 2246959
>> 0 5320603 2 877264 25841 5320603
>> 1601377368 CON
>> 07-08-15 14:54:24 man 2266623
>> 0 5368579 2 878698 25483 5368579
>> 1654951448 CON
>> 07-08-15 14:55:24 man 2285971
>> 0 5415552 2 926918 25879 5415552
>> 1707955448 CON
>> 07-08-15 14:56:24 man 2301960
>> 0 5445780 2 389347 15687 5445780
>> 1742634408 CON
>> 07-08-15 14:57:24 man 2314613
>> 0 5466755 2 147994 10038 5466755
>> 1765850208 CON
>> 07-08-15 14:58:24 man 2325099
>> 0 5484015 2 97277 7996 5484015
>> 1784658408 CON
>> 07-08-15 14:59:24 man 2333757
>> 0 5498041 2 68135 6704 5498041
>> 1799888208 CON
>> test4:/var/log/argus vanepp$
>>
>> The same hour from the 2.0.6 sensor (which is listening on the
>> same regen
>> tap as 3.0) the flows should be the same but 2.0.6 is handling it
>> and 3.0 isn't
>> (and the 2.0.6 sensor has only 1 gig of memory not 4)
>>
>> nepp at sniffer:/var/log/argus> ra -r /usr/local/argus/
>> com_argus.archive/2007/08/15/com_argus.2007.08.15.14.00.00.0.gz -
>> nn man
>> 22 Jun 07 07:36:23 man 229.97.122.203
>> v2.0 1 0 0 0 0
>> 0 STA
>> 15 Aug 07 13:56:27 man 229.97.122.203 v2.0
>> 2299836494 1308855001157 0 3426164216 130783 CON
>> 15 Aug 07 14:01:27 man 229.97.122.203 v2.0
>> 2300022324 1242885022304 0 3454202706 139418 CON
>> 15 Aug 07 14:06:27 man 229.97.122.203 v2.0
>> 2300205390 1201094752165 0 3171199840 135152 CON
>> 15 Aug 07 14:11:27 man 229.97.122.203 v2.0
>> 2300376928 1220234876630 0 3360631524 122151 CON
>> 15 Aug 07 14:16:27 man 229.97.122.203 v2.0
>> 2300552396 1232065068302 0 3576425899 122621 CON
>> 15 Aug 07 14:21:27 man 229.97.122.203 v2.0
>> 2300728449 1211074847986 0 3346915526 129684 CON
>> 15 Aug 07 14:26:27 man 229.97.122.203 v2.0
>> 2300904485 1200494506089 0 2933591340 127401 CON
>> 15 Aug 07 14:31:27 man 229.97.122.203 v2.0
>> 2301070396 1197084221477 0 2643569732 119551 CON
>> 15 Aug 07 14:36:27 man 229.97.122.203 v2.0
>> 2301243118 1203024476135 0 2957584557 125656 CON
>> 15 Aug 07 14:41:27 man 229.97.122.203 v2.0
>> 2301419626 1213234313786 0 2692091848 126701 CON
>> 15 Aug 07 14:46:27 man 229.97.122.203 v2.0
>> 2301595126 1227394322412 0 2660038990 122037 CON
>> 15 Aug 07 14:51:27 man 229.97.122.203 v2.0
>> 2301758903 1228524444856 0 2784876374 118980 CON
>> vanepp at sniffer:/var/log/argus>
>>
>> the 2.0.6 sensor has a much smaller footprint on the same traffic:
>>
>> %!ps
>> ps auxwwww | grep argus
>> root 944 2.0 20.5 215068 214168 ?? Ss 22Jun07 3162:09.10 /
>> usr/local/bin/argus_bpf -dJR -P 561 -i em2 -i em3
>>
>>
>> I'm wondering if 3.0 is keeping more flows open for some reason and
>> thus eating a lot more memory (or is failing to close flows it
>> should be and
>> running out of memory because of it). At this point it looks like
>> the memory
>> allocation is pretty clean just something is using too much of it.
>>
>> Peter Van Epp / Operations and Technical Support
>> Simon Fraser University, Burnaby, B.C. Canada
>>
>
More information about the argus
mailing list