Argus memory issues

Russell Fulton r.fulton at auckland.ac.nz
Sun Aug 19 23:50:07 EDT 2007


my sensors are seeing around two to three million flows per hour -- not
that high.  From observing the memory footprint of my argus instances
after startup I see what appears to be a classic exponential approach to
a more or less stable limit.  Which fits with it being flow related not
a leak which we would expect to be linear and unbounded.

Looks like what we need to examine is the flow timeout or garbage
collection.

One thing that I find interesting is that the memory is firmly locked in
physical memory suggesting that it must be being used all the time.  
Snort OTOH only manages to keep half its memory from being swapped.

Russell

Peter Van Epp wrote:
> On Sun, Aug 19, 2007 at 11:00:07PM -0400, Carter Bullard wrote:
>   
>> So it looks like we could be just processing a massive amount of
>> flows.  I do have a fixed memory model for argus, so that we don't
>> grow over a certain memory foot print, we'll aggressively reclaim
>> records and hold low packet flows for only the status interval,
>> (usually 5 secs),  that may work very well for you.
>>
>> So what are we thinking then, are we not leaking memory and you've got
>> a lot of flows?  Should I switch to thinking about how to keep your
>> argus better behaved?
>>
>> Carter
>>
>>     
>
> 	Yes sort of. The puzzling part is that my 2.0.6 system is still chugging
> along fine on the same data (but without user data) that is currently choking
> 3.0 (at this point without threads, user data or even output to a client).
>
> argus-3.0 (chosen from an hour that didn't seem to run out of memory :-)):
>
> test4:/var/log/argus vanepp$ ra3 -r /archive/argus3/com_argus.archive/2007/08/15/com_argus.2007.08.15.14.00.00.0.gz -n man
> 07-08-15 14:00:24             man            1172637      0                  2786116      2   980417    27043      2786116   2980874864   CON
> 07-08-15 14:01:24             man            1195209      0                  2840347      2   966201    27590      2840347   3040374744   CON
> 07-08-15 14:02:24             man            1217026      0                  2891733      2  1000500    27616      2891733   3098033784   CON
> 07-08-15 14:03:24             man            1239216      0                  2944188      2  1000624    27052      2944188   3155959864   CON
> 07-08-15 14:04:24             man            1259885      0                  2993361      2  1045410    26387      2993361   3210878984   CON
> 07-08-15 14:05:24             man            1286106      0                  3054195      2  1035640    27954      3054195   3275422984   CON
> 07-08-15 14:06:24             man            1306951      0                  3102666      2   941859    28763      3102666   3331541144   CON
> 07-08-15 14:07:24             man            1327541      0                  3150287      2   976290    29432      3150287   3387613104   CON
> 07-08-15 14:08:24             man            1354025      0                  3205331      2   969769    30268      3205331   3450989784   CON
> 07-08-15 14:09:24             man            1374984      0                  3253937      2   970571    28929      3253937   3507411144   CON
> 07-08-15 14:10:24             man            1393511      0                  3297526      2   922896    27879      3297526   3559330344   CON
> 07-08-15 14:11:24             man            1412644      0                  3341769      2   911490    29793      3341769   3612703624   CON
> 07-08-15 14:12:24             man            1432303      0                  3388174      2   950633    28603      3388174   3667129144   CON
> 07-08-15 14:13:24             man            1452892      0                  3436351      2   952619    29755      3436351   3723621824   CON
> 07-08-15 14:14:24             man            1472367      0                  3481550      2   968650    29561      3481550   3777819144   CON
> 07-08-15 14:15:24             man            1492529      0                  3529187      2  1011118    28944      3529187   3833547544   CON
> 07-08-15 14:16:24             man            1513232      0                  3577628      2   992951    27440      3577628   3889140944   CON
> 07-08-15 14:17:24             man            1534016      0                  3626696      2  1014643    27236      3626696   3945081104   CON
> 07-08-15 14:18:24             man            1555291      0                  3676958      2  1037774    26918      3676958   4001647144   CON
> 07-08-15 14:19:24             man            1576296      0                  3726748      2  1032067    26140      3726748   4057477624   CON
> 07-08-15 14:20:24             man            1595717      0                  3773567      2   982867    26605      3773567   4111139264   CON
> 07-08-15 14:21:24             man            1614388      0                  3817237      2  1001749    25952      3817237   4162150304   CON
> 07-08-15 14:22:24             man            1633226      0                  3862001      2  1043290    25980      3862001   4213899504   CON
> 07-08-15 14:23:24             man            1653651      0                  3910911      2   981144    29099      3910911   4270437224   CON
> 07-08-15 14:24:24             man            1674914      0                  3961760      2   951822    29301      3961760     33357568   CON
> 07-08-15 14:25:24             man            1694708      0                  4007603      2   959291    28480      4007603     87145088   CON
> 07-08-15 14:26:24             man            1715179      0                  4055899      2   918180    28432      4055899    142542528   CON
> 07-08-15 14:27:24             man            1735529      0                  4103903      2   930887    28628      4103903    197986648   CON
> 07-08-15 14:28:24             man            1756438      0                  4153460      2   964800    29293      4153460    254923688   CON
> 07-08-15 14:29:24             man            1775701      0                  4198996      2   861857    28488      4198996    308770728   CON
> 07-08-15 14:30:24             man            1795511      0                  4246466      2   905501    28822      4246466    364036888   CON
> 07-08-15 14:31:24             man            1815059      0                  4293349      2   842655    25969      4293349    417321168   CON
> 07-08-15 14:32:24             man            1834548      0                  4340388      2   805952    25792      4340388    470724008   CON
> 07-08-15 14:33:24             man            1854168      0                  4387327      2   814452    25715      4387327    524127728   CON
> 07-08-15 14:34:24             man            1873533      0                  4433935      2   830901    25440      4433935    576737568   CON
> 07-08-15 14:35:24             man            1892766      0                  4480385      2   841707    25568      4480385    629654888   CON
> 07-08-15 14:36:24             man            1911651      0                  4525593      2   922177    26136      4525593    681994888   CON
> 07-08-15 14:37:24             man            1930756      0                  4571308      2   966213    26494      4571308    734984768   CON
> 07-08-15 14:38:24             man            1955339      0                  4627545      2   989488    25548      4627545    795340128   CON
> 07-08-15 14:39:24             man            1974897      0                  4673706      2   896919    25180      4673706    847976928   CON
> 07-08-15 14:40:24             man            1994202      0                  4718619      2   792368    26060      4718619    900271048   CON
> 07-08-15 14:41:24             man            2014358      0                  4766426      2   836465    28547      4766426    955643808   CON
> 07-08-15 14:42:24             man            2032739      0                  4809490      2   876541    26979      4809490   1006861768   CON
> 07-08-15 14:43:24             man            2052275      0                  4855722      2   853278    28120      4855722   1060911008   CON
> 07-08-15 14:44:24             man            2071728      0                  4901761      2   797199    28042      4901761   1114472568   CON
> 07-08-15 14:45:24             man            2092752      0                  4951286      2   889380    29089      4951286   1171663688   CON
> 07-08-15 14:46:24             man            2113794      0                  4999579      2   899038    29712      4999579   1228471528   CON
> 07-08-15 14:47:24             man            2133195      0                  5045435      2   817684    29277      5045435   1283055208   CON
> 07-08-15 14:48:24             man            2153557      0                  5094075      2   867185    29250      5094075   1339515288   CON
> 07-08-15 14:49:24             man            2173843      0                  5143886      2   886131    28150      5143886   1395881088   CON
> 07-08-15 14:50:24             man            2193392      0                  5190511      2   898308    27491      5190511   1450181448   CON
> 07-08-15 14:51:24             man            2210999      0                  5232842      2   854348    25653      5232842   1500051688   CON
> 07-08-15 14:52:24             man            2228308      0                  5274645      2   824314    24738      5274645   1548967728   CON
> 07-08-15 14:53:24             man            2246959      0                  5320603      2   877264    25841      5320603   1601377368   CON
> 07-08-15 14:54:24             man            2266623      0                  5368579      2   878698    25483      5368579   1654951448   CON
> 07-08-15 14:55:24             man            2285971      0                  5415552      2   926918    25879      5415552   1707955448   CON
> 07-08-15 14:56:24             man            2301960      0                  5445780      2   389347    15687      5445780   1742634408   CON
> 07-08-15 14:57:24             man            2314613      0                  5466755      2   147994    10038      5466755   1765850208   CON
> 07-08-15 14:58:24             man            2325099      0                  5484015      2    97277     7996      5484015   1784658408   CON
> 07-08-15 14:59:24             man            2333757      0                  5498041      2    68135     6704      5498041   1799888208   CON
> test4:/var/log/argus vanepp$
>
> The same hour from the 2.0.6 sensor (which is listening on the same regen 
> tap as 3.0) the flows should be the same but 2.0.6 is handling it and 3.0 isn't
> (and the 2.0.6 sensor has only 1 gig of memory not 4)
>
> nepp at sniffer:/var/log/argus> ra -r /usr/local/argus/com_argus.archive/2007/08/15/com_argus.2007.08.15.14.00.00.0.gz -nn man
> 22 Jun 07 07:36:23           man  229.97.122.203  v2.0                   1 0     0        0         0            0           STA
> 15 Aug 07 13:56:27           man  229.97.122.203  v2.0          2299836494 1308855001157  0         3426164216   130783      CON
> 15 Aug 07 14:01:27           man  229.97.122.203  v2.0          2300022324 1242885022304  0         3454202706   139418      CON
> 15 Aug 07 14:06:27           man  229.97.122.203  v2.0          2300205390 1201094752165  0         3171199840   135152      CON
> 15 Aug 07 14:11:27           man  229.97.122.203  v2.0          2300376928 1220234876630  0         3360631524   122151      CON
> 15 Aug 07 14:16:27           man  229.97.122.203  v2.0          2300552396 1232065068302  0         3576425899   122621      CON
> 15 Aug 07 14:21:27           man  229.97.122.203  v2.0          2300728449 1211074847986  0         3346915526   129684      CON
> 15 Aug 07 14:26:27           man  229.97.122.203  v2.0          2300904485 1200494506089  0         2933591340   127401      CON
> 15 Aug 07 14:31:27           man  229.97.122.203  v2.0          2301070396 1197084221477  0         2643569732   119551      CON
> 15 Aug 07 14:36:27           man  229.97.122.203  v2.0          2301243118 1203024476135  0         2957584557   125656      CON
> 15 Aug 07 14:41:27           man  229.97.122.203  v2.0          2301419626 1213234313786  0         2692091848   126701      CON
> 15 Aug 07 14:46:27           man  229.97.122.203  v2.0          2301595126 1227394322412  0         2660038990   122037      CON
> 15 Aug 07 14:51:27           man  229.97.122.203  v2.0          2301758903 1228524444856  0         2784876374   118980      CON
> vanepp at sniffer:/var/log/argus>
>
> 	the 2.0.6 sensor has a much smaller footprint on the same traffic:
>
> %!ps
> ps auxwwww | grep argus
> root     944  2.0 20.5 215068 214168  ??  Ss   22Jun07 3162:09.10 /usr/local/bin/argus_bpf -dJR -P 561 -i em2 -i em3
>
>
> 	I'm wondering if 3.0 is keeping more flows open for some reason and 
> thus eating a lot more memory (or is failing to close flows it should be and
> running out of memory because of it).  At this point it looks like the memory
> allocation is pretty clean just something is using too much of it. 
>
> Peter Van Epp / Operations and Technical Support 
> Simon Fraser University, Burnaby, B.C. Canada
>   



More information about the argus mailing list