memory use, bidirectional behavior
MN
m.newton at stanford.edu
Fri Aug 24 16:16:57 EDT 2007
A few data points that might possibly help with the memory leaks and
which also show a strange behavior regarding CPU utilization as well:
I've been testing Argus on some new donated boxes with 10G cards (266MHz
PCI-X buses, x86/32bit), but with only 1G interfaces currently in use.
The two directions of the currently tapped link show up on eth0 and eth1.
Disabling either link and restarting Argus produces a top that looks like:
top - 12:47:53 up 1 day, 21:39, 1 user, load average: 0.99, 1.03, 1.10
Tasks: 71 total, 1 running, 70 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.0%us, 0.0%sy, 0.0%ni, 90.7%id, 0.0%wa, 3.3%hi, 6.0%si, 0.0%st
Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 0.0%us, 0.0%sy, 0.0%ni, 94.0%id, 0.0%wa, 1.3%hi, 4.7%si, 0.0%st
Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2075440k total, 37136k used, 2038304k free, 1044k buffers
Swap: 7815612k total, 8408k used, 7807204k free, 4800k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24833 root 18 0 4672 2264 528 D 0 0.1 0:00.00 argus
even after running for a while. Note that all processors are 90+ idle,
(often all 95% idle) but the load average is ~1 - weird. Also the small
memory use (which is stable).
About 20 seconds after enabling the second interface:
top - 12:54:15 up 1 day, 21:46, 1 user, load average: 0.98, 1.02, 1.16
Tasks: 71 total, 2 running, 69 sleeping, 0 stopped, 0 zombie
Cpu0 : 6.3%us, 13.3%sy, 0.0%ni, 70.7%id, 0.0%wa, 1.0%hi, 8.7%si, 0.0%st
Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 6.7%us, 7.7%sy, 0.0%ni, 78.0%id, 0.0%wa, 1.0%hi, 6.7%si, 0.0%st
Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2075440k total, 120780k used, 1954660k free, 1800k buffers
Swap: 7815612k total, 8404k used, 7807208k free, 7668k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24833 root 15 0 84392 79m 528 R 42 3.9 0:06.39 argus
Still running with both links enabled, after another 1.5 minutes:
top - 12:55:45 up 1 day, 21:47, 1 user, load average: 0.60, 0.89, 1.10
Tasks: 71 total, 2 running, 69 sleeping, 0 stopped, 0 zombie
Cpu0 : 14.3%us, 17.3%sy, 0.0%ni, 48.8%id, 0.0%wa, 3.3%hi, 16.3%si, 0.0%st
Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 11.0%us, 17.9%sy, 0.0%ni, 46.5%id, 0.0%wa, 5.0%hi, 19.6%si, 0.0%st
Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2075440k total, 354272k used, 1721168k free, 2004k buffers
Swap: 7815612k total, 8404k used, 7807208k free, 48232k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24833 root 15 0 269m 267m 528 R 76 13.2 1:07.10 argus
There's nothing else running on this box besides linux daemons & an ssh
server. But with arugs now using 76% of a CPU, the long term load average
settles on that amount - makes more sense than 100% ("1") load average for
a process claiming to use ~ 0%. Also the memory is rapidly growing.
A few minutes more:
top - 12:58:51 up 1 day, 21:50, 1 user, load average: 0.66, 0.82, 1.03
Tasks: 71 total, 2 running, 69 sleeping, 0 stopped, 0 zombie
Cpu0 : 15.3%us, 14.0%sy, 0.0%ni, 50.8%id, 0.0%wa, 1.7%hi, 18.3%si, 0.0%st
Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 15.0%us, 15.9%sy, 0.0%ni, 43.5%id, 0.0%wa, 2.7%hi, 22.9%si, 0.0%st
Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2075440k total, 1002544k used, 1072896k free, 2392k buffers
Swap: 7815612k total, 8404k used, 7807208k free, 109264k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24833 root 15 0 840m 838m 528 R 75 41.4 3:24.94 argus
It will rise to at least 2G virtual over a few hours, at which point
it starts thrashing the box.
Here's a few minutes later (load average is a little high due to
start-of-hour cron daemons):
top - 13:03:43 up 1 day, 21:55, 1 user, load average: 0.87, 0.87, 0.99
Tasks: 71 total, 2 running, 69 sleeping, 0 stopped, 0 zombie
Cpu0 : 8.7%us, 22.0%sy, 0.0%ni, 50.3%id, 0.0%wa, 2.0%hi, 17.0%si, 0.0%st
Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 12.7%us, 16.7%sy, 0.0%ni, 47.3%id, 0.0%wa, 3.3%hi, 20.0%si, 0.0%st
Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2075440k total, 1817512k used, 257928k free, 3504k buffers
Swap: 7815612k total, 8324k used, 7807288k free, 139932k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24833 root 15 0 1602m 1.6g 528 R 74 79.0 6:59.36 argus
It's already up to 1.6g.
================
Restarting Argus, with both links enabled, running for a minute and
then disabling a link:
top - 13:10:28 up 1 day, 22:02, 1 user, load average: 0.88, 1.21, 1.13
Tasks: 71 total, 1 running, 70 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.2%us, 4.2%sy, 0.0%ni, 86.8%id, 0.0%wa, 0.8%hi, 5.0%si, 0.0%st
Mem: 2075440k total, 380352k used, 1695088k free, 528k buffers
Swap: 7815612k total, 8456k used, 7807156k free, 21244k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24862 root 15 0 324m 322m 620 S 36 15.9 0:44.47 argus
top - 13:10:46 up 1 day, 22:02, 1 user, load average: 0.63, 1.13, 1.10
Tasks: 71 total, 1 running, 70 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.5%us, 5.5%sy, 0.0%ni, 86.7%id, 0.0%wa, 0.9%hi, 3.4%si, 0.0%st
Mem: 2075440k total, 433424k used, 1642016k free, 564k buffers
Swap: 7815612k total, 8456k used, 7807156k free, 25872k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24862 root 15 0 371m 369m 620 S 42 18.2 0:51.67 argus
So, even if the link is later disabled, once enabled it will cause
memory to keep growing.
This is w/ the version of Argus on the server yesterday, and with,
depending upon time of day and season of school, 5-20 million flows/hour.
Hope this helps...
- mike
More information about the argus
mailing list