argus memory use on Linux 2.2.18

[Chris Newton]

Thanks Carter, that answers a bunch of my questions.

Chris

>===== Original Message From <carter at qosient.com> =====
>Hey Chris,
>   Great questions.  The CMU guys can elect to tell you
>specifics about the entire CMU community, but I wouldn't
>be surprised with a 4-8X estimate of the number of machines
>CMU supports, but I have no information. Remember CMU has a
>lot of customers on the outside, as they support a number of
>Web servers/ftp servers and so the supported number of IP
>addresses may be in the 100K+ range easy.
>
>   Ok, that's 90K simultaneous flows, with about 200-400
>flows per second arrival rate, average over 60 sec period
>is about right.
>
>   For your hypothetical situation, the real performance
>issues are packet throughput, and disk throughput.  Its
>more bandwidth and duration of flows that get you with
>argus, rather than total systems supported, etc.  So for
>estimates of load, I've pretty  much gone with 5
>simultaneous flows per user system, 20 peak, with the
>average duration around 0.4 secs for local transactions,
>and 0.8 for remote (now these are all averages).  Servers
>get around 20 average, and 100 simultaneous flows, peak,
>per server just as rough numbers.  These of course are
>just rough numbers, so for your 5000 user machines, I'd
>expect anywhere from 25,000 to 100,000 max flows.  I'd
>be surprised to see 500Mbps from 5000 machines, but
>I'm always surprised ;o)
>
>   I would recommend a dual processor PIII 800-900 MHz
>machine with 250-500 MB of memory.  The memory bandwidth
>and disk performance are big issues, so I'd definitely do
>160Mbps SCSI, and the best chip set available (133MHz FSB).
>
>   I'd be very interested in some P4 and Athlon performance
>numbers, especially with the fastest memory bandwidth chipsets.
>I haven't seen any dual processor P4 machines yet, so that's
>why I recommend dual processor PIII's.
>
>   I'm not sure that Sparc's are much better, although
>the network interface performance may be better on Sun's
>than on generic PC's.
>
>Carter
>
>Carter Bullard
>QoSient, LLC
>300 E. 56th Street, Suite 18K
>New York, New York  10022
>
>carter at qosient.com
>Phone +1 212 588-9133
>Fax   +1 212 588-9134
>
>
>> -----Original Message-----
>> From: Chris Newton [mailto:newton at unb.ca]
>> Sent: Monday, February 19, 2001 12:43 PM
>> To: Argus (E-mail); Carter Bullard
>> Subject: RE: argus memory use on Linux 2.2.18
>>
>>
>> >   Argus does around 10-20K packets/sec in this situation,
>> >80-200 Mbps load, average around 10K pkts/sec, 95 Mbps, with
>> about 80-90K
>> >simultaneous flows, and uses about 18-20% CPU.  As I
>> mentioned earlier,
>> >about 400 bytes per flow, and I've seen this machine do as
>> >many as 500,000 simultaneous flows.
>> >
>> >   This is monitoring the CMU to the outside world link, I believe.
>> >
>>
>>
>>   Ok, sounds impressive.  How many machines are on CMU's
>> network, that are
>> generating these 90K flows?
>>
>>   Also, is that 90K flows seen over a period of time (say, 5
>> minutes), or is
>> that 90K flows at 'any given instant' with the total flows
>> for a period of
>> time being much higher?
>>
>>   I'm trying to get an idea of how busy a link (MB/s,
>> thousands of computers
>> on the local net), given limitless hardware (say I had money
>> to buy the
>> fastest box out there, with the fastest network card, fastest
>> disks and tons
>> of memory), could argus  generate accurate flow logs for.
>>
>>   For instance (with the fastest hardware available), can
>> argus generate
>> proper flow stats for a link running at 500 Mbit/s, with say
>> 5000 client
>> computers on the 'local' side, generating the numbers of
>> flows 5000 users
>> would generate?
>>
>>
>> Thanks
>>
>> Chris
>>
>>
>>
>>
>>
>>
>>
>>
>> >> -----Original Message-----
>> >> From: owner-argus at lists.andrew.cmu.edu
>> >> [mailto:owner-argus at lists.andrew.cmu.edu]On Behalf Of Chris Newton
>> >> Sent: Monday, February 19, 2001 11:56 AM
>> >> To: Argus (E-mail); Carter Bullard
>> >> Subject: RE: argus memory use on Linux 2.2.18
>> >>
>> >>
>> >> Hi,
>> >>
>> >>   This loaded machine, can you give me some details about the
>> >> link it is
>> >> watching (how many in/out MB/s), how many machines on the
>> >> network, and what
>> >> the class of the machine is (PIII 600Mhz?), and how much
>> >> memory?  I can't
>> >> really help with the problem you are having... but, I am
>> >> wondering how well
>> >> argus scales.  Also, what % CPU is argus using to watch this link?
>> >>
>> >> Thanks
>> >>
>> >> Chris
>> >>
>> >> >===== Original Message From <carter at qosient.com> =====
>> >> >Gentle people,
>> >> >   I'm going through memory utilization on a really
>> >> >loaded Linux box at CMU, and I've got a question on
>> >> >Linux memory use.
>> >> >
>> >> >   What's with ps reporting memory use?  It seems that
>> >> >regardless of actual memory allocation, ps seems to report
>> >> >the max memory allocated.  Is this correct?
>> >> >
>> >> >   Currently argus has a pretty dynamic memory use
>> >> >profile, we allocate and deallocate memory as needed
>> >> >based on concurrent flow tracking.  Looks like we're
>> >> >doing about 400 bytes per flow, which is not bad.
>> >> >
>> >> >   So what I'm seeing is that this machine will get as
>> >> >many as 400,000 flows, which will take up about 150-200M
>> >> >of memory, but when it comes down to its normal 75-100K
>> >> >flows, ps still states that its using 200M of memory.
>> >> >Size and RSS never go down.
>> >> >
>> >> >So whats up?
>> >> >
>> >> >Carter
>> >> >
>> >> >
>> >> >Carter Bullard
>> >> >QoSient, LLC
>> >> >300 E. 56th Street, Suite 18K
>> >> >New York, New York  10022
>> >> >
>> >> >carter at qosient.com
>> >> >Phone +1 212 588-9133
>> >> >Fax   +1 212 588-9134
>> >>
>> >> _/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/
>> >>
>> >> Chris Newton, Systems Analyst
>> >> Computing Services, University of New Brunswick
>> >> newton at unb.ca 506-447-3212(voice) 506-453-3590(fax)
>> >>
>> >>
>>
>> _/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/
>>
>> Chris Newton, Systems Analyst
>> Computing Services, University of New Brunswick
>> newton at unb.ca 506-447-3212(voice) 506-453-3590(fax)
>>
>>

_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/

Chris Newton, Systems Analyst
Computing Services, University of New Brunswick
newton at unb.ca 506-447-3212(voice) 506-453-3590(fax)