Argus-info Digest, Vol 96, Issue 4
CS Lee
geek00l at gmail.com
Fri Aug 2 02:35:57 EDT 2013
hi Carter,
I still have quite handful of old notes that writing how each of argus
field is calculated in more layman form so people can understand them
easily, maybe I can push them to the wiki so people can refer to them, I
have seen you answering same question over and over again here and that
might make you spending less time for coding and bug fixing.
I think wiki is good place to share user experience, guess we all should
make full use of it in term of documentation. Before anyone forget, we have
wiki here -
http://nsmwiki.org/Argus
Cheers ;)
On Fri, Aug 2, 2013 at 7:02 AM, <argus-info-request at lists.andrew.cmu.edu>wrote:
> Send Argus-info mailing list submissions to
> argus-info at lists.andrew.cmu.edu
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://lists.andrew.cmu.edu/mailman/listinfo/argus-info
> or, via email, send a message with subject or body 'help' to
> argus-info-request at lists.andrew.cmu.edu
>
> You can reach the person managing the list at
> argus-info-owner at lists.andrew.cmu.edu
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Argus-info digest..."
>
>
> Today's Topics:
>
> 1. rabins() does not "ramp up" counts !!! (Carter Bullard)
> 2. Couple things... (Craig Merchant)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 1 Aug 2013 17:37:26 -0400
> From: Carter Bullard <carter at qosient.com>
> Subject: [ARGUS] rabins() does not "ramp up" counts !!!
> To: Matt Brown <matthewbrown at gmail.com>
> Cc: "argus-info at lists.andrew.cmu.edu"
> <argus-info at lists.andrew.cmu.edu>
> Message-ID: <1F71F1AC-E415-4C91-B3E6-8A05C871D4E2 at qosient.com>
> Content-Type: text/plain; charset="windows-1252"
>
> Hey Matt,
> Shorter emails please. These tombs are very difficult for others
> to track, and while you personally maybe getting something out of
> this, the list is intended for the group as a whole.
>
> So, make some changes, give it a test drive, try to figure it out.
> Change your status interval, and look at how argus behaves.
> Share the experience, but keep the email short.
>
> If you think that argus needs better documentation, make a
> suggestion, write a paragraph, submit a new man page, contribute
> to the wiki. Just send your comments to this list. Its
> gets archived, etc? but do contribute.
>
> If you have a question, ask a specific question. Don't write 25
> pages, and then toss to me that I need to document how all the
> fields are calculated. Most of that is already documented, we
> do have tutorials on this stuff, for Pete's sake.
>
> And, please, just for Carter's sake, stop declaring that the
> technology sucks, is inaccurate, or broken, until you actually
> know that it is. I will never say that argus is great software,
> but it doesn't suck. There maybe bugs, but argus and its clients
> are not inaccurate.
>
> And, please, try better subject lines. rabins() is not ramping
> up counts. Doesn't do it. but there are 7 messages in this
> thread that asks why rabins ramps up counts.
>
> How am I suppose to respond to a subject line like that ?????
>
> With every response you send, you have created and are
> perpetuating the myth that rabins() is ramping up counts.
>
> That isn't true.
>
> Carter
>
>
> On Aug 1, 2013, at 4:39 PM, Matt Brown <matthewbrown at gmail.com> wrote:
>
> > Hey Carter,
> >
> > I appreciate the thorough responses as always. I was five steps
> > behind you, but I now understand.
> >
> > Is it worth adding documentation (in the man) on how the "calculated
> > fields" are calculated? I think that would have avoided some of my
> > confusion.
> >
> > I was also confused by how the specific flow I was looking at is a
> > long lived flow that encompasses the bin, and, in fact, the entire
> > time span contained in the source data file; meaning the duration is
> > always 5 seconds regardless if I'm using `-M hard` or not.
> >
> >
> > Here is a summary of what you described (and correct me if I'm wrong):
> >
> > `-M hard`:
> > - Modifies a flow's ltime and stime so that their duration is set to
> > that of the bin timespan.
> >
> > `-M nomodify`:
> > - doesn't modify anything about the records. Specifically, doesn't
> > modify time-wise fields.
> > Q: On the NSM wiki, it's stated that `-M nomodify` is used when you
> > wish to have a flow only appear in a single bin. Is this correct?
> > Would it appear in the single bin at its stime?
> >
> >
> >
> > Given the following two points:
> > 1) rate=pkts/(ltime - stime)
> > 2) dur=ltime-stime
> > ...the reason why `-M hard` affects "ramps up" and "ramps down" `rate`:
> > - Since the `dur` of each record is forced to be equal to the bin
> timespan,
> > -- in this case, the `pkts` sent within the first and last designated
> > bin timespan are only for a small section of the time within the bin
> > timespan (`dur`)...
> > --- therefore, The `rate` calculation result is an abnormally small
> > count, because the `dur` is the not the length of the real `dur` of
> > the shorter lived flow.
> >
> > Q: Wouldn't this indicate that you can not use a field like `rate`
> > (what I called an aggregate field, but what you called a "calculated
> > field") with `-M hard` and expect the first report to be reliable?
> > Q: How can I effectively use `-M hard`?
> >
> > 1) `-B` must always be larger than ARGUS_FLOW_STATUS_INTERVAL since
> > `-B` is a buffer time that waits for all data to arrive for a bin.
> > 2) `-B` should be used to set rabins() to hang out while all the
> > status records arrive for a bin (from the argus cache ?). The status
> > records are delivered at the interval set with
> > argus.conf:ARGUS_FLOW_STATUS_INTERVAL.
> >
> > Q: I think, by what you've stated, when used with a `-B` timespan
> > value that is less than ARGUS_FLOW_STATUS_INTERVAL, rabins() would
> > report nO data, but it clearly reports data. Are you try to say that
> > the data is unpredictable and therefore can't be reliable upon?
> >
> >
> > Clearly, my problem was a mix of misusing `-M hard` and `-B`.
> > Q: I will be lowering my ARGUS_FLOW_STATUS_INTERVAL from 60 to 5 (as
> > reportedly acceptable as the default). What is the cost of this?
> >
> >
> > Thanks again,
> >
> > Matt
> >
> >
> >
> > On Jul 31, 2013, at 11:23 PM, Carter Bullard <carter at qosient.com> wrote:
> >
> >> Matt,
> >>
> >> The reason you are having problems is the " -B 5s " option.
> >>
> >> Don't use it when reading data from a file. Its not intended
> >> for this use, and while it won't hurt anything, it represents
> >> a lack of understanding of the option.
> >>
> >> When you use it with a live argus data source, it must be
> >> greater than the ARGUS_FLOW_STATUS_INTERVAL to have its effect.
> >>
> >> Lets try to understand what this option is doing. The " -B secs "
> >> option is saying how long you have to hold a bin in memory
> >> in order to guarantee that all the data arrives for that time
> >> bin.
> >>
> >> With these options " -M time 5s -B 5s ", you will process
> >> only 2 bins at a time, the current bin, who's range is
> >> [now - (now + 5s)] and the hold buffer, which is [(now - 5s) - now].
> >>
> >> With an ARGUS_FLOW_STATUS_INTERVAL=60s, you will receive data
> >> whose time range could be [(now - 60s) - now]. rabins(), when
> >> carving up the record into 5 second chunks, will generate, at
> >> most 12 records whose ranges are:
> >> 1. [((now - 60s)+0*5s) - ((now - 60s)+1*5s)]
> >> 2. [((now - 60s)+1*5s) - ((now - 60s)+2*5s)]
> >> 3. [((now - 60s)+2*5s) - ((now - 60s)+3*5s)]
> >> ?
> >> 12. [((now - 60s)+11*5s) - ((now - 60s)+12*5s)]
> >>
> >> With a " -B 5s " option, only the 12th record will have a slot
> >> to aggregate into. So you will be throwing away 11/12 of
> >> all your data.
> >>
> >> Increase your "-B sec" option to
> >> (max(ARGUS_FLOW_STATUS_INTERVAL) + someDelay)
> >>
> >> So I would use at a minimum " -B 65s ".
> >>
> >>
> >> OK, lets try to keep the email short in the future,
> >> if its feasible to do so. One topic at a time?
> >>
> >> Carter
> >>
> >>
> >>
> >> On Jul 30, 2013, at 4:19 PM, Matt Brown <matthewbrown at gmail.com> wrote:
> >>
> >>> Hello all,
> >>>
> >>>
> >>> Does rabins() "ramp up to normal" over N bins?
> >>>
> >>>
> >>> I'd like to start working on calculating moving averages to help
> >>> identify performance outliers (like "spikes" in `loss` or `rate`).
> >>>
> >>> For this purpose, I believe grabbing data from the output of rabins()
> >>> would serve me well.
> >>>
> >>>
> >>> For example, if I take historic argus data and run it through the
> >>> following rabins() invocation, I see some odd things that can only be
> >>> noted as "ramping up":
> >>>
> >>>
> >>> for f in $(ls -m1 ~/working/*) ; do (
> >>> rabins -M hard time 5s -B 5s -r $f -m saddr -s ltime rate - port 5432
> >>> and src host 192.168.10.22
> >>> ) >> ~/aggregated_rate ; done
> >>>
> >>>
> >>> The first few and the last few resulting records per file seem to not
> >>> be reporting correctly.
> >>>
> >>> For example, these dudes at 192.168.10.22 utilize a postgres DB
> >>> replication package called bucardo. During idle time, bucardo sends
> >>> heartbeat info, and appears to be holding at about 47-49 packets per
> >>> second (rate).
> >>>
> >>> However, I am seeing the following in my rabins() resultant data (note
> >>> the precense of field label header == the start of a new rabins() from
> >>> the above for..loop):
> >>>
> >>> 2013-07-25 00:59:25.000000 47.400000
> >>> 2013-07-25 00:59:30.000000 47.400000
> >>> 2013-07-25 00:59:35.000000 48.000000
> >>> 2013-07-25 00:59:40.000000 48.000000
> >>> 2013-07-25 00:59:45.000000 40.600000
> >>> 2013-07-25 00:59:50.000000 21.400000
> >>> 2013-07-25 00:59:55.000000 15.400000
> >>> 2013-07-25 01:00:00.000000 5.000000
> >>> 2013-07-25 01:00:05.000000 0.000000
> >>> LastTime Rate
> >>> 2013-07-25 01:00:05.000000 0.200000
> >>> 2013-07-25 01:00:10.000000 0.600000
> >>> 2013-07-25 01:00:15.000000 0.400000
> >>> 2013-07-25 01:00:35.000000 0.400000
> >>> 2013-07-25 01:00:40.000000 1.000000
> >>> 2013-07-25 01:00:45.000000 6.200000
> >>> 2013-07-25 01:00:50.000000 25.400000
> >>> 2013-07-25 01:00:55.000000 32.400000
> >>> 2013-07-25 01:01:00.000000 41.800000
> >>> 2013-07-25 01:01:05.000000 47.600000
> >>> 2013-07-25 01:01:10.000000 48.600000
> >>>
> >>> [The source files were written with rastream().]
> >>>
> >>>
> >>> It is well worth noting that if I start an rabins() reading from the
> >>> argus() socket with the following invocation, the same sort of thing
> >>> occurs:
> >>> # rabins -M hard time 5s -B 5s -S 127.0.0.1:561 -m saddr -s ltime rate
> >>> - port 5432 and src host 192.168.10.22
> >>> LastTime Rate
> >>> 2013-07-30 15:42:55.000000 1.400000
> >>> 2013-07-30 15:43:00.000000 0.600000
> >>> 2013-07-30 15:43:05.000000 33.800000
> >>> 2013-07-30 15:43:10.000000 47.400000
> >>> 2013-07-30 15:43:15.000000 58.600000
> >>> 2013-07-30 15:43:20.000000 87.600000
> >>> 2013-07-30 15:43:25.000000 96.200000
> >>> 2013-07-30 15:43:30.000000 96.000000
> >>> 2013-07-30 15:43:35.000000 134.200000
> >>> 2013-07-30 15:43:40.000000 137.200000
> >>> 2013-07-30 15:43:45.000000 137.400000
> >>> 2013-07-30 15:43:50.000000 136.600000
> >>> 2013-07-30 15:43:55.000000 139.800000
> >>> 2013-07-30 15:44:00.000000 136.200000 <-- `rate` averages about here
> >>> going forward
> >>>
> >>>
> >>> It's irrelevant which field I utilize, the same instance occurs:
> >>> # rabins -M hard time 5s -B 5s -S 127.0.0.1:561 -m saddr -s ltime load
> >>> - port 5432 and src host 192.168.10.22
> >>> LastTime Load
> >>> 2013-07-30 15:50:15.000000 1461.19*
> >>> 2013-07-30 15:50:20.000000 42524.7*
> >>> 2013-07-30 15:50:25.000000 54329.5*
> >>> 2013-07-30 15:50:30.000000 55244.8*
> >>> 2013-07-30 15:50:35.000000 90164.8*
> >>> 2013-07-30 15:50:40.000000 92539.1*
> >>> 2013-07-30 15:50:45.000000 94827.1*
> >>> 2013-07-30 15:50:50.000000 95292.7*
> >>> 2013-07-30 15:50:55.000000 96286.3*
> >>> 2013-07-30 15:51:00.000000 94857.6*
> >>> 2013-07-30 15:51:05.000000 130699.*
> >>> 2013-07-30 15:51:10.000000 149979.*
> >>> 2013-07-30 15:51:15.000000 149320.*
> >>> [killed]# rabins -M hard time 5s -B 5s -S 127.0.0.1:561 -m saddr -s
> >>> ltime load - port 5432 and src host 192.168.2.22
> >>> LastTime Load
> >>> 2013-07-30 15:52:35.000000 33894.4*
> >>> 2013-07-30 15:52:40.000000 3134.84*
> >>> 2013-07-30 15:52:45.000000 39262.4*
> >>> 2013-07-30 15:52:50.000000 40024.0*
> >>> 2013-07-30 15:52:55.000000 41188.7*
> >>> 2013-07-30 15:53:00.000000 40259.2*
> >>> 2013-07-30 15:53:05.000000 75057.6*
> >>> 2013-07-30 15:53:10.000000 97160.0*
> >>> 2013-07-30 15:53:15.000000 106520.*
> >>> 2013-07-30 15:53:20.000000 138504.*
> >>> 2013-07-30 15:53:25.000000 153835.*
> >>> 2013-07-30 15:53:30.000000 152892.*
> >>> 2013-07-30 15:53:35.000000 154017.* <-- `load` averages here going
> forward
> >>>
> >>> This happens whether or not I perform field aggregation (`-m saddr`).
> >>>
> >>>
> >>> Why is this happening?
> >>>
> >>>
> >>> This seems like it will really screw up calculating moving averages
> >>> (figuring out spikes, etc.) from the rabins() resultant data.
> >>>
> >>>
> >>> Thanks!
> >>>
> >>> Matt
> >>
> >
>
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: smime.p7s
> Type: application/pkcs7-signature
> Size: 6837 bytes
> Desc: not available
> Url :
> https://lists.andrew.cmu.edu/mailman/private/argus-info/attachments/20130801/d31f0c16/attachment-0001.bin
>
> ------------------------------
>
> Message: 2
> Date: Thu, 1 Aug 2013 23:02:10 +0000
> From: Craig Merchant <cmerchant at responsys.com>
> Subject: [ARGUS] Couple things...
> To: "Argus (argus-info at lists.andrew.cmu.edu)"
> <argus-info at lists.andrew.cmu.edu>
> Message-ID:
> <
> E28F7E9AAF2CF04682AACE6129FF4F320FF6F5 at CORPEX14-MBN3.us.responsys.com>
>
> Content-Type: text/plain; charset="us-ascii"
>
> Hey, Carter...
>
> I just wanted to check in and see if you anything else from me on the
> labeling issue or argus crashing when trying to convert a pcap file. Let
> me know...
>
> I'm also having some issues with keystroke detection with the latest
> release. The following command used to work in my testing:
>
> /usr/local/bin/ra -S 10.10.10.10:561 -n -u -c "," -s
> "+0dnstroke,+1snstroke" - host 10.1.1.1 and host 10.1.1.2
>
> I tried both a normal and reverse SSH session between the two hosts and
> neither one registered keyboard strokes of varying speeds and intensity.
>
> All I've done is commented out the defaults in argus.conf:
>
> ARGUS_KEYSTROKE="yes"
> ARGUS_KEYSTROKE_CONF="GPC_MAX=4"
>
> I performed pretty much the same testing a couple months ago and got
> plenty of flows where keystrokes were detected. Please let me know what
> you'd recommend for troubleshooting that.
>
> Thanks.
>
> Craig
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> https://lists.andrew.cmu.edu/mailman/private/argus-info/attachments/20130801/43b8c6ed/attachment.html
>
> ------------------------------
>
> _______________________________________________
> Argus-info mailing list
> Argus-info at lists.andrew.cmu.edu
> https://lists.andrew.cmu.edu/mailman/listinfo/argus-info
>
>
> End of Argus-info Digest, Vol 96, Issue 4
> *****************************************
>
--
Best Regards,
CS Lee<geek00L[at]gmail.com>
http://geek00l.blogspot.com
http://defcraft.net
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130802/38c36bb1/attachment.html>
More information about the argus
mailing list