Couple things...

[Carter Bullard]

Hey Craig,
You've got some work to do to understand the metrics you're using.
Since you haven't shared with use any of the argus commands you're using to insert the data,
I can't comment on the quality of your system.  Why don't you do the experiment to see if 
0KB generates the same results as 100MB ???

Carter 

On Sep 19, 2013, at 12:14 PM, Craig Merchant <cmerchant at responsys.com> wrote:

> How does racluster impact that?  I would think that if in a five minute interval I did an ls of the root file system (a consumer behavior) and then SCP’d a 10KB file, that the ABR would still average out to being a consumer.  But if I SCP’d a hundred MB file, it would definitely make me a producer.
>  
> I just tried using bytes*ABR and see what the anomaly detection system said and it didn’t flag my test case (or anything else for that matter).  It’s probably easier for it to handle values that stay within a known range…
>  
> Since I’m presenting at the Splunk conference in a couple weeks, I’m doubtful my company would spare me to go to FloCon…  sadly.
> 
> C
>  
> From: Carter Bullard [mailto:carter at qosient.com] 
> Sent: Thursday, September 19, 2013 7:24 AM
> To: Craig Merchant
> Cc: Argus (argus-info at lists.andrew.cmu.edu)
> Subject: Re: [ARGUS] Couple things...
>  
> Hey Craig,
> Oh, one thing that I forgot to point out. You should get the same results if you SSH'd a 10KB file.
> The ABR metric is specifically designed to be independent of volume, which is critical when developing
> an exfiltration detection system.  Threshold based systems all fail, as the intruder can just keep
> the individual transfers below the trigger threshold.
>  
> Carter
>  
>  
>  
> On Sep 18, 2013, at 3:33 PM, Craig Merchant <cmerchant at responsys.com> wrote:
> 
> 
> Hey, Carter…
>  
> Just wanted to start off by saying the latest release has been awesome.  Everything has been super stable and all our label issues have been solved.  Thanks for all your hard work!
>  
> We’ve also been experimenting with the ABR and an anomaly detection tool for Splunk from a company called Prelert.  I configured one of my servers to SSH into a remote host and run a script that does a recursive ls and then sleeps for a couple minutes.  I then periodically SCP’d 100 MB of files up to that remote host.  With around 2.5 million events over a couple days for that subnet, I got this:
>  
> <image001.png>
>  
> Each one of those spikes when I drilled down represented one of the large SCP uploads.  So, the signal-to-noise ratio for the ABR seems really good (at least for my simplistic test example).
>  
> Couple questions for you… 
>  
> We’re using rastream and racluster to aggregate our flows every five minutes.  My understanding and from what I’ve seen in the data is that if a flow lasts for 15 minutes, I’ll only get a single record when the flow closes.  Am I incorrect?  When I look at the aggregated argus records for the test flow in the example above, where the SSH session stayed open for days, I’m not seeing one event every five minutes.  So, if somebody asked me “how much bandwidth are we using right now?”, it would be tricky to divide bytes by duration and project that backwards in time.
>  
> Is there any way to make racluster write a summary record for flows that are longer than the aggregation interval that shows the summary statistics for that flow only in that window?
>  
> Also…  I’ve seen you use “-M RMON” in several examples on the mailing list.  What is the value of using RMON stats for analysis?
>  
> Thanks.
> 
> Craig

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130919/d0bff237/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130919/d0bff237/attachment.bin>