Question regarding on how to use flows in real time

Fri Jun 12 18:37:15 EDT 2015

Hey Sebas,
I think it's more of a discovery process than a documentation situation, because it is a complex area of computing.

You should not be running argus with long status times.  You should be running argus with reasonable status times, say 5 seconds, and then aggregating the output of argus to suit your needs. 

So if you looked at argus output when the status times are reasonable, say 5 seconds, you will quickly see that roughly 95% of all IP flows are completed in around 2.5 second secs.  DNS is sub 1 second, usually, HTTP transactions are generally under 3 seconds, usually, with the majority under 1.45 seconds.  This is why we recommend a 5 second status interval.  If you collect a days worth of flow records, and then use racluster.1 to merge all the flow records, you will see that racluster.1 does what you want, so it seems that you are looking for a streaming racluster.1 like program.

The problem with holding flows and not reporting on them until they close, is that many long lived flows don’t ever close, by definition, and some, when they do close, don’t close properly.  Then of course there are the flows that have activity after they close which is somewhat frustrating from a flow activity sensors perspective.  Holding a flow cache is expensive, computationally, so what is the point of tracking flows that never terminate ????

Of course, long lived flows can be stateful, such as TCP or stateless, such as IPSec tunnels.  There is no completion indications for an IPSec tunnel, they can last for months/years at a time, and then they just timeout.  Some TCP connections don’t close, sometimes because the end system faults, powers off or reboots, or the network eats the closing packets, as is seen in some poor stateful NAT’ing equipment.  So what do you want to do about flows that don't close, in your proposal ???  Hold onto their caches forever ????  That gets really expensive really fast.
And of course your algorithm is subject to manipulation by intruders that know that your sensing is state based.  They will start what are called half-open connections, and your algorithm will just run out of memory, after a few billion packets.

If you would like for me to add state conditions for flushing records in racluster.1, I’ll put that in argus-clients-3.0.9.

Carter

On Jun 9, 2015, at 10:01 AM, el draco <eldraco at gmail.com <mailto:eldraco at gmail.com>> wrote:

> Hi Carter and List.
> 
> Thanks for considering this.
> Ok, so I agree that adding this to racluster can be great in an 'daemon mode'. 
> The idea would be to 'report the flows as soon as they can be identified as finished',  right? 
> 
> So, yes, TCP is the main candidate.
> Not sure with ICMP, most of the 'ping' flows have only two packets, so those are a good candidates too. Most of other ICMP flows only have 1 packet right? 
> UDP may be more difficult to know when it finished so they can be reported when the ARGUS_FLOW_STATUS_INTERVAL expires. 
> I don't care about the order, we can sort them later
> 
> Sadly, I can not help with the development, but count with me for any other stuff, including writing the man about it, testing, etc. 
> 
> And about my misconceptions about argus internals, is there a quicker place to learn about them that is not reading the argus source code?
> Thanks for your time on this
> Sebas
> 
> 
> On Tue, Jun 2, 2015 at 9:59 PM, Carter Bullard <carter at qosient.com <mailto:carter at qosient.com>> wrote:
> Hey Sebas,
> Actually many of your assumptions about argus are not true, but I don’t want to
> start this thread on a negative note.   Regardless of why you want state based
> flow records, that is something we can and should provide.
> 
> Argus however,should not be the focus of the discussion.   At one time, oh 25
> years ago, argus did generate data based on connection state, but we took that
> out pretty quickly, because it generates a few nasty vulnerabilities in the sensor
> that are hard to overcome and it caused poor streaming analytic performance,
> as flow records were coming out of the sensor out of order.
> 
> The strategy that I would recommend is to build or use a ra* client program that
> assembles state based flows from the argus flow status records.  State based
> reassembly will take a little development, but I would recommend that we
> implement it in racluster.1, and then either make racluster.1 something that
> can be run as a daemon, or integrate this type of functionality into radium.1.
> 
> I presume that you are interested in TCP state driven flow reporting.
> Are there other states you want to use to flush flow records ???
> 
> Carter
> 
>  
> 
>> On Jun 2, 2015, at 8:56 AM, el draco <eldraco at gmail.com <mailto:eldraco at gmail.com>> wrote:
>> 
>> Hi Carter, thanks for your answer, your work and time on this.
>> 
>> About why we are doing this: Our research is not using anomaly detection for this (which I also discourage here), but the detection of known malware behavioral models in the network. 
>> Our approach is like this:
>> 1- Capture and label real malware and normal traffic. (For learning and verification)
>> 2- Create the behavioral models of know malware traffic and actions (More details in [1] if you are interested)
>> 3- Go to an unknown network
>> 4- Search for similar behavior (there is a detection algorithm in here that can generalize)
>> 
>> About how to stop stuff after it finished: We don't want to stop the infection before it happens, we leave that to Antivirus products. Since we know that they are not good for detecting new stuff we want to detect the malicious traffic _while_ it is happening. This is not the best solution, but is better than not detecting it at all. Since malware and C&C can be active for hours, days and even months, we try to detect it ASAP by looking to the network flows. 
>> So we don't care that one flow has finished, we care about the big picture of all the flows and all the connections coming from that host. In this way, by analyzing the _finished_ flows we are able to detect the malware traffic after a short period of time that depends on the malware.
>> 
>> ---
>> 
>> Regarding the _online_ feature I was asking, can you tell me something? Is argus actually tracking when a flow is finished according to its protocol, such as TCP? Is it tracking the changes in states until the flow is ended? I have the impression that it is not right? At least in my tests.
>> 
>> Please correct me if I'm wrong here:
>> 1- Argus always has a report timeout (ARGUS_FLOW_STATUS_INTERVAL). 
>> 2- Flows are _only_ printed when the report timeout is met.
>> 3- Flows are never printed when they are finished (finished according to its protocol) because it is not tracking the changes in states.
>> 4- The values of the fields inside the flow depends on the timeout used (which means that if the flow was finished, all the info is there, if not, part of the info is there).
>> 
>> What I want to know is:
>> - To print a flow as soon as it finished we need to keep track of the changes in states, which can be tricky and expensive. right?
>> 
>> My goal would be to differentiate between the _really_ finished flows and the argus reported flows.
>> 
>> Thanks
>> sebas
>> 
>> [1] https://www.researchgate.net/publication/271204142_Identifying_Modeling_and_Detecting_Botnet_Behaviors_in_the_Network <https://www.researchgate.net/publication/271204142_Identifying_Modeling_and_Detecting_Botnet_Behaviors_in_the_Network>
>> 
>> 
>> On Tue, May 26, 2015 at 11:12 PM, Carter Bullard <carter at qosient.com <mailto:carter at qosient.com>> wrote:
>> Hey Sebas,
>> This is a job for racluster.1, but you are asking for something that I would suggest is not practical and I'll advised.  You can not build a real time sensor with state based reporting.
>> And if you are interested in behavioral based anomaly detection and prevention, state based reporting is the last thing that you want to use for many many many reasons.
>> 
>> How can you stop something if you are only realizing that it exists, after it is finished ???
>> My recommendation is to lower the status interval to 1sec and deal with intermediate records.
>> Or use racluster to flush on specific state, which I don't recommend.
>> Carter
>> 
>> 
>> 
>> On May 26, 2015, at 4:48 PM, el draco <eldraco at gmail.com <mailto:eldraco at gmail.com>> wrote:
>> 
>>> Hi list. We come across some special need with argus and I would like
>>> to have your opinion on it.
>>> I will try to make my self clear.
>>> 
>>> The need:
>>> In our project in the Univ we would like to have something similar to
>>> 'real time' argus flows. The idea is to have a running argus
>>> generating flows from a network and have them reported to us as soon
>>> as possible. However, we _strongly_ depend on the _real_ time
>>> differences between flows, so we don't want to use the "reporting
>>> interval" of argus. We need the real flows being reported 'naturally'.
>>> (More info on stratosphereips.org <http://stratosphereips.org/>)
>>> 
>>> Our first attempted solution was to use an
>>> ARGUS_FLOW_STATUS_INTERVAL=3600. This is based on the idea that since
>>> the protocols in argus are timeouting at 60,30 and 5 secs, then the
>>> connections would timeout before argus reports them. Which in fact
>>> turns out to be like having no argus reporting at all. This was good
>>> for some time. However, despite the fact that using a reporting
>>> interval of 3600 s stops argus from reporting intermediate flows,
>>> argus do not output the flows that are already finished in real time.
>>> 
>>> 
>>> Example:
>>> Let me show an example that is attached in this email. I generated it
>>> with the command :
>>> for i in `seq 1 10`; do wget --limit-rate 1000 -N www.odd.com <http://www.odd.com/> $i; sleep 60; done
>>> 
>>> That is, 10 downloads separated by 1 minute. Each download last few
>>> seconds and is completely done and terminated.
>>> 
>>> In our ideal case I would like to have argus reporting each flow as
>>> soon as it finishes. However, using a reporting time of 5 seconds
>>> gives us intermediate flows:
>>> 
>>> (notice that I'm using -f to simulate a never ending pcap file)
>>> argus -f -S 5 -r 3/3.pcap -w - |ra -Z b -n -F /etc/ra.conf -r -
>>> StartTime,Dur,Proto,SrcAddr,Sport,Dir,DstAddr,Dport,State,sTos,dTos,TotPkts,TotBytes
>>> 2015/05/25 15:11:49.627257,4.717779,tcp,x.x.x.x,57384,
>>> ->,107.22.231.241,80,SPA_SPA,0,0,27,7536
>>> 2015/05/25 15:11:55.056756,2.979670,tcp,x.x.x.x,57384,
>>> ->,107.22.231.241,80,FA_FPA,0,0,16,3823
>>> 2015/05/25 15:12:57.933572,4.620449,tcp,x.x.x.x,57453,
>>> ->,107.22.231.241,80,SPA_SPA,0,0,37,7974
>>> 2015/05/25 15:13:02.946189,3.511176,tcp,x.x.x.x,57453,
>>> ->,107.22.231.241,80,FA_FPA,0,0,23,4844
>>> 2015/05/25 15:14:06.354719,4.507918,tcp,x.x.x.x,57525,
>>> ->,107.22.231.241,80,SPA_SPA,0,0,37,7974
>>> (and 14 more identical flows)
>>> 
>>> And using a reporting time of 3600 seconds give us nothing until 1 hour passes:
>>> argus -f -S 3600 -r 3/3.pcap -w - |ra -Z b -n -F /etc/ra.conf -r -
>>> 
>>> 
>>> What we need is argus reporting each flow as soon as it finishes
>>> without adding intermediate _reported_ flows.
>>> 
>>> 
>>> 
>>> Do you think this is possible? Or there are important drawbacks on
>>> this reasoning?
>>> (Maybe there is a way of differentiating the _reported_ flows from the
>>> completed ones?)
>>> 
>>> Thanks a lot!
>>> Sebas
>>> 
>>> 
>>> 
>>> -- 
>>> https://pgp.mit.edu/pks/lookup?op=get&search=0x9D9A358CA10F1601 <https://pgp.mit.edu/pks/lookup?op=get&search=0x9D9A358CA10F1601>
>>> <3.pcap>
>> 
>> 
>> 
>> -- 
>> https://pgp.mit.edu/pks/lookup?op=get&search=0x9D9A358CA10F1601 <https://pgp.mit.edu/pks/lookup?op=get&search=0x9D9A358CA10F1601>
> 
> 
> 
> -- 
> https://pgp.mit.edu/pks/lookup?op=get&search=0x9D9A358CA10F1601 <https://pgp.mit.edu/pks/lookup?op=get&search=0x9D9A358CA10F1601>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20150612/01d7631b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20150612/01d7631b/attachment.bin>