Question regarding on how to use flows in real time

Tue Jun 2 15:59:40 EDT 2015

Hey Sebas,
Actually many of your assumptions about argus are not true, but I don’t want to
start this thread on a negative note.   Regardless of why you want state based
flow records, that is something we can and should provide.

Argus however,should not be the focus of the discussion.   At one time, oh 25
years ago, argus did generate data based on connection state, but we took that
out pretty quickly, because it generates a few nasty vulnerabilities in the sensor
that are hard to overcome and it caused poor streaming analytic performance,
as flow records were coming out of the sensor out of order.

The strategy that I would recommend is to build or use a ra* client program that
assembles state based flows from the argus flow status records.  State based
reassembly will take a little development, but I would recommend that we
implement it in racluster.1, and then either make racluster.1 something that
can be run as a daemon, or integrate this type of functionality into radium.1.

I presume that you are interested in TCP state driven flow reporting.
Are there other states you want to use to flush flow records ???

Carter

> On Jun 2, 2015, at 8:56 AM, el draco <eldraco at gmail.com> wrote:
> 
> Hi Carter, thanks for your answer, your work and time on this.
> 
> About why we are doing this: Our research is not using anomaly detection for this (which I also discourage here), but the detection of known malware behavioral models in the network. 
> Our approach is like this:
> 1- Capture and label real malware and normal traffic. (For learning and verification)
> 2- Create the behavioral models of know malware traffic and actions (More details in [1] if you are interested)
> 3- Go to an unknown network
> 4- Search for similar behavior (there is a detection algorithm in here that can generalize)
> 
> About how to stop stuff after it finished: We don't want to stop the infection before it happens, we leave that to Antivirus products. Since we know that they are not good for detecting new stuff we want to detect the malicious traffic _while_ it is happening. This is not the best solution, but is better than not detecting it at all. Since malware and C&C can be active for hours, days and even months, we try to detect it ASAP by looking to the network flows. 
> So we don't care that one flow has finished, we care about the big picture of all the flows and all the connections coming from that host. In this way, by analyzing the _finished_ flows we are able to detect the malware traffic after a short period of time that depends on the malware.
> 
> ---
> 
> Regarding the _online_ feature I was asking, can you tell me something? Is argus actually tracking when a flow is finished according to its protocol, such as TCP? Is it tracking the changes in states until the flow is ended? I have the impression that it is not right? At least in my tests.
> 
> Please correct me if I'm wrong here:
> 1- Argus always has a report timeout (ARGUS_FLOW_STATUS_INTERVAL). 
> 2- Flows are _only_ printed when the report timeout is met.
> 3- Flows are never printed when they are finished (finished according to its protocol) because it is not tracking the changes in states.
> 4- The values of the fields inside the flow depends on the timeout used (which means that if the flow was finished, all the info is there, if not, part of the info is there).
> 
> What I want to know is:
> - To print a flow as soon as it finished we need to keep track of the changes in states, which can be tricky and expensive. right?
> 
> My goal would be to differentiate between the _really_ finished flows and the argus reported flows.
> 
> Thanks
> sebas
> 
> [1] https://www.researchgate.net/publication/271204142_Identifying_Modeling_and_Detecting_Botnet_Behaviors_in_the_Network <https://www.researchgate.net/publication/271204142_Identifying_Modeling_and_Detecting_Botnet_Behaviors_in_the_Network>
> 
> 
> On Tue, May 26, 2015 at 11:12 PM, Carter Bullard <carter at qosient.com <mailto:carter at qosient.com>> wrote:
> Hey Sebas,
> This is a job for racluster.1, but you are asking for something that I would suggest is not practical and I'll advised.  You can not build a real time sensor with state based reporting.
> And if you are interested in behavioral based anomaly detection and prevention, state based reporting is the last thing that you want to use for many many many reasons.
> 
> How can you stop something if you are only realizing that it exists, after it is finished ???
> My recommendation is to lower the status interval to 1sec and deal with intermediate records.
> Or use racluster to flush on specific state, which I don't recommend.
> Carter
> 
> 
> 
> On May 26, 2015, at 4:48 PM, el draco <eldraco at gmail.com <mailto:eldraco at gmail.com>> wrote:
> 
>> Hi list. We come across some special need with argus and I would like
>> to have your opinion on it.
>> I will try to make my self clear.
>> 
>> The need:
>> In our project in the Univ we would like to have something similar to
>> 'real time' argus flows. The idea is to have a running argus
>> generating flows from a network and have them reported to us as soon
>> as possible. However, we _strongly_ depend on the _real_ time
>> differences between flows, so we don't want to use the "reporting
>> interval" of argus. We need the real flows being reported 'naturally'.
>> (More info on stratosphereips.org <http://stratosphereips.org/>)
>> 
>> Our first attempted solution was to use an
>> ARGUS_FLOW_STATUS_INTERVAL=3600. This is based on the idea that since
>> the protocols in argus are timeouting at 60,30 and 5 secs, then the
>> connections would timeout before argus reports them. Which in fact
>> turns out to be like having no argus reporting at all. This was good
>> for some time. However, despite the fact that using a reporting
>> interval of 3600 s stops argus from reporting intermediate flows,
>> argus do not output the flows that are already finished in real time.
>> 
>> 
>> Example:
>> Let me show an example that is attached in this email. I generated it
>> with the command :
>> for i in `seq 1 10`; do wget --limit-rate 1000 -N www.odd.com <http://www.odd.com/> $i; sleep 60; done
>> 
>> That is, 10 downloads separated by 1 minute. Each download last few
>> seconds and is completely done and terminated.
>> 
>> In our ideal case I would like to have argus reporting each flow as
>> soon as it finishes. However, using a reporting time of 5 seconds
>> gives us intermediate flows:
>> 
>> (notice that I'm using -f to simulate a never ending pcap file)
>> argus -f -S 5 -r 3/3.pcap -w - |ra -Z b -n -F /etc/ra.conf -r -
>> StartTime,Dur,Proto,SrcAddr,Sport,Dir,DstAddr,Dport,State,sTos,dTos,TotPkts,TotBytes
>> 2015/05/25 15:11:49.627257,4.717779,tcp,x.x.x.x,57384,
>> ->,107.22.231.241,80,SPA_SPA,0,0,27,7536
>> 2015/05/25 15:11:55.056756,2.979670,tcp,x.x.x.x,57384,
>> ->,107.22.231.241,80,FA_FPA,0,0,16,3823
>> 2015/05/25 15:12:57.933572,4.620449,tcp,x.x.x.x,57453,
>> ->,107.22.231.241,80,SPA_SPA,0,0,37,7974
>> 2015/05/25 15:13:02.946189,3.511176,tcp,x.x.x.x,57453,
>> ->,107.22.231.241,80,FA_FPA,0,0,23,4844
>> 2015/05/25 15:14:06.354719,4.507918,tcp,x.x.x.x,57525,
>> ->,107.22.231.241,80,SPA_SPA,0,0,37,7974
>> (and 14 more identical flows)
>> 
>> And using a reporting time of 3600 seconds give us nothing until 1 hour passes:
>> argus -f -S 3600 -r 3/3.pcap -w - |ra -Z b -n -F /etc/ra.conf -r -
>> 
>> 
>> What we need is argus reporting each flow as soon as it finishes
>> without adding intermediate _reported_ flows.
>> 
>> 
>> 
>> Do you think this is possible? Or there are important drawbacks on
>> this reasoning?
>> (Maybe there is a way of differentiating the _reported_ flows from the
>> completed ones?)
>> 
>> Thanks a lot!
>> Sebas
>> 
>> 
>> 
>> -- 
>> https://pgp.mit.edu/pks/lookup?op=get&search=0x9D9A358CA10F1601 <https://pgp.mit.edu/pks/lookup?op=get&search=0x9D9A358CA10F1601>
>> <3.pcap>
> 
> 
> 
> -- 
> https://pgp.mit.edu/pks/lookup?op=get&search=0x9D9A358CA10F1601 <https://pgp.mit.edu/pks/lookup?op=get&search=0x9D9A358CA10F1601>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20150602/834110c5/attachment.html>