Direction and IP/TCP timeout settings

Fri Jul 12 18:48:59 EDT 2013

I’ll send you some of the packet captures offline…

Just for the record, we’re running pf_ring 5.5.3.  We have a couple of Silicom 10g fiber cards.  We’re using the DNA/libzero drivers.  And thanks to the code that Chris Wakelin shared here, we use DNA to split the flows into 28 queues that Snort listens on and then makes a full copy of all flows on a queue that argus listens on (Thanks, Chris!).

I took a look at the capture files and what wireshark saw on the Windows box that initiated the SCP session is pretty different than what tcpdump saw on the server running argus.  A lot of duplicate ACK packets and a few other things I’m not familiar with.  But the SYN and SYNACK packets were there on both sides.

I Googled the RA_LOCAL config and found a couple of messages you sent out in December.  What I don’t see is how to specify locality by protocol and port.  I’m not clear on how ra clients will deal with two servers that are both defined as local.  Make the first match local?  I’d like to be able to say things like, “TCP 3389 is always the destination” or “TCP 1433 10.10.10.10 is the destination”.

Is that functionality available in the current RA_LOCAL support?

Thx.

Craig

From: Carter Bullard [mailto:carter at qosient.com]
Sent: Friday, July 12, 2013 1:49 PM
To: Craig Merchant
Cc: Argus (argus-info at lists.andrew.cmu.edu)
Subject: Re: [ARGUS] Direction and IP/TCP timeout settings

Hey Craig,
A bunch of things here.  The pf_ring could be an issue...but we need to see the issue first.  So racluster() won't generate ? In the direction field, it has the chance to actually reduce the number.

Argus and racluster will only generate records when there is activity, so 1 packet in a flow, 20 hours later another packet, argus generates 2 records,  racluster, in default mode (no idle timeouts) will generate 1 record.

So we have direction based on a list, RA_LOCAL that the clients can use to determine direction.  Can that work for your needs?

Carter

On Jul 12, 2013, at 4:24 PM, Craig Merchant <cmerchant at responsys.com<mailto:cmerchant at responsys.com>> wrote:
I’ll setup wireshark and do some testing…  I did check the stats for the interfaces that argus listens on.  In the data center that is running around 3 Gbps today, there was 0% packet loss.  In the other data center that was doing 8.5 Gbps today, it had 0.3% packet loss.

I should mention that we’re using racluster to aggregate our flows by saddr, daddr, dport, and  protocol every 5 minutes if that could be having any impact…  We’re also using the DNA/libzero drivers from pf_ring if that could be having any impact.

I searched over my argus data for flows with a direction of “?>” and summed the duration by saddr, daddr, dport, and protocol and looked at the flows with the highest durations over the last 20 hours.  For the top connection, racluster produced 75 records with a total duration of 3234 seconds.

I’m not enough of an expert here, but if a flow remains open for 20 hours (even if it isn’t passing data), shouldn’t I see at least one record from racluster for each 5 minute interval?

How difficult would it be to add a feature to argus that would let us “hard code” directional settings?  I could envision feeding argus a NMAP or Nessus scan or a label file with host/protocol/port information that would tell argus to always consider that to be the destination if argus isn’t sure about the direction.

I’ll let you know the testing goes.  Thanks again for your consistently quick replies!

Craig

From: Carter Bullard [mailto:carter at qosient.com]
Sent: Friday, July 12, 2013 10:54 AM
To: Craig Merchant
Subject: Re: [ARGUS] Direction and IP/TCP timeout settings

Well, its all about seeing the SYN or SYNACK.

So are these flows all so long lived that they all started before 18 hours?
If not, are you seeing the SYN and SYNACK and argus is forgetting about it?
startup a packet capture, and then send some test TCP connections.  Make
sure they last for a while.  Do an ssh, and let there be an 120 second idle
time and then start typing again, and see if argus is screwing up.

The packet capture should have all the packets, and argus should generate
the same problems just reading that packet file, so hopefully we'll capture
the error.

Carter

On Jul 12, 2013, at 1:35 PM, Craig Merchant <cmerchant at responsys.com<mailto:cmerchant at responsys.com>> wrote:

I’ve been running Argus for about 18 hours now with a two hour timeout setting and there hasn’t been any change in the number of flows that it is unsure of the direction…

Let me know if there is anything I can do to help test this…

C

From: Carter Bullard [mailto:carter at qosient.com<http://qosient.com>]
Sent: Friday, July 12, 2013 6:37 AM
To: Craig Merchant
Cc: Argus (argus-info at lists.andrew.cmu.edu<mailto:argus-info at lists.andrew.cmu.edu>)
Subject: Re: [ARGUS] Direction and IP/TCP timeout settings

Hmmmm, do the new timeouts change the direction problem?
That will be the real test, if the memory issues aren't showing themselves,
the cool, as long as your traffic looks better.

If not, I'll take a look.  Never know where things break down.
In some cases, we'll try to make the direction indicator match the traffic,
with the central character indicating the confidence.  So, when there is
a " ? ", the < or > should change to indicate direction of traffic, since
the assignment of flow direction isn't " on ".

Carter

On Jul 11, 2013, at 7:28 PM, Craig Merchant <cmerchant at responsys.com<mailto:cmerchant at responsys.com>> wrote:

Hey, Carter…

We’re finding that for about 70% of our flows, Argus can’t figure out the direction.  From previous posts, it would seem that the 60 second TCP session timeout is too short.  If I understand correctly, a flow longer than 60 seconds will have its session timeout in the cache and then argus can’t really determine what the direction is.

The argus.conf file warns of the hit on memory if those settings are adjusted from the defaults.  I’ve been steadily increasing the TCP and IP timeout values and watching to see if memory consumption jumps up dramatically or if we’re seeing less events where the direction is uncertain.

I’ve gone as high up as two hour session timeout.  We do something like 2.5-8 Gbps 24 hours a day, so I would expect to see a huge increase in Argus memory consumption when increase the timeout value.  The machine has like 64 GB of memory and top says argus is only using .2%.

The settings look like:

ARGUS_IP_TIMEOUT=3600
ARGUS_TCP_TIMEOUT=7200
#ARGUS_ICMP_TIMEOUT=5
#ARGUS_IGMP_TIMEOUT=30
#ARGUS_FRAG_TIMEOUT=5
#ARGUS_ARP_TIMEOUT=5
#ARGUS_OTHER_TIMEOUT=30

Am I doing something wrong here?  Is there some other setting I need to enable to increase that timeout value?

Also, what’s the difference between a direction value of ?> vs <?>?

Thanks!

Craig

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130712/1e988523/attachment.html>