Couple things...

Wed Aug 7 14:56:29 EDT 2013

Each data center oscillates between 2 Gbps and 8 Gbps most of the time, depending on when our customers launch their marketing campaigns.

Which features do you recommend disabling for testing purposes?  Since we do have some inter-data center communications, we definitely want to take advantage of the flow deduplication features of radium.  Not sure if the self-synch setting impacts that or not.

I’d love to hear your recommendations!

Thanks.

Craig

From: Carter Bullard [mailto:carter at qosient.com]
Sent: Wednesday, August 07, 2013 11:29 AM
To: Craig Merchant
Cc: Argus (argus-info at lists.andrew.cmu.edu)
Subject: Re: [ARGUS] Couple things...

So, I have not had a chance to fix the label problems, but I do see the problem.  Please bug me about this, as I do want to release 3.0.8 as stable soon...and this is important !!!

Sensor performance problems are tough, because there are so many places it can go wrong.  I've seen smart systems that ate every TCP control packet, so this is not a strange problem to have.  I would like to think the gigamon will do a good job.

How fast are you going ???  If we need to, turning off some argus features will improve argus per packet performance...like self synchronization... We use that when there are multiple sensors and we want to correlate traffic between them.

Carter

On Aug 7, 2013, at 2:12 PM, Craig Merchant <cmerchant at responsys.com<mailto:cmerchant at responsys.com>> wrote:
Thanks to you and David for all your help!

I converted the tcpdump file and sucked it into Splunk.  Argus couldn’t figure out the direction of about half of the flows.  So, it would appear that whatever is happening to our SYN/SYNACK packets, it doesn’t have anything to do with Argus.  I’ve tried connecting to each of our sensors directly and the higher utilized sensor has directional issues about 35-45% of the time.  The less utilized sensor has issues 20-35% of the time.

Ever since you made the nanosleep setting more aggressive, I haven’t seen much of a difference between running Argus on the Intel ixgbe driver vs the pf_ring DNA/libzero driver.

We do have some Gigamon network taps between the Cisco 6500s and our sensors.  I may ask our network team to plug the sensors in directly and see if that changes anything.  Carter, I imagine you have 20x my experience working with netflow in different network architectures.  Have you ever seen this kind of issue with lost SYN/SYNACK packets and either Cisco 6500s or Gigamon taps?

At this point, it think the only weirdness we’re still experiencing has to do with label files in radium.  Ralabel works fine, but radium will either not include a label or it will include it multiple times.  I think I’ve sent you everything you asked for there.  Let me know if I still need to send you anything.

Thanks!

Craig

From: Carter Bullard [mailto:carter at qosient.com]
Sent: Wednesday, August 07, 2013 10:53 AM
To: Craig Merchant
Cc: David Edelman; Argus (argus-info at lists.andrew.cmu.edu<mailto:argus-info at lists.andrew.cmu.edu>)
Subject: Re: [ARGUS] Couple things...

Hey Craig,
Excellent !!!  I was beginning to really question mental stability ,O)
My quess would be  the self synchronization logic, as that does a lot of processing to trigger early flow record generation.  I'll do some testing with your argus.conf file tonight !!!!

Thanks Craig for being persistent  !!!!!

Carter

On Aug 7, 2013, at 1:40 PM, Craig Merchant <cmerchant at responsys.com<mailto:cmerchant at responsys.com>> wrote:
That worked!  Thanks, David.  Not sure what in my argus.conf could be causing the problem.  Here it is if you’re curious:

ARGUS_FLOW_TYPE="Bidirectional"
ARGUS_FLOW_KEY="CLASSIC_5_TUPLE"
ARGUS_DAEMON=no
ARGUS_MONITOR_ID="ids01-dc1"
ARGUS_ACCESS_PORT=561
ARGUS_BIND_IP="10.10.10.10"
ARGUS_INTERFACE=dnacluster:10 at 28
ARGUS_GO_PROMISCUOUS=no
ARGUS_SET_PID=yes
ARGUS_PID_PATH="/var/run"
ARGUS_FLOW_STATUS_INTERVAL=5
ARGUS_IP_TIMEOUT=900
ARGUS_TCP_TIMEOUT=1800
ARGUS_GENERATE_RESPONSE_TIME_DATA=yes
ARGUS_GENERATE_APPBYTE_METRIC=yes
ARGUS_GENERATE_TCP_PERF_METRIC=yes
ARGUS_GENERATE_BIDIRECTIONAL_TIMESTAMPS=yes
ARGUS_CAPTURE_DATA_LEN=10
ARGUS_SELF_SYNCHRONIZE=yes
ARGUS_KEYSTROKE="yes"

From: David Edelman [mailto:dedelman at iname.com]
Sent: Tuesday, August 06, 2013 8:42 PM
To: Craig Merchant; Carter Bullard
Cc: Argus (argus-info at lists.andrew.cmu.edu<mailto:argus-info at lists.andrew.cmu.edu>)
Subject: Re: [ARGUS] Couple things...

Craig,

Just in case you are running into something odd in the argus.conf file, I suggest that you add –X as the very first argument to the invocation of argus. I suggest something very simple like:

# /usr/local/bin/argus –X –r somefile.pcap –w /tmp/somefile.argus

If that works (and /tmp is almost always a good place to write the output because it avoids permission problems) then use recount() on the /tmp/somefile.argus to make sure that everything is as expected and let us know what happened.

--Dave

From: Craig Merchant <cmerchant at responsys.com<mailto:cmerchant at responsys.com>>
Date: Tuesday, August 6, 2013 11:28 PM
To: Carter Bullard <carter at qosient.com<mailto:carter at qosient.com>>
Cc: Argus <argus-info at lists.andrew.cmu.edu<mailto:argus-info at lists.andrew.cmu.edu>>
Subject: Re: [ARGUS] Couple things...

I don’t know what to tell you.  If you want me to run that trace tool and send you the output, let me know where to get it and I’ll figure it out.

Did you take a look at the pcap file to see if there were a lot of missing SYN/SYNACK packets?

Thanks.

Craig

From: Carter Bullard [mailto:carter at qosient.com]
Sent: Tuesday, August 06, 2013 10:02 AM
To: Craig Merchant
Cc: Argus (argus-info at lists.andrew.cmu.edu<mailto:argus-info at lists.andrew.cmu.edu>)
Subject: Re: [ARGUS] Couple things...

Hey Craig,
I'm not having any problems reading your tcpdump.pcap file
with my version of argus, so I can't reproduce a fault.

% thoth:Data carter$ argus -r tcpdump*pcap -w - | racount
racount   records     total_pkts     src_pkts       dst_pkts       total_bytes        src_bytes          dst_bytes
    sum   402665      9999999        5205934        4794065        4795152829         2664296730         2130856099

Is there a specific feature or command line option that generates
your problem?

Carter

On Aug 3, 2013, at 2:23 PM, Carter Bullard <carter at qosient.com<mailto:carter at qosient.com>> wrote:

OK, with the pcap we'll figure it out.

So the ssh keystroke algorithm is round trip sensitive, and its tuned for the enterprise border viewing, but there are a lot of knobs that can be turned.  The real trick is having, again, a packet file of a session so we can see what the algorithm is doing.

Grab a few and we can go over it packet for packet.

Carter

Carter Bullard, QoSient, LLC
150 E. 57th Street Suite 12D
New York, New York 10022
+1 212 588-9133 Phone
+1 212 588-9134 Fax

On Aug 2, 2013, at 3:06 PM, Craig Merchant <cmerchant at responsys.com<mailto:cmerchant at responsys.com>> wrote:
I don’t know what to tell you, Carter.  The version of 3.0.7.4 that I’m running has the same MD5 sum as the latest in qosient.com/dev<http://qosient.com/dev>…

I’ve uploaded the pcap file I’m trying to convert to your FTP server.

I’ve attached the debug file, but after further testing I think it’s an algorithm configuration issue.  I’ve tried testing normal and reverse keystroke detection between hosts that were in the same data center and dnstroke and snstroke always show up as “0,0” or “,,” (the latter happens more when there are directional issues).  But if I watch a host that I ssh into over the VPN from my home connection, Argus detects keystrokes.

I’ve tried reading through the academic paper you guys published on the keystroke detection and it’s beyond me.  If it works for a slower network connection and not a faster network connection (or maybe I should say lower/higher latency connection), which configuration options should I experiment with to find the right balance?

Thanks.

Craig

From: Carter Bullard [mailto:carter at qosient.com]
Sent: Friday, August 02, 2013 8:37 AM
To: Craig Merchant
Cc: Argus (argus-info at lists.andrew.cmu.edu<mailto:argus-info at lists.andrew.cmu.edu>)
Subject: Re: [ARGUS] Couple things...

Hey Craig,
Was in Calif all last week, and just now catching up.

I really think the argus crashing issue is fixed.  At least
it works with all data that has been uploaded.  But if you have
packet data that is blowing argus up, can you send ???

There is a possibility that you may not have the most recent
version of argus-3.0.7.4.  I sometimes put up new software
without changing the number, like if I make a mistake and
put up the wrong version.  So, there could be a race condition.
Check the md5 or date times, or just grab again, if there is
any doubt.

You have to turn on keystroke detection, so, don't comment out
the ARGUS_KEYSTROKE="yes" line.  The CONF line you can comment
out.

To troubleshoot the keystroke algorithm, with argus running, but
not as a daemon, you can send a USR1 signal to it,

   # kill -USR1 argus.pid

and it will print out stats that include the keystroke algorithm
configuration, if its turned on. When you send a USR1 signal to
argus, you increment the Debug flag setting for all of argus, and
so you should start getting debug messages, if the debug facility
is compiled in. Send another USR1 and you'll increase the debug
information.  Most of the per packet keystroke debugging is at
debug level 5.

Send a USR2 signal to argus ( # kill -USR2 argus.pid ) to turn
debug reporting off.

Carter

On Aug 1, 2013, at 7:02 PM, Craig Merchant <cmerchant at responsys.com<mailto:cmerchant at responsys.com>> wrote:

Hey, Carter…

I just wanted to check in and see if you anything else from me on the labeling issue or argus crashing when trying to convert a pcap file.  Let me know…

I’m also having some issues with keystroke detection with the latest release.  The following command used to work in my testing:

/usr/local/bin/ra -S 10.10.10.10:561 -n -u -c "," -s "+0dnstroke,+1snstroke" - host 10.1.1.1 and host 10.1.1.2

I tried both a normal and reverse SSH session between the two hosts and neither one registered keyboard strokes of varying speeds and intensity.

All I’ve done is commented out the defaults in argus.conf:

ARGUS_KEYSTROKE="yes"
ARGUS_KEYSTROKE_CONF="GPC_MAX=4"

I performed pretty much the same testing a couple months ago and got plenty of flows where keystrokes were detected.  Please let me know what you’d recommend for troubleshooting that.

Thanks.

Craig

<debug.zip>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130807/d8773eb6/attachment.html>