Argus-info Digest, Vol 93, Issue 16
manaf gharaibeh
manafhgh at yahoo.com
Mon May 6 23:43:33 EDT 2013
Hey Carter,
Regarding your comment (replace ARGUS_TM_GMTOFF to be HAVE_TM_GMTOFF. ), this is what I found in the configure file in
the argus-clients directory:
$as_echo "#define HAVE_TM_GMTOFF /**/" >>confdefs.h
ARGUS_TM_GMTOFF is not there at all. So I'm not sure if there is anything to change there.
The version I'm using is argus-clients-3.0.6.2.
I'll be looking into solving the issue via telling rrdtool which time-zone to use as suggested in your previous reply.
Thanks,
-Manaf
________________________________
From: "argus-info-request at lists.andrew.cmu.edu" <argus-info-request at lists.andrew.cmu.edu>
To: argus-info at lists.andrew.cmu.edu
Sent: Monday, May 6, 2013 4:34 PM
Subject: Argus-info Digest, Vol 93, Issue 16
Send Argus-info mailing list submissions to
argus-info at lists.andrew.cmu.edu
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.andrew.cmu.edu/mailman/listinfo/argus-info
or, via email, send a message with subject or body 'help' to
argus-info-request at lists.andrew.cmu.edu
You can reach the person managing the list at
argus-info-owner at lists.andrew.cmu.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Argus-info digest..."
Today's Topics:
1. Re: Shifting ragraph X-axis (Carter Bullard)
2. Re: normalized appbyte ratio for producer/consumer
relationship (John Gerth)
----------------------------------------------------------------------
Message: 1
Date: Mon, 6 May 2013 18:12:44 -0400
From: Carter Bullard <carter at qosient.com>
Subject: Re: [ARGUS] Shifting ragraph X-axis
To: manaf gharaibeh <manafhgh at yahoo.com>
Cc: "argus-info at lists.andrew.cmu.edu"
<argus-info at lists.andrew.cmu.edu>
Message-ID: <5AB7646C-276E-42C6-8906-D67B837A7905 at qosient.com>
Content-Type: text/plain; charset="iso-8859-1"
Hey Manaf,
Your email made me curious, and I found a bug in rasplit() that
generated an error that could be causing you a problem.
If you could grab the new version of argus-clients-3.0.7.8, which
I'll upload tonight, there is a fix for rasplit() that will get the records
into files without the time shift that you are experiencing.
If you want to fix it yourself, and verify the error, edit your copy
of the ./configure program, in your clients root directory to replace
ARGUS_TM_GMTOFF to be HAVE_TM_GMTOFF. Then, rerun
./configure, and make.
% cd to your argus-clients root directory
% edit your copy of ./configure
% ./configure
% make clean; make
Then rerun your rasplit(), writing to a new directory, so that you don't
append to any existing data files. You should find that the records
are now in their correct files.
Sorry for any inconvenience,
Carter
On May 5, 2013, at 4:39 PM, Carter Bullard <carter at qosient.com> wrote:
> Hey Manaf,
> The timestamps in the records are identical to those in the packet capture file.
> Any difference you are experiencing is probably a timezone issue. Print some
> of the records using the " -U " option, which prints the time as seconds from
> the epoch of time, then use "date -r secs" with one of the seconds outputs to
> see if its gives reasonable values.
>
> ra* programs print in localtime(), which uses either the machine timezone or
> one specified in the users environment. So when you want year, month, day,
> hour, etc... ra* programs are using your timezone definition, unless you are
> having it print the time in GMT, which can be set using the RA_TIME_FORMAT
> variable, in the .rarc file. What timezone are you in?
>
> You can define the zone to be anything you want using the rarc file
> configuration variable RA_TZ. In the example rarc file in ./support/Config,
> we provide an example to set the timezone to US Eastern Standard Time,
> with Daylight Savings time definitions.
>
> #RA_TZ="EST5EDT4,M3.2.0/02,M11.1.0/02"
>
> You can use this variable, or you can set your own environment variables
> to set the timezone to use.
>
> I recommend that you use a personal ~/.rarc file to get the time set the way
> you like, then all the ra* programs will do the right thing.
>
> Carter
>
>
> On May 5, 2013, at 5:24 AM, manaf gharaibeh <manafhgh at yahoo.com> wrote:
>
>> Hi,
>>
>> I used argus to transform and aggregate a set of pcap files into an argus file. The resulting argus file has timestamps that are 2 hours behind the original timestamps when the packets were captured. So a 4pm timestamp would be 2pm in the generated argus file. That's fine, but is there a way to shift the ragraph X-axis (add 2 hours to the lables)? I looked into ragraph options, and ra-options but couldn't find what I need.
>>
>> Thanks,
>> -Manaf
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.andrew.cmu.edu/mailman/private/argus-info/attachments/20130506/e0470063/attachment.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
Url : https://lists.andrew.cmu.edu/mailman/private/argus-info/attachments/20130506/e0470063/attachment.bin
------------------------------
Message: 2
Date: Mon, 06 May 2013 15:34:12 -0700
From: John Gerth <gerth at graphics.stanford.edu>
Subject: Re: [ARGUS] normalized appbyte ratio for producer/consumer
relationship
To: Carter Bullard <carter at qosient.com>
Cc: Argus <argus-info at lists.andrew.cmu.edu>
Message-ID:
<23389_1367879659_r46MYHl0031756_51882FE4.8090702 at graphics.stanford.edu>
Content-Type: text/plain; charset=ISO-8859-1
Yes, well an abr of 0.0, even without using -0.0 can have multiple meanings
since you will get 0 as long as s=d even if they are large. The use of -0.0
is perhaps a bit cute, and, as you point out, since IEEE 754 requires 0.0 == -0.0
in all relational tests, one has to use signbit() to disambiguate in C.
Even so, I think your example shows that abr has good potential as a metric.
John Gerth gerth at graphics.stanford.edu Gates 378
On 5/6/2013 2:09 PM, Carter Bullard wrote:
> Hey John,
> OK, so there is a problem, in that IEEE floating point has ( -0.0 == 0.0 )
> by definition. So, I fixed it in my compiler, but others are going to have
> issues when needing to discriminate between 0.0 and -0.0.
>
> So, with the new argus-clients, you can filter for any value for abr, using
> any of the ra* programs. Still have to work on graphing abr, but I'll get
> to that later tonight.
>
> Here is the abr behavior for every DNS request here at QoSient World HQ
> for 2013, that weren't ServFail errors:
>
> thoth:tmp carter$ rahisto -H abr 10:-1.0-1.0 -r argus*domain* -s mean stddev - src pkts 1 and dst pkts 1 and not abr 0.0
> N = 1009841 mean = -0.739084 stddev = 0.102829 max = -0.162791 min = -0.909605
> median = -0.749129 95% = -0.653846
> mode = -0.782609
> Class Interval Freq Rel.Freq Cum.Freq Mean StdDev
> 1 -1.000000e+00 225379 22.3183% 22.3183% -0.815238 0.000650
> 2 -8.000000e-01 738148 73.0955% 95.4137% -0.740887 0.043837
> 3 -6.000000e-01 10553 1.0450% 96.4588% -0.534672 0.048717
> 4 -4.000000e-01 35511 3.5165% 99.9752% -0.283067 0.021374
> 5 -2.000000e-01 250 0.0248% 100.0000% -0.162791 0.000051
> 6 0.000000e+00 0 0.0000% 100.0000%
> 7 2.000000e-01 0 0.0000% 100.0000%
> 8 4.000000e-01 0 0.0000% 100.0000%
> 9 6.000000e-01 0 0.0000% 100.0000%
> 10 8.000000e-01 0 0.0000% 100.0000%
>
>
> So, anything positive would be a behavioral anomaly, from the perspective of this host.
>
> ra -r file.from.the.host - udp and port domain and src pkts 1 and dst pkts 1 and abr gt 0.0
>
> and this would pick out candidates for DNS server availability errors:
>
> ra -r file.from.the.host - udp and port domain and src pkts 1 and dst pkts 1 and abr eq 0.0
>
> Of course, you can do an analysis of every service, and get a rather interesting set of
> " what is normal " behaviors, using this simple type of metric. For those services where
> the abr is always positive or always negative, seeing a shift to the other side, can
> indicate events that should be of interest.
>
> Hope this is helpful,
>
> Carter
>
>
> On May 6, 2013, at 2:41 PM, Carter Bullard <carter at qosient.com <mailto:carter at qosient.com>> wrote:
>
>> Hey John,
>> If no appbytes, currently we return -0.0, but the library knows if there are
>> appbytes or not, so we can return nada, when printing out the values.
>> Right now, when using xml format, you won't get a value.
>>
>> Having problems getting my compiler to tell the difference between 0.0 and -0.0,
>> but should hopefully have this working by this afternoon.
>>
>> Carter
>>
>>
>> On May 6, 2013, at 2:37 PM, John Gerth <gerth at graphics.stanford.edu <mailto:gerth at graphics.stanford.edu>> wrote:
>>
>>> Nice example. I'm looking forward to using this.
>>>
>>> As your example shows, this metric is available for any existing argus
>>> files that were created containing appbyte values. I'm assuming that if
>>> the sensor wasn't configured to capture those, 'abr' is not available.
>>>
>>>
>>> --
>>> John Gerth gerth at graphics.stanford.edu <mailto:gerth at graphics.stanford.edu> Gates 378 (650) 725-3273 fax 725-6949
>>>
>>> On 5/6/2013 11:19 AM, Carter Bullard wrote:
>>>> Hey John,
>>>> OK, so I've implemented " abr " as a new metric, using our normalized equation:
>>>>
>>>> abr = (sappbytes - dappbytes)/(sappbytes + dappbytes)
>>>>
>>>> This generates values between +1.0 - -1.0. +1.0 means that all the app bytes
>>>> were from the source, indicating that the source is a pure PRODUCER, and the
>>>> destination is a pure CONSUMER. You see this in FTP PUT file transfers,
>>>> as an example. The sign bit reverses this relationship.
>>>>
>>>> -0.0 denotes the special case, when there are no appbytes seen.
>>>>
>>>> In the new argus-clients that I'll put up later today, you can print this out using:
>>>>
>>>> ra -r argus.data -s +abr
>>>>
>>>> You can also do operations using this metric, such as filter and generate histograms.
>>>> Here is a run that I did to show how this maybe used in an anomaly detection
>>>> application. Here is the simple frequency distribution for all the internal DNS
>>>> requests made to my local DNS server from a specific client, for all of 2013:
>>>>
>>>> thoth:06 carter$ pwd
>>>> /Volumes/Data/Archive/QoSient/192.168.0.68/2013
>>>> thoth:tmp carter$ rahisto -H abr 10:-1.0-1.0 -R . -s mean stddev - udp port domain and src pkts 1 and dst pkts 1
>>>> N = 1027764 mean = -0.726195 stddev = 0.140532 max = 0.000000 min = -0.909605
>>>> median = -0.749129 95% = -0.292517
>>>> mode = -0.782609
>>>> Class Interval Freq Rel.Freq Cum.Freq Mean StdDev
>>>> 1 -1.000000e+00 225379 21.9291% 21.9291% -0.815238 0.000650
>>>> 2 -8.000000e-01 738148 71.8208% 93.7498% -0.740887 0.043837
>>>> 3 -6.000000e-01 10553 1.0268% 94.7766% -0.534672 0.048717
>>>> 4 -4.000000e-01 35511 3.4552% 98.2318% -0.283067 0.021374
>>>> 5 -2.000000e-01 250 0.0243% 98.2561% -0.162791 0.000051
>>>> 6 0.000000e+00 17923 1.7439% 100.0000% 0.000000 0.000000
>>>> 7 2.000000e-01 0 0.0000% 100.0000%
>>>> 8 4.000000e-01 0 0.0000% 100.0000%
>>>> 9 6.000000e-01 0 0.0000% 100.0000%
>>>> 10 8.000000e-01 0 0.0000% 100.0000%
>>>>
>>>>
>>>> OK, should be very clear, that my host is a net CONSUMER of DNS data, not a net PRODUCER
>>>> because the " abr <= 0 ". The corollary holds true, the local DNS service is a net PRODUCER of
>>>> data, and not a net CONSUMER of data, from the prospective of this particular end system.
>>>> So testing filters like this:
>>>> ra -r daily.file - abr gt 0 and port domain and src pkts 1 and dst pkts 1
>>>>
>>>> Should reveal flows that deserve a closer look.
>>>>
>>>> OK, there were a lot of flows where the ( abr == 0 ), which was surprising.
>>>> When DNS experiences a ServFail, the response is the same as the request, just with an error bit
>>>> set in the DNS header. QoSient had a big issue in Jan, 2013, when 17923 DNS ServFail failures
>>>> occurred, so that is where the ( abr == 0 ) flows occured. Important to know this when evaluating
>>>> DNS as a channel for CONSUMER to PRODUCER conversion.
>>>>
>>>> But for DNS health and operability, looking for flows where the ( sappbytes == dappbytes ) is
>>>> also a pretty interesting thing to look for.
>>>>
>>>> Hope this is helpful,
>>>>
>>>> Carter
>>>>
>>>> Carter Bullard
>>>> CEO/President
>>>> QoSient, LLC
>>>> 150 E. 57th Street Suite 12D
>>>> New York, New York 10022
>>>>
>>>> +1 212 588-9133 Phone
>>>> +1 212 588-9134 Fax
>>>>
>>>
>>
>
------------------------------
_______________________________________________
Argus-info mailing list
Argus-info at lists.andrew.cmu.edu
https://lists.andrew.cmu.edu/mailman/listinfo/argus-info
End of Argus-info Digest, Vol 93, Issue 16
******************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130506/1c780c63/attachment.html>
More information about the argus
mailing list