argus data labels and DNS names
Carter Bullard
carter at qosient.com
Fri May 17 13:51:22 EDT 2013
Hey Dave,
I see that you're putting name resolutions in your flow data labels. Good idea…
There are a lot of buttons and dials for name lookups in argus and the clients.
Now that someone is doing this on the list, we should turn this stuff on for labels.
In the new clients that I'll put up tonight/tomorrow, there are a few new
variables in the ./support/Config/ralabel.conf file to turn on/off various DNS
functions.
We have a non-blocking DNS resolver in the library, and clients like ratop() and
rasqlinsert() currently use it so that they are not blocking, waiting for the DNS query to
return. There is a new variable to turn that on or off. If you MUST have a
DNS name at the time of labeling, then you would set this to "no". If you
can handle lazy lookups, which keeps radium() going fast, then I would set
this to "yes".
There are a few new variables to specify what you want in the name,
host name only (truncate the domain name) or just the domain name
(snip off the leading chars upto the first ' . '). These can be helpful.
Currently, radium(), when it does name lookups, caches the name …..
forever. A name cache timeout can be useful, especially since programs
like radium() can run for a long time. 0 means turn the cache off, which
can be expensive. -1 will yield an infinite cache.
Give these a try, when they come out in argus-clients-3.0.7.10, later today/tomorrow.
Carter
On May 17, 2013, at 12:32 PM, Carter Bullard <carter at qosient.com> wrote:
> Hey Dave,
> Its not ArgusMergeLabels() that has a problem. raservices() is munging the
> string that ArgusMergeLabels() returns. Copy this version of raservices.c,
> to ./examples/raservices, and re-make. Should fix things.
>
> Carter
> <raservices.c>
>
>
> On May 17, 2013, at 11:42 AM, "Dave Edelman" <dedelman at iname.com> wrote:
>
>> Carter,
>>
>> I have it working but I think that there is a problem with ArgusMergeLabels() when it is set for ArgusUnion. If you look at the attached file, it seems that the buffer is not being cleared correctly. I am running raservices againstunclustered flow records that have been labeled by radium as they arrive from the argus collector. I can provide the equivalent ra output if you want, that’s why I included the offset.
>>
>> --Dave
>>
>>
>> From: Carter Bullard [mailto:carter at qosient.com]
>> Sent: Thursday, May 16, 2013 11:14 AM
>> To: Dave Edelman
>> Cc: 'Matt Brown'; argus-info at lists.andrew.cmu.edu
>> Subject: Re: [ARGUS] raservices crashes when processing
>>
>> Hey Dave,
>> Of course, everything in the clients has a constant defined somewhere.
>> Change the value of ARGUSMAXSIGFILE in ./include/argus_client.h to
>> something like this:
>>
>> ==== //depot/argus/clients/include/argus_client.h#64 - /Volumes/Users/carter/argus/clients/include/argus_client.h ====
>> 142c142
>> < #define ARGUSMAXSIGFILE 2048
>> ---
>> > #define ARGUSMAXSIGFILE 0x80000
>>
>>
>> Carter
>>
>> On May 16, 2013, at 8:51 AM, "Dave Edelman" <dedelman at iname.com> wrote:
>>
>>
>> The std.sig is fine but it is 435 lines long.
>> If I use rauserdata to create a filter file which is longer than 2048 lines (including empty lines) raservices segfaults. If I take the first middle or last 2048 lines of my filter file, raservices is fine. If I remove all of the blank lines from my filter file I can still use any 2048 lines with no problem but raservices segfaults on 2049 lines in the filter file.
>>
>> --Dave
>>
>> From: Carter Bullard [mailto:carter at qosient.com]
>> Sent: Thursday, May 16, 2013 8:37 AM
>> To: Dave Edelman
>> Cc: Matt Brown; <argus-info at lists.andrew.cmu.edu>
>> Subject: Re: [ARGUS] raservices crashes when processing
>>
>> Hey Dave,
>> Not sure that I follow your situation. So you're having problems with the provided sig.std or one you created?
>>
>> Carter
>>
>> On May 15, 2013, at 8:59 PM, "Dave Edelman" <dedelman at iname.com> wrote:
>>
>> I had the same results so I looked at an example in the argus-client distribution. /support/Config/std.sig has this header:
>>
>> # Services fingerprint file, generated by:
>> # rauserdata -d16 -e encode32
>> #
>> # with modifications.
>> #
>>
>> The –e option is for regular expression pattern matching so I replaced it with -M printer=’encode32’ and I didn’t use a –d parameter and the output looked much closer to the sample. I can now get raservices to core dump reliably with a segfault.
>>
>> When I use the sample signature file and I tell raservices to output the label by using the –s +label:50 I do get a bunch of labels with the value srv=xxxxxx
>>
>> My data is already the output of a day’s worth of flows run through racluster.
>>
>> raservices -r argusTestData_2013_05_09 -f std.sig -s +label:50
>>
>> 2013-05-09-01:28:18.230 *U udp 10.1.1.50 61266 -> 10.1.1.10 disca* 1 0 148 0 INT
>> srv=ndmp
>> 2013-05-09-16:10:32.206 *U udp 10.1.1.50 61389 -> 10.1.1.10 disca* 1 0 148 0 INT
>> srv=ndmp
>>
>> So I took the first 500 lines of my filter file and attempted to use that rather than the full file
>> head -500 userdata.out > smallUesrData
>> raservices -r argusTestData_2013_05_09 -f smallUesrData -s +label:50 | head -30
>> StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport SrcPkts DstPkts SrcBytes DstBytes State
>> Label
>> 2013-05-09-23:00:03.294 man 0 0 0 0 0 0 0 0 STA
>> 2013-05-09-00:00:01.993 * d tcp 10.1.1.45 50899 <?> 10.1.1.10 micro* 9075967 13492977 1560166894 15748525452 CON
>> srv=microsoft-ds
>> 2013-05-09-00:00:28.890 * & tcp 10.1.1.50 iad3 <?> 10.1.1.10 micro* 2034770 2870198 349463675 3595651910 CON
>> srv=microsoft-ds
>>
>> Now for the binary search J my filter file has 4316 lines. If I use the first 2048 I am fine, 2049 ends us with a segfault. I delete the first 100 lines of the original file and the first 2048 still works and 2049 still dies. This could be a clue.
>>
>> --Dave
>>
>>
>>
>>
>>
>>
>> From: argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu [mailto:argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu] On Behalf Of Matt Brown
>> Sent: Wednesday, May 15, 2013 5:56 PM
>> To: argus-info at lists.andrew.cmu.edu
>> Subject: [ARGUS] raservices crashes when processing
>>
>> Hello all,
>>
>> I took a day's worth of argus data and, as suggested on http://thread.gmane.org/gmane.network.argus/6228/focus=6234, I analyzed it with rauserdata as follows:
>>
>> #racluster -r * -w day.cache
>> #rauserdata -r day.cache > /tmp/raservices.conf
>>
>>
>> I then inspected /tmp/raservices.conf and it's messy (lots of single lines with arbirary ports, likely sport maybe rpc?), but I figured why not give raservices a shot:
>>
>> #racluster -r * -w - | raservices -f raservices.conf
>>
>> I receive the following error:
>> raservices[21315]: 16:51:00.727719 RaCreateSrvEntry: format error Service: http
>>
>>
>> I straced the process, and I see no occurances of "http" in the output (other than the writev()); the data appears to be read correctly until a blank line is read [read(3, "", 4096) = 0]:
>>
>> read(3, "\" \n\nService: 48956 "..., 4096) = 4096
>> read(3, "...xxxxxx" dst ="..., 4096) = 4096
>> read(3, "xxxx"..., 4096) = 689
>> read(3, "", 4096) = 0
>> close(3) = 0
>> munmap(0xb766e000, 4096) = 0
>> gettimeofday({1368651683, 272271}, NULL) = 0
>> time(NULL) = 1368651683
>> writev(2, [{"raservices[21523]: 17:01:23.2722"..., 79}, {"\n", 1}], 2raservices[21523]: 17:01:23.272271 RaCreateSrvEntry: format error Service: http
>> ) = 80
>>
>>
>> Any idea on why this would be? Is my data processing flow incorrect?
>>
>>
>> Both clients are 3.0.7.8.
>>
>>
>> Thanks,
>>
>> Matt
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130517/3b5a7d08/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130517/3b5a7d08/attachment.bin>
More information about the argus
mailing list