raservices crashes when processing

Thu May 16 12:53:59 EDT 2013

Hey Matt,
Well 2K of signatures is too small, obviously, so, thanks for starting that fix.
But, if you don't mind, using the word crash is not good, so lets use the word
" fails ", unless, of course, it really does crash, then crash is the best term ;O)

So rauserdata() is designed to generate upto 16 signatures per application.
While it does want to try to leverage port numbers as application identifiers,
and since there are 64K ports, we probably should be ready for .5M of flows,
I suppose.  Just didn't want to allocate a chunk of memory, and not use it.

You don't need to aggregate the flows to build signatures, or to label
traffic.  I don't really recommend it, but it is a good starting point so, no
harm, no foul. 

Flows can change their character during the life of the flow, but if
you aggregate, you will only match on the " first X bytes " in the
flow.  The feature is really designed to allow you to continuously
monitor flows for application conformance, allow you to know if
the application is still what it started out to be.  Now, of course
there are some flows that start out as one thing, and end up
doing something else.  Starts out http, but downloads code, video,
audio, etc… would be nice to know that it does that.

One of the ways that I demonstrated this feature, was to ftp some
large directories that had lots of different file types, including argus
data, encrypted payloads, packet data, etc…   With argus generating
1-5 second status records, and graphing the services label that
raservices() generated for the status records, you could see when
the ftp changed file types, and the labels would reflect the content
types of the file….  Pretty cool.

Carter

On May 16, 2013, at 11:58 AM, Matt Brown <matthewbrown at gmail.com> wrote:

> Good Morning Carter,
>  
>  
> As far as collecting user data, looks good to me:
>  
> # radump -r * -s suser duser | wc -l
> 195492
> # radump -r * -s suser duser | grep 's\[0\]=""' | wc -l
> 36307
> # radump -r * -s suser duser | grep 's\[[1-9].*' | wc -l
> 159184
>  
> I used the data file produced with rastream:
> rastream -d -S 127.0.0.1:561 -B 15s -M time 1h -w /var/opt/argus/%Y-%m-%d/argus_%T -f /usr/local/bin/rastream.sh
>  
> argus running as:
> argus -d -i eth0 -P 561
>  
> argus.conf with ARGUS_CAPTURE_DATA_LEN set:
> # cat /etc/argus.conf | egrep -v '^$|^[#]'
> ARGUS_FLOW_TYPE="Bidirectional"
> ARGUS_FLOW_KEY="CLASSIC_5_TUPLE"
> ARGUS_MONITOR_ID="..." #         // String
> ARGUS_SET_PID=yes
> ARGUS_PID_PATH="/var/run"
> ARGUS_FLOW_STATUS_INTERVAL=60
> ARGUS_MAR_STATUS_INTERVAL=300
> ARGUS_CAPTURE_DATA_LEN=256
>  
>  
>  
> Working off the contents of ../support/Config/sig.std and Dave's great advice, I performed the following:
> # racluster -r * -w day.cache
> # rauserdata -r day.cache -M printer="encode32" > /tmp/raservices.conf
>  
> Even without editing the file (clearly needs to be analyzed and thinned down to be useful), I tried to run raservices to analyze some data:
>  
> # racluster -r * -s saddr daddr dport suser duser -w - | raservices -f /tmp/raservices.conf
> and it segfaults… (as with Dave)
>  
> # racluster -r * -N 50 -s saddr daddr dport suser duser -w - | raservices -f ~/argus-clients-3.0.7.8/support/Config/std.sig -s saddr daddr dport bytes label:20
> .. produces though and is really very cool!
>  
>  
> Thanks for drilling into the problem Dave!
> Thanks Carter for the solution: adjusting '#define ARGUSMAXSIGFILE` in ../clients/include/argus_client.h then recompiling
> I suppose the file should not need to be larger than 2048 anyway, right?
>  
>  
> Does anyone have any interest sharing their own raservice conf file?
>  
>  
> I also performed what Dave explained, encoding to all other data types, and 'hex' seems to be accepted by raservices, but segfaults (without changing ARGUSMAXSIGFILE from 2048):
> rauserdata -r day.cache -M printer="hex" | head -n 50 > /tmp/raservices.conf
> racluster -r * -s saddr daddr dport suser duser -w - | raservices -f /tmp/raservices.conf -s saddr daddr dport bytes label:20
>  
>  
> So, the raservices conf file should be only as large as it needs to be to define suser and duser contents as a protocol.  It can contain data encoded in 32 bit chars, or maybe hex, and if it's over 2048 bytes, you must adjust the constant and recompile the clients.  Sounds right?
>  
>  
> Thanks very much guys!
>  
> Matt
> 
> On May 16, 2013, at 8:30 AM, Carter Bullard <carter at qosient.com> wrote:
> 
>> Hey Matt,
>> This is not a crash, which is a programatic unrecoverable fault.  You just didn't generate a good raservices() configuration file.
>> 
>> Try using the provided ./support/Config/sig.std, as a starting point for raservices(), to see if you can get good labels?
>> 
>> Are you sucessfully generating user data yet?
>> 
>> Carter
>> 
>> On May 15, 2013, at 5:55 PM, Matt Brown <matthewbrown at gmail.com> wrote:
>> 
>>> Hello all,
>>>  
>>> I took a day's worth of argus data and, as suggested on http://thread.gmane.org/gmane.network.argus/6228/focus=6234, I analyzed it with rauserdata as follows:
>>>  
>>> #racluster -r * -w day.cache
>>> #rauserdata -r day.cache > /tmp/raservices.conf
>>>  
>>>  
>>> I then inspected /tmp/raservices.conf and it's messy (lots of single lines with arbirary ports, likely sport maybe rpc?), but I figured why not give raservices a shot:
>>>  
>>> #racluster -r * -w - | raservices -f raservices.conf
>>>  
>>> I receive the following error:
>>> raservices[21315]: 16:51:00.727719 RaCreateSrvEntry: format error Service: http
>>>  
>>>  
>>> I straced the process, and I see no occurances of "http" in the output (other than the writev()); the data appears to be read correctly until a blank line is read [read(3, "", 4096)                       = 0]:
>>>  
>>> read(3, "\"  \n\nService: 48956             "..., 4096) = 4096
>>> read(3, "...xxxxxx"  dst ="..., 4096) = 4096
>>> read(3, "xxxx"..., 4096) = 689
>>> read(3, "", 4096)                       = 0
>>> close(3)                                = 0
>>> munmap(0xb766e000, 4096)                = 0
>>> gettimeofday({1368651683, 272271}, NULL) = 0
>>> time(NULL)                              = 1368651683
>>> writev(2, [{"raservices[21523]: 17:01:23.2722"..., 79}, {"\n", 1}], 2raservices[21523]: 17:01:23.272271 RaCreateSrvEntry: format error Service: http
>>> ) = 80
>>>  
>>>  
>>> Any idea on why this would be?  Is my data processing flow incorrect?
>>>  
>>>  
>>> Both clients are 3.0.7.8.
>>>  
>>>  
>>> Thanks,
>>>  
>>> Matt

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130516/bc502ef0/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130516/bc502ef0/attachment.bin>