Fwd: raports results after racluster or ra -w .. of original data files

Carter Bullard carter at qosient.com
Wed Jul 21 15:21:05 EDT 2010



Begin forwarded message:

> From: Carter Bullard <carter at qosient.com>
> Date: July 21, 2010 3:17:55 PM EDT
> To: Michael Sanderson <sanders at cs.ubc.ca>
> Subject: Re: [ARGUS] raports results after racluster or ra -w .. of original data files
> 
> Hey Michael,
> Fixed the segfault in argus-clients-3.0.3.16.  Looking at the other issues now.  This
> patch should be usable on your current dist.
> 
> thoth:common carter$ p4 diff ...
> ==== //depot/argus/clients/common/argus_client.c#184 - /home/carter/argus/clients/common/argus_client.c ====
> 4450d4449
> <                            case ARGUS_FLOW_INDEX:
> 4468a4468,4474
>>                           case ARGUS_FLOW_INDEX: {
>>                              if ((retn->dsrs[i] = ArgusCalloc(1, sizeof(struct ArgusFlow))) == NULL)
>>                                 ArgusLog (LOG_ERR, "ArgusCopyRecordStruct: ArgusCalloc error %s\n", strerror(errno));
>>                              bcopy((char *)rec->dsrs[i], (char *)retn->dsrs[i], len * 4);
>>                              break;
>>                           }
>> 
> 
> 
> Carter
> 
> 
> On Jul 21, 2010, at 12:25 PM, Michael Sanderson wrote:
> 
>> On 07/20/10 02:40 PM, Michael Sanderson wrote:
>>> On 07/20/10 02:24 PM, Michael Sanderson wrote:
>>>> On 07/20/10 01:14 PM, Carter Bullard wrote:
>>>>> ftp://qosient.com/incoming
>>>> 
>>>> data.2010.07.12-00:00:01.gz is there now. Clients are running on
>>>> OpenSuSE 11.1 32-bit, compiled locally.
>>>> 
>>>> On a different note, racluster from 3.0.3.15 appears to be suffering
>>>> from a double free when run against files from a 3.0.2 'ra -S sensor'.
>>>> They seem to run fine through the 3.0.2 racluster. Assuming that it
>>>> wasn't something else that caused this, I'll track down a particular
>>>> file and pass it along.
>> 
>> Carter, I just discovered that the racluster that had successfully processed the files was a 3.0.0 racluster, not the 3.0.2 racluster. 3.0.2 segfaults like 3.0.3.15.
>> 
>>   Michael Sanderson
>> 
>>> data.2010.07.06-02:00:01.gz is from a 3.0.2 'ra -S sensor', renamed and
>>> then compressed. Using
>>> 
>>> racluster -M none -r data.2010.07.06-02:00:01.gz -w data
>>> 
>>> I get a segmentation fault. More specifically:
>>> 
>>> *** glibc detected *** racluster: free(): invalid pointer: 0x08700000 ***
>>> ======= Backtrace: =========
>>> /lib/libc.so.6[0xb76cf654]
>>> /lib/libc.so.6(cfree+0x9c)[0xb76d0f3c]
>>> racluster[0x80aae43]
>>> racluster[0x804bfb7]
>>> racluster[0x804cb65]
>>> /lib/libc.so.6(__libc_start_main+0xe5)[0xb7679705]
>>> racluster[0x804ab51]
>>> ======= Memory map: ========
>>> 08048000-080cd000 r-xp 00000000 08:07 1396
>>> /var/tmp/argus-clients-3.0.3.15/bin/racluster
>>> 080cd000-080ce000 r--p 00084000 08:07 1396
>>> /var/tmp/argus-clients-3.0.3.15/bin/racluster
>>> 080ce000-080d7000 rw-p 00085000 08:07 1396
>>> /var/tmp/argus-clients-3.0.3.15/bin/racluster
>>> 080d7000-0b6d4000 rw-p 080d7000 00:00 0 [heap]
>>> b7200000-b7221000 rw-p b7200000 00:00 0
>>> b7221000-b7300000 ---p b7221000 00:00 0
>>> b7379000-b7383000 r-xp 00000000 08:01 69105 /lib/libnss_files-2.9.so
>>> b7383000-b7384000 r--p 00009000 08:01 69105 /lib/libnss_files-2.9.so
>>> b7384000-b7385000 rw-p 0000a000 08:01 69105 /lib/libnss_files-2.9.so
>>> b7393000-b73a8000 r-xp 00000000 08:01 69099 /lib/libnsl-2.9.so
>>> b73a8000-b73a9000 r--p 00014000 08:01 69099 /lib/libnsl-2.9.so
>>> b73a9000-b73aa000 rw-p 00015000 08:01 69099 /lib/libnsl-2.9.so
>>> b73aa000-b73ac000 rw-p b73aa000 00:00 0
>>> b73ac000-b73b5000 r-xp 00000000 08:01 69109 /lib/libnss_nis-2.9.so
>>> b73b5000-b73b6000 r--p 00008000 08:01 69109 /lib/libnss_nis-2.9.so
>>> b73b6000-b73b7000 rw-p 00009000 08:01 69109 /lib/libnss_nis-2.9.so
>>> b749e000-b749f000 rw-p b749e000 00:00 0
>>> b74fb000-b7663000 rw-p b74fb000 00:00 0
>>> b7663000-b77b8000 r-xp 00000000 08:01 69088 /lib/libc-2.9.so
>>> b77b8000-b77b9000 ---p 00155000 08:01 69088 /lib/libc-2.9.so
>>> b77b9000-b77bb000 r--p 00155000 08:01 69088 /lib/libc-2.9.so
>>> b77bb000-b77bc000 rw-p 00157000 08:01 69088 /lib/libc-2.9.so
>>> b77bc000-b77bf000 rw-p b77bc000 00:00 0
>>> b77bf000-b77d5000 r-xp 00000000 08:01 69114 /lib/libpthread-2.9.so
>>> b77d5000-b77d6000 r--p 00015000 08:01 69114 /lib/libpthread-2.9.so
>>> b77d6000-b77d7000 rw-p 00016000 08:01 69114 /lib/libpthread-2.9.so
>>> b77d7000-b77d9000 rw-p b77d7000 00:00 0
>>> b77d9000-b77ec000 r-xp 00000000 08:01 69263 /lib/libz.so.1.2.3
>>> b77ec000-b77ed000 r--p 00012000 08:01 69263 /lib/libz.so.1.2.3
>>> b77ed000-b77ee000 rw-p 00013000 08:01 69263 /lib/libz.so.1.2.3
>>> b77ee000-b7815000 r-xp 00000000 08:01 69096 /lib/libm-2.9.so
>>> b7815000-b7816000 r--p 00026000 08:01 69096 /lib/libm-2.9.so
>>> b7816000-b7817000 rw-p 00027000 08:01 69096 /lib/libm-2.9.so
>>> b783b000-b7848000 r-xp 00000000 08:01 89881 /lib/libgcc_s.so.1
>>> b7848000-b7849000 r--p 0000c000 08:01 89881 /lib/libgcc_s.so.1
>>> b7849000-b784a000 rw-p 0000d000 08:01 89881 /lib/libgcc_s.so.1
>>> b784a000-b784b000 rw-p b784a000 00:00 0
>>> b784b000-b7869000 r-xp 00000000 08:01 69081 /lib/ld-2.9.so
>>> b7869000-b786a000 r--p 0001d000 08:01 69081 /lib/ld-2.9.so
>>> b786a000-b786b000 rw-p 0001e000 08:01 69081 /lib/ld-2.9.so
>>> bfa0b000-bfa4f000 rw-p bffbc000 00:00 0 [stack]
>>> ffffe000-fffff000 r-xp 00000000 00:00 0 [vdso]
>>> Abort
>>> 
>>> Runs against other data files from a 3.0.2 ra can produce a similar
>>> backtrace, but with 'double free or corruption (out):...' messages.
>>> 
>>> Running a 3.0.3.15 'ra -r 2010.07.06-02:00:01.gz -w data' is fine.
>>> Running 3.0.3.15 'racluster -M none -r data -w data2' seg faults.
>>> 
>>> Michael Sanderson
>>> 
>>>> Michael Sanderson
>>>> 
>>>>> Thanks!!!!
>>>>> Carter
>>>>> 
>>>>> 
>>>>> On Jul 20, 2010, at 3:38 PM, Michael Sanderson wrote:
>>>>> 
>>>>>> On 07/15/10 05:28 PM, Carter Bullard wrote:
>>>>>>> Sounds like a bug. Do you have a data file that I can check to see
>>>>>>> what's up?
>>>>>> 
>>>>>> I was trying to find a way to get a smaller file created, but it
>>>>>> seems that any rewriting generates something that has corrected the
>>>>>> flow.
>>>>>> 
>>>>>> The file I have is about 15MB. Where can I drop it? It's a little
>>>>>> large to send via email.
>>>>>> 
>>>>>>> raports -r data.2010.07.12-00:00:01.gz - host 10.91.129.8
>>>>>> 142.103.7.9 udp: (1) 815
>>>>>> 10.91.129.8 tcp: (2) 2049, 39813
>>>>>> 10.91.129.8 udp: (1) 769
>>>>>>> racluster -M none -r data.2010.07.12-00:00:01.gz -w data \
>>>>>> - host 10.91.129.8
>>>>>>> raports -r data
>>>>>> 142.103.7.9 udp: (1) 815
>>>>>> 10.91.129.8 tcp: (1) 2049
>>>>>> 10.91.129.8 udp: (1) 769
>>>>>> 142.103.4.102 tcp: (1) 139
>>>>>> 
>>>>>> Michael Sanderson
>>>>>> 
>>>>>>> Carter
>>>>>>> 
>>>>>>> 
>>>>>>> On Jul 15, 2010, at 7:11 PM, Michael Sanderson wrote:
>>>>>>> 
>>>>>>>> On 07/15/10 02:43 PM, Carter Bullard wrote:
>>>>>>>>> Hey Michael,
>>>>>>>>> I assume that you are using argus-clients-3.0.3.x clients.
>>>>>>>> 
>>>>>>>> Yes, 3.0.3.15, not sure when this copy was downloaded.
>>>>>>>> 
>>>>>>>>> Argus doesn't make any determination regarding flow source or
>>>>>>>>> destination, this is all done
>>>>>>>>> in the clients.
>>>>>>>> 
>>>>>>>> Understood.
>>>>>>>> 
>>>>>>>>> If you read data using any of the ra* programs, they have an
>>>>>>>>> opportunity to
>>>>>>>>> 'correct' a flow record for direction, and we do have some logic
>>>>>>>>> for TCP
>>>>>>>>> traffic.
>>>>>>>>> If we haven't seen the syn or the synack, we test the source and
>>>>>>>>> destination ports to
>>>>>>>>> see if the src is in the IPPORT_RESERVED range, 1-1024, and the
>>>>>>>>> dst port
>>>>>>>>> is above it.
>>>>>>>>> If so, if the sport is not ftp-data, and there is an entry in the
>>>>>>>>> /etc/services file for the sport
>>>>>>>>> value, so its an identified service port, we'll reverse the
>>>>>>>>> direction of
>>>>>>>>> the record.
>>>>>>>>> 
>>>>>>>>> Does that seem useful, or do you see a problem?
>>>>>>>> 
>>>>>>>> Yes, this seems useful and was accurate in this case. The problem
>>>>>>>> is that I don't get the same output when I run raports on data
>>>>>>>> written by 'ra -S ... -w file' and when I run raports on data that
>>>>>>>> has been preprocessed via 'ra -r file -w newfile - host a.b.c.d' or
>>>>>>>> 'racluster -M none -r file -w newfile'. (See original post for the
>>>>>>>> specifics of how things are run and actual output.) It appears that
>>>>>>>> it takes the act of writing the records out to a file to get the
>>>>>>>> direction correction to completely take effect, otherwise the
>>>>>>>> different raports runs would show the same results. Is that accurate?
>>>>>>>> 
>>>>>>>> Michael Sanderson
>>>>>>>> 
>>>>>>>>> Carter
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Jul 14, 2010, at 4:43 PM, Michael Sanderson wrote:
>>>>>>>>> 
>>>>>>>>>> I'm experiencing some slight differences in the results of raports,
>>>>>>>>>> depending upon how I run it.
>>>>>>>>>> 
>>>>>>>>>> I rotate my logs every 15 minutes and do some post processing.
>>>>>>>>>> 
>>>>>>>>>> 1) raports -r data.2010.07.12-00:* - host 10.91.129.8
>>>>>>>>>> ...
>>>>>>>>>> 10.91.129.8 tcp: (2) 2049, 39813
>>>>>>>>>> 10.91.129.8 udp: (3) 769, 789, 824
>>>>>>>>>> ...
>>>>>>>>>> 
>>>>>>>>>> 2) ra -r data.2010.07.12-00:* -w d10.91.129.8 - host 10.91.129.8
>>>>>>>>>> raports -r d10.91.129.8
>>>>>>>>>> ...
>>>>>>>>>> 10.91.129.8 tcp: (1) 2049
>>>>>>>>>> 10.91.129.8 udp: (3) 769, 789, 824
>>>>>>>>>> ...
>>>>>>>>>> 
>>>>>>>>>> 3) racluster -r data.2010.07.12-00:* -w clustered.2010.07.12-00
>>>>>>>>>> raports -r clustered.2010.07.12-00 - host 10.91.129.8
>>>>>>>>>> ...
>>>>>>>>>> 10.91.129.8 tcp: (1) 2049
>>>>>>>>>> 10.91.129.8 udp: (3) 769, 789, 824
>>>>>>>>>> ...
>>>>>>>>>> 
>>>>>>>>>> raports on the original data files is suggesting that 10.91.129.8
>>>>>>>>>> port
>>>>>>>>>> 39813 is a destination. raports on processed files is suggesting
>>>>>>>>>> it is
>>>>>>>>>> a source. Looking at the flows from the original data files for
>>>>>>>>>> 10.91.129.8 port 39813 shows that direction wasn't determined.
>>>>>>>>>> 
>>>>>>>>>> 10/07/11 23:49:23 M tcp 10.91.129.8.39813<?> a.b.c.d.netbio 2 126
>>>>>>>>>> CON
>>>>>>>>>> 
>>>>>>>>>> Looking at the processed files (d10.91.129.8 and
>>>>>>>>>> clustered.2010.07.12-00) via ra, they also so that the direction
>>>>>>>>>> was
>>>>>>>>>> not determined.
>>>>>>>>>> 
>>>>>>>>>> What is happening with the processed files that convinces raports
>>>>>>>>>> that
>>>>>>>>>> 10.91.129.8 port 39813 is a source, not a destination? In this
>>>>>>>>>> particular case it is right, but I'm curious how it is getting it
>>>>>>>>>> right.
>>>>>>>>>> 
>>>>>>>>>> Michael Sanderson
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> Carter Bullard
>>>>>>> CEO/President
>>>>>>> QoSient, LLC
>>>>>>> 150 E 57th Street Suite 12D
>>>>>>> New York, New York 10022
>>>>>>> 
>>>>>>> +1 212 588-9133 Phone
>>>>>>> +1 212 588-9134 Fax
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> Carter Bullard
>>>>> CEO/President
>>>>> QoSient, LLC
>>>>> 150 E 57th Street Suite 12D
>>>>> New York, New York 10022
>>>>> 
>>>>> +1 212 588-9133 Phone
>>>>> +1 212 588-9134 Fax
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> 
> 
> Carter Bullard
> CEO/President
> QoSient, LLC
> 150 E 57th Street Suite 12D
> New York, New York  10022
> 
> +1 212 588-9133 Phone
> +1 212 588-9134 Fax
> 
> 
> 

Carter Bullard
CEO/President
QoSient, LLC
150 E 57th Street Suite 12D
New York, New York  10022

+1 212 588-9133 Phone
+1 212 588-9134 Fax



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20100721/2c5af633/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3815 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20100721/2c5af633/attachment.bin>


More information about the argus mailing list