Fwd: raports results after racluster or ra -w .. of original data files
Carter Bullard
carter at qosient.com
Wed Jul 21 15:21:05 EDT 2010
Begin forwarded message:
> From: Carter Bullard <carter at qosient.com>
> Date: July 21, 2010 3:17:55 PM EDT
> To: Michael Sanderson <sanders at cs.ubc.ca>
> Subject: Re: [ARGUS] raports results after racluster or ra -w .. of original data files
>
> Hey Michael,
> Fixed the segfault in argus-clients-3.0.3.16. Looking at the other issues now. This
> patch should be usable on your current dist.
>
> thoth:common carter$ p4 diff ...
> ==== //depot/argus/clients/common/argus_client.c#184 - /home/carter/argus/clients/common/argus_client.c ====
> 4450d4449
> < case ARGUS_FLOW_INDEX:
> 4468a4468,4474
>> case ARGUS_FLOW_INDEX: {
>> if ((retn->dsrs[i] = ArgusCalloc(1, sizeof(struct ArgusFlow))) == NULL)
>> ArgusLog (LOG_ERR, "ArgusCopyRecordStruct: ArgusCalloc error %s\n", strerror(errno));
>> bcopy((char *)rec->dsrs[i], (char *)retn->dsrs[i], len * 4);
>> break;
>> }
>>
>
>
> Carter
>
>
> On Jul 21, 2010, at 12:25 PM, Michael Sanderson wrote:
>
>> On 07/20/10 02:40 PM, Michael Sanderson wrote:
>>> On 07/20/10 02:24 PM, Michael Sanderson wrote:
>>>> On 07/20/10 01:14 PM, Carter Bullard wrote:
>>>>> ftp://qosient.com/incoming
>>>>
>>>> data.2010.07.12-00:00:01.gz is there now. Clients are running on
>>>> OpenSuSE 11.1 32-bit, compiled locally.
>>>>
>>>> On a different note, racluster from 3.0.3.15 appears to be suffering
>>>> from a double free when run against files from a 3.0.2 'ra -S sensor'.
>>>> They seem to run fine through the 3.0.2 racluster. Assuming that it
>>>> wasn't something else that caused this, I'll track down a particular
>>>> file and pass it along.
>>
>> Carter, I just discovered that the racluster that had successfully processed the files was a 3.0.0 racluster, not the 3.0.2 racluster. 3.0.2 segfaults like 3.0.3.15.
>>
>> Michael Sanderson
>>
>>> data.2010.07.06-02:00:01.gz is from a 3.0.2 'ra -S sensor', renamed and
>>> then compressed. Using
>>>
>>> racluster -M none -r data.2010.07.06-02:00:01.gz -w data
>>>
>>> I get a segmentation fault. More specifically:
>>>
>>> *** glibc detected *** racluster: free(): invalid pointer: 0x08700000 ***
>>> ======= Backtrace: =========
>>> /lib/libc.so.6[0xb76cf654]
>>> /lib/libc.so.6(cfree+0x9c)[0xb76d0f3c]
>>> racluster[0x80aae43]
>>> racluster[0x804bfb7]
>>> racluster[0x804cb65]
>>> /lib/libc.so.6(__libc_start_main+0xe5)[0xb7679705]
>>> racluster[0x804ab51]
>>> ======= Memory map: ========
>>> 08048000-080cd000 r-xp 00000000 08:07 1396
>>> /var/tmp/argus-clients-3.0.3.15/bin/racluster
>>> 080cd000-080ce000 r--p 00084000 08:07 1396
>>> /var/tmp/argus-clients-3.0.3.15/bin/racluster
>>> 080ce000-080d7000 rw-p 00085000 08:07 1396
>>> /var/tmp/argus-clients-3.0.3.15/bin/racluster
>>> 080d7000-0b6d4000 rw-p 080d7000 00:00 0 [heap]
>>> b7200000-b7221000 rw-p b7200000 00:00 0
>>> b7221000-b7300000 ---p b7221000 00:00 0
>>> b7379000-b7383000 r-xp 00000000 08:01 69105 /lib/libnss_files-2.9.so
>>> b7383000-b7384000 r--p 00009000 08:01 69105 /lib/libnss_files-2.9.so
>>> b7384000-b7385000 rw-p 0000a000 08:01 69105 /lib/libnss_files-2.9.so
>>> b7393000-b73a8000 r-xp 00000000 08:01 69099 /lib/libnsl-2.9.so
>>> b73a8000-b73a9000 r--p 00014000 08:01 69099 /lib/libnsl-2.9.so
>>> b73a9000-b73aa000 rw-p 00015000 08:01 69099 /lib/libnsl-2.9.so
>>> b73aa000-b73ac000 rw-p b73aa000 00:00 0
>>> b73ac000-b73b5000 r-xp 00000000 08:01 69109 /lib/libnss_nis-2.9.so
>>> b73b5000-b73b6000 r--p 00008000 08:01 69109 /lib/libnss_nis-2.9.so
>>> b73b6000-b73b7000 rw-p 00009000 08:01 69109 /lib/libnss_nis-2.9.so
>>> b749e000-b749f000 rw-p b749e000 00:00 0
>>> b74fb000-b7663000 rw-p b74fb000 00:00 0
>>> b7663000-b77b8000 r-xp 00000000 08:01 69088 /lib/libc-2.9.so
>>> b77b8000-b77b9000 ---p 00155000 08:01 69088 /lib/libc-2.9.so
>>> b77b9000-b77bb000 r--p 00155000 08:01 69088 /lib/libc-2.9.so
>>> b77bb000-b77bc000 rw-p 00157000 08:01 69088 /lib/libc-2.9.so
>>> b77bc000-b77bf000 rw-p b77bc000 00:00 0
>>> b77bf000-b77d5000 r-xp 00000000 08:01 69114 /lib/libpthread-2.9.so
>>> b77d5000-b77d6000 r--p 00015000 08:01 69114 /lib/libpthread-2.9.so
>>> b77d6000-b77d7000 rw-p 00016000 08:01 69114 /lib/libpthread-2.9.so
>>> b77d7000-b77d9000 rw-p b77d7000 00:00 0
>>> b77d9000-b77ec000 r-xp 00000000 08:01 69263 /lib/libz.so.1.2.3
>>> b77ec000-b77ed000 r--p 00012000 08:01 69263 /lib/libz.so.1.2.3
>>> b77ed000-b77ee000 rw-p 00013000 08:01 69263 /lib/libz.so.1.2.3
>>> b77ee000-b7815000 r-xp 00000000 08:01 69096 /lib/libm-2.9.so
>>> b7815000-b7816000 r--p 00026000 08:01 69096 /lib/libm-2.9.so
>>> b7816000-b7817000 rw-p 00027000 08:01 69096 /lib/libm-2.9.so
>>> b783b000-b7848000 r-xp 00000000 08:01 89881 /lib/libgcc_s.so.1
>>> b7848000-b7849000 r--p 0000c000 08:01 89881 /lib/libgcc_s.so.1
>>> b7849000-b784a000 rw-p 0000d000 08:01 89881 /lib/libgcc_s.so.1
>>> b784a000-b784b000 rw-p b784a000 00:00 0
>>> b784b000-b7869000 r-xp 00000000 08:01 69081 /lib/ld-2.9.so
>>> b7869000-b786a000 r--p 0001d000 08:01 69081 /lib/ld-2.9.so
>>> b786a000-b786b000 rw-p 0001e000 08:01 69081 /lib/ld-2.9.so
>>> bfa0b000-bfa4f000 rw-p bffbc000 00:00 0 [stack]
>>> ffffe000-fffff000 r-xp 00000000 00:00 0 [vdso]
>>> Abort
>>>
>>> Runs against other data files from a 3.0.2 ra can produce a similar
>>> backtrace, but with 'double free or corruption (out):...' messages.
>>>
>>> Running a 3.0.3.15 'ra -r 2010.07.06-02:00:01.gz -w data' is fine.
>>> Running 3.0.3.15 'racluster -M none -r data -w data2' seg faults.
>>>
>>> Michael Sanderson
>>>
>>>> Michael Sanderson
>>>>
>>>>> Thanks!!!!
>>>>> Carter
>>>>>
>>>>>
>>>>> On Jul 20, 2010, at 3:38 PM, Michael Sanderson wrote:
>>>>>
>>>>>> On 07/15/10 05:28 PM, Carter Bullard wrote:
>>>>>>> Sounds like a bug. Do you have a data file that I can check to see
>>>>>>> what's up?
>>>>>>
>>>>>> I was trying to find a way to get a smaller file created, but it
>>>>>> seems that any rewriting generates something that has corrected the
>>>>>> flow.
>>>>>>
>>>>>> The file I have is about 15MB. Where can I drop it? It's a little
>>>>>> large to send via email.
>>>>>>
>>>>>>> raports -r data.2010.07.12-00:00:01.gz - host 10.91.129.8
>>>>>> 142.103.7.9 udp: (1) 815
>>>>>> 10.91.129.8 tcp: (2) 2049, 39813
>>>>>> 10.91.129.8 udp: (1) 769
>>>>>>> racluster -M none -r data.2010.07.12-00:00:01.gz -w data \
>>>>>> - host 10.91.129.8
>>>>>>> raports -r data
>>>>>> 142.103.7.9 udp: (1) 815
>>>>>> 10.91.129.8 tcp: (1) 2049
>>>>>> 10.91.129.8 udp: (1) 769
>>>>>> 142.103.4.102 tcp: (1) 139
>>>>>>
>>>>>> Michael Sanderson
>>>>>>
>>>>>>> Carter
>>>>>>>
>>>>>>>
>>>>>>> On Jul 15, 2010, at 7:11 PM, Michael Sanderson wrote:
>>>>>>>
>>>>>>>> On 07/15/10 02:43 PM, Carter Bullard wrote:
>>>>>>>>> Hey Michael,
>>>>>>>>> I assume that you are using argus-clients-3.0.3.x clients.
>>>>>>>>
>>>>>>>> Yes, 3.0.3.15, not sure when this copy was downloaded.
>>>>>>>>
>>>>>>>>> Argus doesn't make any determination regarding flow source or
>>>>>>>>> destination, this is all done
>>>>>>>>> in the clients.
>>>>>>>>
>>>>>>>> Understood.
>>>>>>>>
>>>>>>>>> If you read data using any of the ra* programs, they have an
>>>>>>>>> opportunity to
>>>>>>>>> 'correct' a flow record for direction, and we do have some logic
>>>>>>>>> for TCP
>>>>>>>>> traffic.
>>>>>>>>> If we haven't seen the syn or the synack, we test the source and
>>>>>>>>> destination ports to
>>>>>>>>> see if the src is in the IPPORT_RESERVED range, 1-1024, and the
>>>>>>>>> dst port
>>>>>>>>> is above it.
>>>>>>>>> If so, if the sport is not ftp-data, and there is an entry in the
>>>>>>>>> /etc/services file for the sport
>>>>>>>>> value, so its an identified service port, we'll reverse the
>>>>>>>>> direction of
>>>>>>>>> the record.
>>>>>>>>>
>>>>>>>>> Does that seem useful, or do you see a problem?
>>>>>>>>
>>>>>>>> Yes, this seems useful and was accurate in this case. The problem
>>>>>>>> is that I don't get the same output when I run raports on data
>>>>>>>> written by 'ra -S ... -w file' and when I run raports on data that
>>>>>>>> has been preprocessed via 'ra -r file -w newfile - host a.b.c.d' or
>>>>>>>> 'racluster -M none -r file -w newfile'. (See original post for the
>>>>>>>> specifics of how things are run and actual output.) It appears that
>>>>>>>> it takes the act of writing the records out to a file to get the
>>>>>>>> direction correction to completely take effect, otherwise the
>>>>>>>> different raports runs would show the same results. Is that accurate?
>>>>>>>>
>>>>>>>> Michael Sanderson
>>>>>>>>
>>>>>>>>> Carter
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Jul 14, 2010, at 4:43 PM, Michael Sanderson wrote:
>>>>>>>>>
>>>>>>>>>> I'm experiencing some slight differences in the results of raports,
>>>>>>>>>> depending upon how I run it.
>>>>>>>>>>
>>>>>>>>>> I rotate my logs every 15 minutes and do some post processing.
>>>>>>>>>>
>>>>>>>>>> 1) raports -r data.2010.07.12-00:* - host 10.91.129.8
>>>>>>>>>> ...
>>>>>>>>>> 10.91.129.8 tcp: (2) 2049, 39813
>>>>>>>>>> 10.91.129.8 udp: (3) 769, 789, 824
>>>>>>>>>> ...
>>>>>>>>>>
>>>>>>>>>> 2) ra -r data.2010.07.12-00:* -w d10.91.129.8 - host 10.91.129.8
>>>>>>>>>> raports -r d10.91.129.8
>>>>>>>>>> ...
>>>>>>>>>> 10.91.129.8 tcp: (1) 2049
>>>>>>>>>> 10.91.129.8 udp: (3) 769, 789, 824
>>>>>>>>>> ...
>>>>>>>>>>
>>>>>>>>>> 3) racluster -r data.2010.07.12-00:* -w clustered.2010.07.12-00
>>>>>>>>>> raports -r clustered.2010.07.12-00 - host 10.91.129.8
>>>>>>>>>> ...
>>>>>>>>>> 10.91.129.8 tcp: (1) 2049
>>>>>>>>>> 10.91.129.8 udp: (3) 769, 789, 824
>>>>>>>>>> ...
>>>>>>>>>>
>>>>>>>>>> raports on the original data files is suggesting that 10.91.129.8
>>>>>>>>>> port
>>>>>>>>>> 39813 is a destination. raports on processed files is suggesting
>>>>>>>>>> it is
>>>>>>>>>> a source. Looking at the flows from the original data files for
>>>>>>>>>> 10.91.129.8 port 39813 shows that direction wasn't determined.
>>>>>>>>>>
>>>>>>>>>> 10/07/11 23:49:23 M tcp 10.91.129.8.39813<?> a.b.c.d.netbio 2 126
>>>>>>>>>> CON
>>>>>>>>>>
>>>>>>>>>> Looking at the processed files (d10.91.129.8 and
>>>>>>>>>> clustered.2010.07.12-00) via ra, they also so that the direction
>>>>>>>>>> was
>>>>>>>>>> not determined.
>>>>>>>>>>
>>>>>>>>>> What is happening with the processed files that convinces raports
>>>>>>>>>> that
>>>>>>>>>> 10.91.129.8 port 39813 is a source, not a destination? In this
>>>>>>>>>> particular case it is right, but I'm curious how it is getting it
>>>>>>>>>> right.
>>>>>>>>>>
>>>>>>>>>> Michael Sanderson
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> Carter Bullard
>>>>>>> CEO/President
>>>>>>> QoSient, LLC
>>>>>>> 150 E 57th Street Suite 12D
>>>>>>> New York, New York 10022
>>>>>>>
>>>>>>> +1 212 588-9133 Phone
>>>>>>> +1 212 588-9134 Fax
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> Carter Bullard
>>>>> CEO/President
>>>>> QoSient, LLC
>>>>> 150 E 57th Street Suite 12D
>>>>> New York, New York 10022
>>>>>
>>>>> +1 212 588-9133 Phone
>>>>> +1 212 588-9134 Fax
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>
> Carter Bullard
> CEO/President
> QoSient, LLC
> 150 E 57th Street Suite 12D
> New York, New York 10022
>
> +1 212 588-9133 Phone
> +1 212 588-9134 Fax
>
>
>
Carter Bullard
CEO/President
QoSient, LLC
150 E 57th Street Suite 12D
New York, New York 10022
+1 212 588-9133 Phone
+1 212 588-9134 Fax
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20100721/2c5af633/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3815 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20100721/2c5af633/attachment.bin>
More information about the argus
mailing list