Raconvert does not produce the same binary output file

Mon Sep 28 22:42:36 EDT 2015

Hey John,
Here is a raconvert.c that should be better than what you’ve experienced.  Its fixes the errors and generates consistent data.
In order to get the data to behave, especially TCP traffic, you need to provide raconvert.1 enough information to preserve the state that you want.  The way to preserve the state field and the direction, is to use the “-z” option when creating the ascii csv file.
This will print the TCP state machine, rather than SYN, CON, FIN, CLO.  This data is needed to seed the conversion algorithm with enough info to set the direction and the TCP state.

So this works for me with your data … we use racount.1 to test simple integrity of the data:
  racount -r argus.file
  ra -r argus.file -zc, | raconvert -c, -w - | racount
  ra -r argus.file -zc, | raconvert -c, -w - | ra -zc, | raconvert -c, w - | racount

These seem to generate the same data using your data.
Give this new raconvert.c a run, and give us a thumbs up or down !!!

Carter

> On Sep 28, 2015, at 6:29 PM, Carter Bullard <carter at qosient.com> wrote:
> 
> Hey John,
> Yes, I’ve made some good progress !!!  I found and fixed the bug that caused the error regarding strtol.
> 
> The primary bug I’m seeing is that man records are not getting processed correctly, and TCP connections are getting messed up in  their direction.  This is logical, but not quite right, as you are not printing all the information needed to absolutely know what the direction is suppose to be.  Some records are getting reversed (which is allowed for TCP when the direction is unknown) because of this, but I’m somewhat disappointed that the TCP state is getting munged.
> 
> I’m working this now, and should have a solution today.
> I can send you a copy of the new raconvert.c when we get all the issues resolved.
> 
> Carter
> 
>> On Sep 28, 2015, at 1:36 PM, Ngo, John W <john.w.ngo at lmco.com <mailto:john.w.ngo at lmco.com>> wrote:
>> 
>> Hi Carter,
>>  
>> I wanted to follow up to see if you had the chance to look into this issue yet?
>>  
>> I’m re-attaching the files you requested in case you didn’t receive it the first time.  
>> (As a reminder, the .allow extension needs to be removed when saving the file.)
>>  
>> Please let me know if you need additional information. 
>>  
>> Thanks,
>> John
>>  
>> From: Ngo, John W 
>> Sent: Wednesday, September 16, 2015 4:12 PM
>> To: 'Carter Bullard' <carter at qosient.com <mailto:carter at qosient.com>>
>> Cc: argus-info at lists.andrew.cmu.edu <mailto:argus-info at lists.andrew.cmu.edu>
>> Subject: Re: [ARGUS] Raconvert does not produce the same binary output file
>>  
>> Hi Carter,
>>  
>> Attached is are the files that demonstrate the error.  You’ll need to remove the allow extension since my company’s firewall blocks zip extensions.  Here’s how I’ve generated the included files. 
>>  
>> 1.)    Starting with the Argus binary archive, convert to Netflow:
>> ra -r argus.2015-09-14T19:20:01Z.colsv2500808.gz -F /etc/logsearch/ra.conf &> argus.2015-09-14T19:20:01Z.colsv2500808.netflow
>>  
>> 2.)    Convert Netflow to Binary
>> raconvert -r argus.2015-09-14T19:20:01Z.colsv2500808.netflow -w argus.2015-09-14T19:20:01Z.colsv2500808_derived.gz
>>  
>> 3.)    Convert Binary back to Netflow
>> ra -r argus.2015-09-14T19:20:01Z.colsv2500808_derived.gz -F /etc/logsearch/ra.conf &> argus.2015-09-14T19:20:01Z.colsv2500808_derived.netflow
>>  
>> If you compare the original netflow against the derived netflow using a diff tool like WinMerge, you’ll see differences in the flags, direction, and state field.  Also for some reason, I had to move the ordering of the state field after destination port (dport).  If make state the very last column, I get the following error:
>>  
>> raconvert[58788]: 14:07:08.327692 ArgusParseSrcPacketsLabel(0x2cee9010s, CON) strtol error Success
>>  
>> Thanks again for your help. It is much appreciated.
>>  
>> John
>>  
>> From: Carter Bullard [mailto:carter at qosient.com <mailto:carter at qosient.com>] 
>> Sent: Wednesday, September 16, 2015 9:00 AM
>> To: Ngo, John W <john.w.ngo at lmco.com <mailto:john.w.ngo at lmco.com>>
>> Cc: argus-info at lists.andrew.cmu.edu <mailto:argus-info at lists.andrew.cmu.edu>
>> Subject: Re: [ARGUS] Raconvert does not produce the same binary output file
>>  
>> Hey John,
>> That's a pretty good test of conversion !!!
>> Any idea as to what stage the discrepancies appear in ??
>> If you can share a file that can demonstrate the errors, with the command line to recreate it, I'll take a look at it !!
>> Carter
>> 
>> On Sep 15, 2015, at 9:48 PM, Ngo, John W <john.w.ngo at lmco.com <mailto:john.w.ngo at lmco.com>> wrote:
>> 
>> Greetings.  I’m currently having issues with the raconvert tool.  What I’m trying to do is convert an Argus binary file to a Netflow using the ra command. Then I am using raconvert to turn it back to binary, and then use the ra tool on the derived binary file to produce a second netflow file.  I’m comparing both netflow files and I’m noticing significant differences between the two.  Some netflow events appear to match, however most are off by a few flags.  Has anyone tried this and noticed these discrepancies using the raconvert command?  Is there a particular configuration I should be using when generating the first netflow file?
>>  
>>  
>> <argus_files.zip.allow>
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20150928/dc84428f/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: raconvert.c
Type: application/octet-stream
Size: 90294 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20150928/dc84428f/attachment.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20150928/dc84428f/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20150928/dc84428f/attachment.bin>