Raconvert does not produce the same binary output file

[Carter Bullard]

Hey John,
Yes, I’ve made some good progress !!!  I found and fixed the bug that caused the error regarding strtol.

The primary bug I’m seeing is that man records are not getting processed correctly, and TCP connections are getting messed up in  their direction.  This is logical, but not quite right, as you are not printing all the information needed to absolutely know what the direction is suppose to be.  Some records are getting reversed (which is allowed for TCP when the direction is unknown) because of this, but I’m somewhat disappointed that the TCP state is getting munged.

I’m working this now, and should have a solution today.
I can send you a copy of the new raconvert.c when we get all the issues resolved.

Carter

> On Sep 28, 2015, at 1:36 PM, Ngo, John W <john.w.ngo at lmco.com> wrote:
> 
> Hi Carter,
>  
> I wanted to follow up to see if you had the chance to look into this issue yet?
>  
> I’m re-attaching the files you requested in case you didn’t receive it the first time.  
> (As a reminder, the .allow extension needs to be removed when saving the file.)
>  
> Please let me know if you need additional information. 
>  
> Thanks,
> John
>  
> From: Ngo, John W 
> Sent: Wednesday, September 16, 2015 4:12 PM
> To: 'Carter Bullard' <carter at qosient.com <mailto:carter at qosient.com>>
> Cc: argus-info at lists.andrew.cmu.edu <mailto:argus-info at lists.andrew.cmu.edu>
> Subject: Re: [ARGUS] Raconvert does not produce the same binary output file
>  
> Hi Carter,
>  
> Attached is are the files that demonstrate the error.  You’ll need to remove the allow extension since my company’s firewall blocks zip extensions.  Here’s how I’ve generated the included files. 
>  
> 1.)    Starting with the Argus binary archive, convert to Netflow:
> ra -r argus.2015-09-14T19:20:01Z.colsv2500808.gz -F /etc/logsearch/ra.conf &> argus.2015-09-14T19:20:01Z.colsv2500808.netflow
>  
> 2.)    Convert Netflow to Binary
> raconvert -r argus.2015-09-14T19:20:01Z.colsv2500808.netflow -w argus.2015-09-14T19:20:01Z.colsv2500808_derived.gz
>  
> 3.)    Convert Binary back to Netflow
> ra -r argus.2015-09-14T19:20:01Z.colsv2500808_derived.gz -F /etc/logsearch/ra.conf &> argus.2015-09-14T19:20:01Z.colsv2500808_derived.netflow
>  
> If you compare the original netflow against the derived netflow using a diff tool like WinMerge, you’ll see differences in the flags, direction, and state field.  Also for some reason, I had to move the ordering of the state field after destination port (dport).  If make state the very last column, I get the following error:
>  
> raconvert[58788]: 14:07:08.327692 ArgusParseSrcPacketsLabel(0x2cee9010s, CON) strtol error Success
>  
> Thanks again for your help. It is much appreciated.
>  
> John
>  
> From: Carter Bullard [mailto:carter at qosient.com <mailto:carter at qosient.com>] 
> Sent: Wednesday, September 16, 2015 9:00 AM
> To: Ngo, John W <john.w.ngo at lmco.com <mailto:john.w.ngo at lmco.com>>
> Cc: argus-info at lists.andrew.cmu.edu <mailto:argus-info at lists.andrew.cmu.edu>
> Subject: Re: [ARGUS] Raconvert does not produce the same binary output file
>  
> Hey John,
> That's a pretty good test of conversion !!!
> Any idea as to what stage the discrepancies appear in ??
> If you can share a file that can demonstrate the errors, with the command line to recreate it, I'll take a look at it !!
> Carter
> 
> On Sep 15, 2015, at 9:48 PM, Ngo, John W <john.w.ngo at lmco.com <mailto:john.w.ngo at lmco.com>> wrote:
> 
> Greetings.  I’m currently having issues with the raconvert tool.  What I’m trying to do is convert an Argus binary file to a Netflow using the ra command. Then I am using raconvert to turn it back to binary, and then use the ra tool on the derived binary file to produce a second netflow file.  I’m comparing both netflow files and I’m noticing significant differences between the two.  Some netflow events appear to match, however most are off by a few flags.  Has anyone tried this and noticed these discrepancies using the raconvert command?  Is there a particular configuration I should be using when generating the first netflow file?
>  
>  
> <argus_files.zip.allow>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20150928/44c96036/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20150928/44c96036/attachment.bin>