Raconvert does not produce the same binary output file

[Ngo, John W]

Oops. Here’s the attachment that was supposed to go in the last e-mail.

John

From: Ngo, John W
Sent: Friday, October 02, 2015 2:23 PM
To: 'Carter Bullard' <carter at qosient.com>
Cc: Argus <argus-info at lists.andrew.cmu.edu>
Subject: Re: [ARGUS] Raconvert does not produce the same binary output file

Hi Carter,

Thanks a bunch for working on the issue!  It is very much appreciated!!

So the good news is that I’m no longer getting that strtol error when I run raconvert.

The bad news is that I’m still seeing discrepancies in the data.

Attached are the results using the new raconvert.c file you had sent to me.  It ran against the same argus binary file I had sent to you last time. Only the filenames have been shortened for simplicity.  Here are the steps I have used to generate the files:


1.)    Starting with the Argus binary archive, convert to Netflow:
ra -r argus.2015-09-14.gz -zc, > argus.2015-09-14.ra


2.)    Convert Netflow to Binary
raconvert -r argus.2015-09-14.ra -w argus.2015-09-14_derived.gz


3.)    Convert Binary back to Netflow

ra -r argus.2015-09-14_derived.gz -zc, > argus.2015-09-14_derived.ra



4.)    Diff argus.2015-09-14.ra  and argus.2015-09-14_derived.ra

I made sure to use the “-z” option to print the TCP state machine to preserve the state this time.  Unfortunately I am seeing the following differences as shown in my screenshots below:


-          The direction appears to be off.  Some <?> are missing the first ‘<’ upon conversion.  Sometimes it should be bi-directional, but it gets converted to uni-directional.

-          The state CON is converted to REQ.  Sometimes it is converted INT.

-          All asterisk * flags are converted to e.

[cid:image001.png at 01D0FD1E.21378FC0]

[cid:image002.png at 01D0FD1E.21378FC0]

Please let me know if I am missing something.

Thanks!
John

From: Carter Bullard [mailto:carter at qosient.com]
Sent: Monday, September 28, 2015 10:43 PM
To: Ngo, John W <john.w.ngo at lmco.com<mailto:john.w.ngo at lmco.com>>
Cc: Argus <argus-info at lists.andrew.cmu.edu<mailto:argus-info at lists.andrew.cmu.edu>>
Subject: EXTERNAL: Re: [ARGUS] Raconvert does not produce the same binary output file

Hey John,
Here is a raconvert.c that should be better than what you’ve experienced.  Its fixes the errors and generates consistent data.
In order to get the data to behave, especially TCP traffic, you need to provide raconvert.1 enough information to preserve the state that you want.  The way to preserve the state field and the direction, is to use the “-z” option when creating the ascii csv file.
This will print the TCP state machine, rather than SYN, CON, FIN, CLO.  This data is needed to seed the conversion algorithm with enough info to set the direction and the TCP state.

So this works for me with your data … we use racount.1 to test simple integrity of the data:
  racount -r argus.file
  ra -r argus.file -zc, | raconvert -c, -w - | racount
  ra -r argus.file -zc, | raconvert -c, -w - | ra -zc, | raconvert -c, w - | racount

These seem to generate the same data using your data.
Give this new raconvert.c a run, and give us a thumbs up or down !!!

Carter

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20151002/d41f6631/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 73935 bytes
Desc: image001.png
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20151002/d41f6631/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 44641 bytes
Desc: image002.png
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20151002/d41f6631/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: argus.2015-09-14.zip.allow
Type: application/octet-stream
Size: 128377 bytes
Desc: argus.2015-09-14.zip.allow
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20151002/d41f6631/attachment.obj>