EXTERNAL: Re: Raconvert does not produce the same binary output file

Mon Oct 12 12:31:07 EDT 2015

Hey John,
So this is now a bug, so I’ll look into it.  To be clear, your interested in the encapsulation flag being consistent when using raconvert.1 and you need the INT and REG to also agree.

Does that capture it ???
Carter

> On Oct 12, 2015, at 12:27 PM, Ngo, John W <john.w.ngo at lmco.com> wrote:
> 
> Hi Carter,
>  
> I hope this message finds you well.  I just wanted to touch base with you to see if you had a chance to work on that INT to REQ conversion issue.
>  
> As for the asterisk issue, I added the ‘senc’ and ‘denc’ fields to my config as you had suggested. However, the * flags are still not being converted and are showing up as ‘e’.  
>  
> So I printed out all 174 fields.  Attached are the results showing all possible fields.  I was hoping that by printing out all available fields, it would convey enough information for the converter to determine if the flag should be an asterisk.  Unfortunately, printing all fields did not fix the issue.
>  
> Is there something else I could be missing?
>  
> Thanks,
> John
>  
> From: Carter Bullard [mailto:carter at qosient.com] 
> Sent: Friday, October 02, 2015 4:06 PM
> To: Ngo, John W (US) <john.w.ngo at lmco.com>
> Cc: Argus <argus-info at lists.andrew.cmu.edu>
> Subject: EXTERNAL: Re: [ARGUS] Raconvert does not produce the same binary output file
>  
> Hey John,
> Great !!!  OK, the truth is that INT and REQ are equivalents.  REQ is specific to certain protocols, so I can fix that.
> The asterisks issue may not be solvable with the limited set of fields that you’re printing, and I’m not sure what fields will fix it.
> We have the ’senc’ and ‘denc'  fields, which should do it, but I’m not 100% confident.
>  
> I’ll be able to test it later, if you could add these fields to your test, and if it works, that would be most excellent,
> Carter
>  
> On Oct 2, 2015, at 3:06 PM, Ngo, John W <john.w.ngo at lmco.com <mailto:john.w.ngo at lmco.com>> wrote:
>  
> Carter,
>  
> I’m sorry, my mistake!  Earlier I was only printing out the total packets and total bytes.
>  
> I re-ran it again using my ra config file (rarc file) print out the source and destination packets and bytes (not the total) and I’m getting much better results. With very few differences.  I’ve attached it for your comparison.
>  
> Basically this solves all the direction issues.  Including all the CON conversion issues are resolved. 
>  
> The only issue that now remains is the asterisk issue and that all INT’s are converted to REQ’s.
>  
> John
>  
> From: Ngo, John W 
> Sent: Friday, October 02, 2015 2:45 PM
> To: 'Carter Bullard' <carter at qosient.com <mailto:carter at qosient.com>>
> Cc: Argus <argus-info at lists.andrew.cmu.edu <mailto:argus-info at lists.andrew.cmu.edu>>
> Subject: RE: EXTERNAL: Re: [ARGUS] Raconvert does not produce the same binary output file
>  
> Carter,
> 
> Thanks for looking into the issue.  It could be possible I’m not printing all the fields.  Here are the current fields I am printing:
>  
> stime flgs proto saddr sport dir daddr dport spkts dpkts sbytes dbytes state
>  
> In short, all I’m really doing is just printing out the default fields given to me when running the ra –r command.  I know there are a ton of other fields I can include.
>  
> Do you have a suggestion as to what other fields I should be printing out in order to properly convert back to *?
>  
> Thanks,
> John
>  
> From: Carter Bullard [mailto:carter at qosient.com <mailto:carter at qosient.com>] 
> Sent: Friday, October 02, 2015 2:37 PM
> To: Ngo, John W (US) <john.w.ngo at lmco.com <mailto:john.w.ngo at lmco.com>>
> Cc: Argus <argus-info at lists.andrew.cmu.edu <mailto:argus-info at lists.andrew.cmu.edu>>
> Subject: EXTERNAL: Re: [ARGUS] Raconvert does not produce the same binary output file
>  
> John,
> Thanks, I’ll look into it.  It is possible that you are not printing enough fields to convey all the information needed to get the printouts to be the same.
> The Flgs fields have a massive amount of information that is summarized.  the “*” and the “e” as an example, these are indications of encapsulation types.  Are you printing out all the fields needed to preserve all the encapsulation information needed to retain the “*” ???
>  
> If you are using total packets and total bytes in the conversion, then you aren’t retaining enough information to preserve the direction arrows or the state information.
>  
> So, …, by printing out only 8-10 of the 150 fields we have, you will get a data minimizing effect, and as a result you won’t get identical output.
> To preserve certain fields, you may need to add a few fields.  What fields are you using ????
>  
> Carter
>  
> On Oct 2, 2015, at 2:22 PM, Ngo, John W <john.w.ngo at lmco.com <mailto:john.w.ngo at lmco.com>> wrote:
>  
> Hi Carter,
>  
> Thanks a bunch for working on the issue!  It is very much appreciated!!
>  
> So the good news is that I’m no longer getting that strtol error when I run raconvert.
>  
> The bad news is that I’m still seeing discrepancies in the data.
>  
> Attached are the results using the new raconvert.c file you had sent to me.  It ran against the same argus binary file I had sent to you last time. Only the filenames have been shortened for simplicity.  Here are the steps I have used to generate the files:
>  
> 1.)    Starting with the Argus binary archive, convert to Netflow:
> ra -r argus.2015-09-14.gz -zc, > argus.2015-09-14.ra
>  
> 2.)    Convert Netflow to Binary
> raconvert -r argus.2015-09-14.ra -w argus.2015-09-14_derived.gz
>  
> 3.)    Convert Binary back to Netflow
> ra -r argus.2015-09-14_derived.gz -zc, > argus.2015-09-14_derived.ra
>  
> 4.)    Diff argus.2015-09-14.ra  and argus.2015-09-14_derived.ra
>  
> I made sure to use the “-z” option to print the TCP state machine to preserve the state this time.  Unfortunately I am seeing the following differences as shown in my screenshots below:
>  
> -          The direction appears to be off.  Some <?> are missing the first ‘<’ upon conversion.  Sometimes it should be bi-directional, but it gets converted to uni-directional.
> -          The state CON is converted to REQ.  Sometimes it is converted INT.
> -          All asterisk * flags are converted to e.
>  
> <image001.png>
>  
> <image002.png>
>  
> Please let me know if I am missing something.
>  
> Thanks!
> John
>  
> From: Carter Bullard [mailto:carter at qosient.com <mailto:carter at qosient.com>] 
> Sent: Monday, September 28, 2015 10:43 PM
> To: Ngo, John W <john.w.ngo at lmco.com <mailto:john.w.ngo at lmco.com>>
> Cc: Argus <argus-info at lists.andrew.cmu.edu <mailto:argus-info at lists.andrew.cmu.edu>>
> Subject: EXTERNAL: Re: [ARGUS] Raconvert does not produce the same binary output file
>  
> Hey John,
> Here is a raconvert.c that should be better than what you’ve experienced.  Its fixes the errors and generates consistent data.
> In order to get the data to behave, especially TCP traffic, you need to provide raconvert.1 enough information to preserve the state that you want.  The way to preserve the state field and the direction, is to use the “-z” option when creating the ascii csv file.
> This will print the TCP state machine, rather than SYN, CON, FIN, CLO.  This data is needed to seed the conversion algorithm with enough info to set the direction and the TCP state.
>  
> So this works for me with your data … we use racount.1 to test simple integrity of the data:
>   racount -r argus.file
>   ra -r argus.file -zc, | raconvert -c, -w - | racount
>   ra -r argus.file -zc, | raconvert -c, -w - | ra -zc, | raconvert -c, w - | racount
>  
> These seem to generate the same data using your data.
> Give this new raconvert.c a run, and give us a thumbs up or down !!!
>  
> Carter
>  
> <argus.2015-09-14.ra><argus.2015-09-14_derived.ra>
>  
> <argus.2015-09-14.ra><argus.2015-09-14_derived.ra>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20151012/768f664e/attachment.html>