man records in 3.0

[Peter Van Epp]

On Mon, Sep 22, 2008 at 10:08:18PM -0400, Carter Bullard wrote:
> Hey Peter,
> Just a few questions on these changes.  Not sure why you are preserving
> the 2.x version numbers in the management records since the stream has
> been converted to 3.0.  Is there value in knowing that its history  
> involves
> converting from 2.x to 3.0?

	Yes thats the entire point. The direction flags are different between
2 and 3 and 3.0 clients correctly preserve the 2 values when processing a V2
input file and work as expected (the application being figuring out if a 
scanning host is ours or remote which is suprisingly complex in some scanning
contexts where the remote side is making argus think our host is the one 
scanning). However when fed a 3.0 data file the scans script breaks because
it is expecting 2.0 flags. If the original version number is in the man 
records I can select which set of flag values to use. I suppose is also 
possible there is a better way to do what I'm doing though, but I have this
particular hammer, now to make everything look like a nail :-). 

> 
> We worked hard to get the 3.0 clients to print the 2.x fields using  
> the same
> format, so is there something specific to 2.x output that the 3.0 code  
> is not
> doing?

	Nope, works fine as far as I know, my issue is with the V3 changes.

> 
> When we print mar records, all the fields should have values or  
> blanks, so
> that the format is the same as the far records.  Do we need to specify  
> these
> other fields to have values rather than blanks?

	I don't think so other than to maybe add the initial version number
and perhaps the V2 argusid (neither of which I could find in the clients-3.0
man output). As noted the argusid isn't bothering me because I don't set it
and thus it is garbage in my case but if someone else is using it / needs it
on V2 records it doesn't seem to be in the man records (although I may just 
not have found it). 

> 
> Printing the starting mar is not a problem (need to insert the call to  
> print
> the initial mar in the ArgusReadConnection() routine).   So if the
> ARGUS_PRINT_MAR variable is set in the .rarc, then we'll print the  
> starting
> mar record?
> 
	That would be good as then from the first record I know if the file
is V2 or V3. While I could as suggested modify the magics file (although that
didn't immediately work on freeBSD probably because I did the wrong thing to
get it the magic file, I expect it would) and use file to decide what kind of
file it is, but since I'm already reading records with ra if it is in the man
records it comes for free.

> I'll wait to upload the new code until we hash some these out.
> Hope all is most excellent,
> 
> Carter
> 

	As I recall (its been a month or so since I ran the last test at which
point a V2 file being read by ra3 worked fine but the same file captured by
a v3 sensor on the regen tap was radically different in the scan processing,
the traffic counts were as expected close to identical). I expect it is 
tripping in this bit of perl which tries to correct the source and dest (and
perhaps just needs to be jumpered out for V3 records) directions if it sees
a service port such as 80 listed as the source (many of the scans cause this
to happen meaning the flow doesn't get correctly classified as a scan as the
"dest" port is always changing 

	# If argus looks to have got source and dst inverted,
	# uninvert them.

	if ((($src_port < 1024) && ($dst_port > 1023)) || 
	    (($mid_flag eq '<?') && ($end_flag eq "TIM")) ||
	    (($mid_flag eq '<?>') && ($end_flag eq "TIM")) ||
            (($mid_flag eq '?>') && ($end_flag eq "R"))) { 

		# invert src and dst cause its probably wrong!

                $tmp_ip    = $src_ip;
                $tmp_port  = $src_port;
                $tmp_cnt   = $src_bytes;
                $src_ip    = $dst_ip;
                $src_port  = $dst_port;
                $src_bytes = $dst_bytes;
                $dst_ip    = $tmp_ip;
                $dst_port  = $tmp_port;
                $dst_bytes = $tmp_cnt;
        }

	This is of course V2 specific and as noted I don't know if V3 is 
getting it more correct. An example of v2 output that needs this:

23 Sep 08 07:02:14           tcp   142.58.200.35.80     ?>     74.6.18.220.60999 2        2         3028         120         RST
23 Sep 08 07:02:15           tcp   142.58.200.35.80     ?>     74.6.18.220.2969  2        2         3028         120         RST
23 Sep 08 07:02:15           tcp   142.58.200.35.80     ?>     74.6.18.220.40185 2        2         3028         120         RST

	For scan detection purposes this needs to be inverted (for proper 
traffic classification too for that matter but I think that works since 
traffic as I recall looked pretty right which is why I think just removing
this on V3 may work) so that 80 shows as the destination port instead of
source. There are a variety of half open scan and forged address backlash 
(where we only see the reply to a message forged from somewhere else) that
are simialar. 

Peter Van Epp / Operations and Technical Support 
Simon Fraser University, Burnaby, B.C. Canada