man records in 3.0
Carter Bullard
carter at qosient.com
Tue Sep 23 13:51:42 EDT 2008
Hey Peter,
I think the most important thing is to make the changes so the script is
logical and intelligible. So, if we are having direction problems, we
really should fix them. Between v2 and v3, I took out the cleverness
that we had to 'correct' the direction, as it was slightly flawed (scans
especially), and it hid the "ground truth" of the packets on the wire.
The direction had been a complex set of conditions in the probe
and in the client reading the data stream, so the work to clean that
up should have worked. If not, definitely need to fix it.
Adding support for printing missing fields is always high on the list.
Regarding the argusid of the v3 records, printing the 'srcid' of a v3
man record doesn't print the argusid. Definitely need to fix that.
OK, so this thread has a number of changes to consider, could you
give me a brief list of what you need to get the ball rolling? I've
got print initial mar record with the version string in the 'dir' field
as the #1. Anything next?
Carter
On Sep 23, 2008, at 11:38 AM, Peter Van Epp wrote:
> On Mon, Sep 22, 2008 at 10:08:18PM -0400, Carter Bullard wrote:
>> Hey Peter,
>> Just a few questions on these changes. Not sure why you are
>> preserving
>> the 2.x version numbers in the management records since the stream
>> has
>> been converted to 3.0. Is there value in knowing that its history
>> involves
>> converting from 2.x to 3.0?
>
> Yes thats the entire point. The direction flags are different between
> 2 and 3 and 3.0 clients correctly preserve the 2 values when
> processing a V2
> input file and work as expected (the application being figuring out
> if a
> scanning host is ours or remote which is suprisingly complex in some
> scanning
> contexts where the remote side is making argus think our host is the
> one
> scanning). However when fed a 3.0 data file the scans script breaks
> because
> it is expecting 2.0 flags. If the original version number is in the
> man
> records I can select which set of flag values to use. I suppose is
> also
> possible there is a better way to do what I'm doing though, but I
> have this
> particular hammer, now to make everything look like a nail :-).
>
>>
>> We worked hard to get the 3.0 clients to print the 2.x fields using
>> the same
>> format, so is there something specific to 2.x output that the 3.0
>> code
>> is not
>> doing?
>
> Nope, works fine as far as I know, my issue is with the V3 changes.
>
>>
>> When we print mar records, all the fields should have values or
>> blanks, so
>> that the format is the same as the far records. Do we need to
>> specify
>> these
>> other fields to have values rather than blanks?
>
> I don't think so other than to maybe add the initial version number
> and perhaps the V2 argusid (neither of which I could find in the
> clients-3.0
> man output). As noted the argusid isn't bothering me because I don't
> set it
> and thus it is garbage in my case but if someone else is using it /
> needs it
> on V2 records it doesn't seem to be in the man records (although I
> may just
> not have found it).
>
>>
>> Printing the starting mar is not a problem (need to insert the call
>> to
>> print
>> the initial mar in the ArgusReadConnection() routine). So if the
>> ARGUS_PRINT_MAR variable is set in the .rarc, then we'll print the
>> starting
>> mar record?
>>
> That would be good as then from the first record I know if the file
> is V2 or V3. While I could as suggested modify the magics file
> (although that
> didn't immediately work on freeBSD probably because I did the wrong
> thing to
> get it the magic file, I expect it would) and use file to decide
> what kind of
> file it is, but since I'm already reading records with ra if it is
> in the man
> records it comes for free.
>
>> I'll wait to upload the new code until we hash some these out.
>> Hope all is most excellent,
>>
>> Carter
>>
>
> As I recall (its been a month or so since I ran the last test at
> which
> point a V2 file being read by ra3 worked fine but the same file
> captured by
> a v3 sensor on the regen tap was radically different in the scan
> processing,
> the traffic counts were as expected close to identical). I expect it
> is
> tripping in this bit of perl which tries to correct the source and
> dest (and
> perhaps just needs to be jumpered out for V3 records) directions if
> it sees
> a service port such as 80 listed as the source (many of the scans
> cause this
> to happen meaning the flow doesn't get correctly classified as a
> scan as the
> "dest" port is always changing
>
> # If argus looks to have got source and dst inverted,
> # uninvert them.
>
> if ((($src_port < 1024) && ($dst_port > 1023)) ||
> (($mid_flag eq '<?') && ($end_flag eq "TIM")) ||
> (($mid_flag eq '<?>') && ($end_flag eq "TIM")) ||
> (($mid_flag eq '?>') && ($end_flag eq "R"))) {
>
> # invert src and dst cause its probably wrong!
>
> $tmp_ip = $src_ip;
> $tmp_port = $src_port;
> $tmp_cnt = $src_bytes;
> $src_ip = $dst_ip;
> $src_port = $dst_port;
> $src_bytes = $dst_bytes;
> $dst_ip = $tmp_ip;
> $dst_port = $tmp_port;
> $dst_bytes = $tmp_cnt;
> }
>
> This is of course V2 specific and as noted I don't know if V3 is
> getting it more correct. An example of v2 output that needs this:
>
> 23 Sep 08 07:02:14 tcp 142.58.200.35.80 ?>
> 74.6.18.220.60999 2 2 3028 120 RST
> 23 Sep 08 07:02:15 tcp 142.58.200.35.80 ?>
> 74.6.18.220.2969 2 2 3028 120 RST
> 23 Sep 08 07:02:15 tcp 142.58.200.35.80 ?>
> 74.6.18.220.40185 2 2 3028 120 RST
>
> For scan detection purposes this needs to be inverted (for proper
> traffic classification too for that matter but I think that works
> since
> traffic as I recall looked pretty right which is why I think just
> removing
> this on V3 may work) so that 80 shows as the destination port
> instead of
> source. There are a variety of half open scan and forged address
> backlash
> (where we only see the reply to a message forged from somewhere
> else) that
> are simialar.
>
> Peter Van Epp / Operations and Technical Support
> Simon Fraser University, Burnaby, B.C. Canada
>
More information about the argus
mailing list