new rapolicy code
Carter Bullard
carter at qosient.com
Mon Mar 18 11:05:33 EDT 2013
OK Dave,
You are probably going to end up hating me, but I'm starting to make some
changes to incorporate the new rapolicy code that you've developed.
I'm going to add to the ArgusParser a single 32 bit int for the policy stuff.
Definitely, features that other clients may want to use, so a variable that all
client programs will see is a good thing. So far, I've called it RaPolicyStatus. Since
you had 4 ints, holding 4 bits of semantics, I'm going to give policy 31 status
bits, and we'll use constants as bit indicators. I'm trying to shrink the parser now,
as its a bit too big for some L2 caches and good performance, so hopefully you
won't think its too much of an imposition.
Working the port now, I'll send lots of info your way today.
Carter
On Mar 17, 2013, at 5:41 PM, "Dave Edelman" <dedelman at iname.com> wrote:
> Carter,
>
> Thanks for the pointers, I changed the code to copy, process, delete and it seems to be working as advertised. I’ve sent a tarfile based on 3.7.0.6 separately, please feel free to make any changes.
>
> I am still testing this code but it does seem stable and functional. The precedence, IP options, and DSCP processing should be next along with the formal test material. I’d be very interested to hear of any ACLs that it doesn’t parse or flows that seem to be mishandled. If you build it with debug enabled then –D 3 will tell you more than you ever wanted to know.
>
> The one significant thing to understand is that this is not an ACL syntax verification system. It will happily accept TCP flags parameters on a UDP entry. I recommend that you feed it ACLs that have been output by “show running” which ensures that the syntax is acceptable to IOS. The ACL can be standard or extended, numbered or named, with or without sequence numbers.
>
> --Dave
>
>
> From: Carter Bullard [mailto:carter at qosient.com]
> Sent: Sunday, March 17, 2013 1:48 PM
> To: Dave Edelman
> Subject: Re: Possible argus_label Memory Leak
>
> So, you need to understand that the record being passed to RaProcessRecord,
> is a static record, that will be over-written on each subsequent call to RaProcessRecord.
> None of the buffers have been malloc'd, they are all in the ArgusParserStruct single
> canonical argus record.
>
> If you want to work with the record, and use it later, you need to copy the record,
> using ArgusCopyRecord(). After that all the DSRs buffers etc have been allocated,
> and you can do with them as you please.
>
> Remember, you can run argus or radium() in real time mode, so that it plays back
> the data as if it were in real time. Good for testing algorithms that maybe time
> dependent.
>
> Do I have the latest rapolicy tarfile? I'll put it in today. I may need to modify a
> bunch of it, but so far, it won't be too bad. I just need to make sure that its thread
> safe, etc....
>
> Carter
>
>
> On Mar 17, 2013, at 1:13 PM, "Dave Edelman" <dedelman at iname.com> wrote:
>
>
> Carter,
>
> Sorry, I see where it is being released. I started looking when I ran into a problem with rapolicy. I decided that (subject to .rarc parameters) to attach a label to any flow that was permitted or denied indicating the permit/deny decision, the name (or number) of the ACL and the line that caused the action. The rc setting can prevent it, or enable it for all flows or just the flows that have the “log” parameter.
>
> I stole the code from ArgusAddToRecordLabel() and it consistently segfaults on the free() or if I comment out the free() it segfaults on the ArgusFree(). If I comment them both out (I’m running on a system with 64GB so I can run for a while) it actually works quite nicely. I’ll attempt to change the code to just call the ArgusAddToRecordLabel() which is probably what I should have done in the first place.
>
> I have the pcap files from the 2009 West Point inter-service cyber simulation. I am about to run them through Argus -> Radium with labeling enabled so if there is a memory leak I expect to find out about it within the first few gig of pcaps. I’ll keep you informed
>
> I am testing rapolicy with a pretty simple setup. I have argus running and providing data on port 1000, radium (with labels enabled) connects to argus and provides data on port 561. I have an instance of ra reading the radium data as well as two instances of rapolicy reading the same data. The first copy of rapolicy has a resource file with a parameter set to only show the traffic that matches a deny. The second is set to only show the matches for a permit. I also have added a resource parameter that allows traffic that isn’t subject to the access list eg: arp and link-level stuff to pass through rapolicy or to be suppressed. With rapolicy set to label the flows I can compare the outputs and see what is happening. I also have added some code at the exit handler that dumps a table of each ACL line with the number of flows, packets, and bytes that matched. When I turn this over, I’ll provide the test material and results as well. I am usingnping and ostinato to create the test flows.
>
> I have all the ACL parsing tested and I haven’t found any problems. I have IPv4 address and port matching working and well tested. TOS and TCP flags are working but need more testing. ICMP is working and well tested. IGMP is up for testing now. I am looking at the DSCP code point handling, I expect that the six bits are in the same DSR as the TOS value. IP options are something that you may not like. I parse them and ignore them. I hate to say this but the other guys may have a point with their bit vector representation of the option values so that it is possible to handle the possibility of several being set. This can only get worse with IPv6.
>
> .
>
> --Dave
>
>
>
> From: Carter Bullard [mailto:carter at qosient.com]
> Sent: Sunday, March 17, 2013 11:38 AM
> To: David Edelman
> Subject: Re: Possible argus_label Memory Leak
>
> BIG changes to argus_label.c, for the last set of bugs, so.... there maybe a new leak.
> Hmmm, no, looks good to me. The allocated buffer that holds the old label is assigned
> to lstr, an new buffer is allocated for the new label, and lstr is free()'d at the
> end of the block.
>
> Maybe a leak somewhere else?
> Carter
>
> On Mar 17, 2013, at 10:30 AM, David Edelman <dedelman at iname.com> wrote:
>
>
>
> Carter,
>
> I've moved to the new code base and things look good for rapolicy.
>
> I'm confused about argus_label.c I think that there is a memory leak in ArgusLabelRecord after the label is assembled if there is an existing label on the flow. The else clause to if(lstruct==NULL) does a calloc for the replacement l_un.label but I don't see where the old buffer is released. I'm comparing this to similar code in ArgusAddToRecordLabel which seems to take no prisoners killing off the entire DSR with both a free and an ArgusFree am I missing something?
>
> Dave Edelman
>
>
> On Mar 15, 2013, at 11:24, Carter Bullard <carter at qosient.com> wrote:
>
> Hey Dave,
> The new argus-clients-3.0.7.6 does not have your code in it, as I been plagued
> trying to fix a complex bug. I'll put your stuff in 3.0.7.7.
> Sorry to be a bit behind,
>
> Carter
>
> On Mar 13, 2013, at 1:02 PM, "Dave Edelman" <dedelman at iname.com> wrote:
>
>
>
> Carter,
>
> It sounds like you are feeling a bit better, I certainly hope so.
>
> I have found a way to rigorously test the ICMP filtering and the remaining anomaly is with the reply types. I have no problem with an echo request but the echo reply just like the timestamp reply, information reply, address mask reply don’t seem to make it to rapolicy in a way that I expect to see them. It might be some manipulation of source as compared to destination and the sense of the flow. At one point when I was watching a real ping interchange, I saw echo-request / echo-reply pairs and rapolicy treated them as a set of unified flows. I saw five flows with ten packets and the correct number of bytes. On the other hand, it I send an echo-request to a non-existent address, I see the echo-request, but if I send an echo-reply to a non-existent address I see nothing. If could be the flow modeler enforcing reality.
>
> I now have several tools in a test configuration and the one that seems to work best is nping sending traffic to a non-existent host on a subnet that consists of my second Ethernet interface and a switch just to provide link. I have argus watching the interface in promiscuous mode and radium collecting the data from argus. I have rapolicy and ra taking data from radium. Ostinato is a very nice tool but not stable in my environment (FC18.) If you are willing to put it out as a work in progress, I’ll provide the most recent material. If you would prefer to wait, I’ll keep on working on validation. Please let me know which you prefer. I hate filling up your email with attachments.
>
> By the way, I’m pretty sure that I saw the label problem that you are discussing but I don’t remember how I worked around it. I have some fuzzy memories of making the other label material lowercase but that was a while ago and I’ve been in DDoSHeck since December.
>
> --Dave
>
> From: argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu [mailto:argus-info-bounces+dedelman=iname.com at lists.andrew.cmu.edu] On Behalf Of Carter Bullard
> Sent: Wednesday, March 13, 2013 12:06 PM
> To: Craig Merchant
> Cc: Argus (argus-info at lists.andrew.cmu.edu)
> Subject: Re: [ARGUS] A couple of labeling questions/issues...
>
> Hey Craig,
> Here is the deal with all the labeling and country code issues.
>
> The problem is that we implemented country code labeling as a generic label, and there were assumptions regarding what the labels looked like. This worked very well, until recently, when we extended labeling with a bunch more label sources and formats, to exploit the general implementation that we had done.
>
> As a result, when you do IANA based labels and RIR based labels, both at the same time, because they are both address based labels (i.e. this CIDR, this range, this specific IP address gets this label) , we use the same Patricia Tree data structure to hold the data. No problem, actually not a bad idea. Bur currently, when we go to read the country code from the general label structure, we expect it to be at a specific place.
>
> However, when you provide your own labels, this convention breaks, and our methods break. I.e., we get your bug, where your country codes look like arbitrary 2 character snippets from the general label structure.
>
> OK, now, the fix...... I'm going to continue to overload the address Patricia Tree, as you get some great side effects doing this. But I have to add country codes as a specific object in the generic address structures. I should have a working fix by this afternoon, and will upload as either a refreshed 3.0.7.6 or a new 3.0.7.7. (I haven't really announced 6, so was hoping to just reuse it ).
>
> Thanks for all the data, and the config files. Makes fixing bugs actually possible !!!!!
>
> Carter
>
>
> On Mar 13, 2013, at 10:49 AM, Carter Bullard <carter at qosient.com> wrote:
>
>
>
>
> Hey Craig,
> Status.....
>
> For this problem
> I’ve tried using the –M label=”regex” option, but the regex never seems to match.
>
> ralabel -S 10.230.174.40:561 -f /usr/local/argus/ralabel.conf-n -u -c "," -s +sco,+dco,+label:200 -M label="blacklisted", for example, doesn’t match the following events:
>
> The -M label="regex" test is done on input, but its ralabel() that is putting in the label, so the option
> given to ralabel() cannot work, even theoretically ;O). Need to pipe the ralabel() output to
> something that can do the test. This would work:
>
> ralabel -S 10.230.174.40:561 -f /usr/local/argus/ralabel.conf -w - | \
> ra -n -u -c "," -s +sco,+dco,+label:200 -M label="blacklisted"
>
> definitely works with the example files you provided. Sorry I didn't see the problem immediately.
> With this type of strategy, using radium() to label the records make some sense. That way you
> would just run:
>
> ra -S radium -n -u -c "," -s +sco,+dco,+label:200 -M label="blacklisted"
>
> To get your blacklisted flows. Still working on the country code problem.
>
> Carter
>
>
>
> On Mar 13, 2013, at 10:35 AM, Craig Merchant <cmerchant at responsys.com> wrote:
>
>
>
>
> Yeah, that covers it. Or more precisely, when I use “-s +sco,+dco”, one of the country code fields is incorrect and the incorrect field seems to be populated with characters from the label.
>
> I experimented early on with the argus-flow file, but once I realized how much easier it was to manipulate my data into the iana formatted files, I stopped playing around with it. So, I don’t really have an argus-flow file to speak of.
>
> Thanks!
>
> Craig
>
> From: Carter Bullard [mailto:carter at qosient.com]
> Sent: Wednesday, March 13, 2013 7:16 AM
> To: Craig Merchant
> Cc: Argus (argus-info at lists.andrew.cmu.edu)
> Subject: Re: [ARGUS] A couple of labeling questions/issues...
>
> Hey Craig,
> Yes I did get your Argus.zip file. No email files showed up.
>
> While your ralabel.conf has RALABEL_ARGUS_FLOW=no, you still refer to
> an argus-flow file specification, but you didn't send that. Can you provide?
> I'll test to see if that causes an issue (directive = no, but conf provided ).
>
> It will take me a little while to wade through this, but to keep focus on the issue,
> your bug is that you are seeing incorrect country codes being reporred with -s +sco +dco ?
> And you are not getting correct grep matches using the " -M label="regex" " ?
>
> Does that cover it ?
>
> Carter
>
> On Mar 11, 2013, at 8:35 PM, Craig Merchant <cmerchant at responsys.com> wrote:
>
>
>
>
>
> I just posted Argus.zip to your FTP server. It contains all of my config files, scripts called by rastream, our label file, and some flow data. Let me know if you need anything else.
>
> Thanks!
>
> Craig
>
> From: Carter Bullard [mailto:carter at qosient.com]
> Sent: Monday, March 11, 2013 11:24 AM
> To: Craig Merchant
> Cc: Argus (argus-info at lists.andrew.cmu.edu)
> Subject: Re: [ARGUS] A couple of labeling questions/issues...
>
> Hey Craig,
> Fractured pelvis is pretty tough. New hip sounds traumatic, but they
> get you going very early. Not really sure how I'm able to do it, it does
> seem like so early given what they did.
>
> ftp the files to ftp://qosient.com/incoming
> Its a blind directory.
>
> Thanks!!!!!
>
> Carter
>
> On Mar 10, 2013, at 5:40 PM, Craig Merchant <cmerchant at responsys.com> wrote:
>
>
>
>
>
>
> Given the surgery you’re recovering from, your continued support has been quite admirable! My doctor won’t even let me start on any kind of physical therapy for the fractured pelvis for at least another couple of weeks. I’m going a bit stir crazy myself.
>
> I’ll experiment with your suggestions this afternoon. I’ll send you some flow records, my label file, and all of the scripts I’ve been using later today. Do have any idea what the largest attachment is that your email server will accept? I can also share that via Dropbox if you’d like.
>
> Thanks again!
>
> Craig
>
> From: Carter Bullard [mailto:carter at qosient.com]
> Sent: Sunday, March 10, 2013 9:57 AM
> To: Craig Merchant
> Cc: Argus (argus-info at lists.andrew.cmu.edu)
> Subject: Re: [ARGUS] A couple of labeling questions/issues...
>
> Hey Craig,
> Sorry for the delayed response, therapy is a bit time consuming.
> OK, well the you've described about 20 issues, so which one would you like to work on first?
>
> So, the file format is the NIC registrar's RIR file format. See this link for a description:
> https://www.apnic.net/publications/media-library/documents/resource-guidelines/rir-statistics-exchange-format
>
> ZZ is used when looking up a country code results in no returned value.
>
> Some of the values you're interested in are labels, but some are not. Country codes are not labels, there is
> a COCODE DSR that holds the source and destination 2 or three character country codes. If the country codes
> are screwed up, you may need to remove the errant data, in order to get new values added.
>
> I'm worried that you're use of " -M dsrs='+..." could have inserted country code DSRs without initalized values.
> So run this with your ralabel() command to see if you get something better:
>
> ralabel -M dsrs="-cocode" .............
>
>
> Of course, I can't comment on your problems, until you provide the complete configurations, and argus data
> that I can use to recreate the problem. Then I can try to fix it ?
>
>
> Carter
>
> On Mar 9, 2013, at 6:14 PM, Craig Merchant <cmerchant at responsys.com> wrote:
>
>
>
>
>
>
>
> Hey, Carter… I’ve been testing the labeling functionality and trying to investigate further the country code issue I posted about earlier in the week.
>
> I’m currently using the delegated-ipv4-latest file that is included with the 3.0.7.5 client. I’ve noticed that Argus isn’t finding a country code match for our public IP range. I’d like to add those ranges to the delegated-ipv4-latest, but I’m not sure what the format is. Is there any way to make the file recognize a subnet mask? I notice that the entry for 10.0.0.0 isn’t causing the country codes for my internal networks to show up as “ZZ”. What do the numbers in the second and third to last columns represent?
>
> delegated-afrinic-latest:afrinic|ZA|ipv4|41.73.32.0|8192|20100112|allocated
>
> Is it possible to use the MaxMind GeoIP database for country codes instead of the iana file? I compiled the clients with GeoIP support. The ralabel.conf file says the GEOIP labels will be saved as scity,dcity,icity. I’ve tried adding those fields to my searches and nothing shows up. I looked through rarc.print.all.conf to see if I could find a field in there that looked related to GeoIP data, but nothing popped out at me. I’m using the same GeoLiteCity.dat file that we use on our Splunk server. The GeoIP part of my ralabel.conf is:
>
> RALABEL_GEOIP_CITY="saddr,daddr,inode:cco,cname"
> RALABEL_GEOIP_CITY_FILE="/usr/local/argus/GeoIPCity.dat"
>
> I’ve tried using the –M label=”regex” option, but the regex never seems to match.
>
> ralabel -S 10.230.174.40:561 -f /usr/local/argus/ralabel.conf -n -u -c "," -s +sco,+dco,+label:200 -M label="blacklisted", for example, doesn’t match the following events:
>
> 1362868911.000236, e s ,tcp,199.7.204.127,39935, ->,82.98.86.167,25,2,148,REQ,pu,bl,saddr=public,DC2,Outbound:daddr=DE,blacklisted
> 1362868911.000282, e s ,tcp,199.7.204.127,56477, ->,82.98.86.161,25,2,148,REQ,pu,bl,saddr=public,DC2,Outbound:daddr=DE,blacklisted
> 1362868912.000211, e s ,tcp,199.7.202.248,44177, ->,82.98.86.171,25,2,148,REQ,pu,bl,saddr=public,DC2,Outbound:daddr=DE,blacklisted
> 1362868913.000465, e s ,tcp,199.7.204.127,42282, ->,82.98.86.171,25,2,148,REQ,pu,bl,saddr=public,DC2,Outbound:daddr=DE,blacklisted
>
> The events above also show the issue with country codes. I’m noticing a few things… The erroneous country codes are always a pair of letters found somewhere in the label, but I don’t see any pattern as to which letters are used. If I disable the iana label file in ralabel.conf, country codes display normally. If a country code is found for both a source and destination, one of the country codes is always incorrect, but the country code is properly displayed in the label field:
>
> 1362869594.000036, e ,udp,12.130.136.11,53, ->,69.154.227.43,53,1,70,INT,pu,US,saddr=US,public,DC1,mta-external:daddr=US
> 1362869594.000036, e ,udp,12.130.136.12,53, ->,200.26.226.6,53,1,68,INT,pu,AN,saddr=US,public,DC1,mta-external:daddr=AN
>
> In the first set of events, both country codes are incorrect (dest code not shown). In the events above, the destination code is shown correctly. I have noticed that if, as above, the 12.130.136.12 has its sco field displayed incorrectly, it will also have its dco field displayed incorrectly when it is the destination of a flow.
>
> If there is no label for a source or destination, the country code for that host will be empty:
>
> 1362866671.000438, e ,udp,74.125.17.82,42492,<->,192.168.30.41,53,4,1260,CON,,dn,daddr=internal,DC4,SP1--Apache,dns,ext-DC4
>
> I enabled label support and radium and ran the same query as above, except with ra. The issue with the country codes is the same as when I run the searches with ralabel. Radium has a fairly significant memory leak when labels are enabled.
>
> Let me know if you would like me to send you any binary flow files or my iana label file.
>
> Thanks!
>
> Craig
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130318/3ffa44c1/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2589 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130318/3ffa44c1/attachment.bin>
More information about the argus
mailing list