ragrep newbie question ?

Wed Mar 17 13:28:41 EDT 2010

Hey Stéphane and John,
I now have ragrep() working for many  of the options that grep supports.  It will
take me a day to get it all down, but I've run into some things that I'd love
some opinions on.

First I have given ragrep() a 1M grep buffer, so we can have big regular
expressions.  I've added parsing from a file, and right now, I support
one regular expression, on a single line (using conventional '\' at the end
of the line to create long single lines).  I'll add support for multiple
expressions in the file,  tonight.

I've added the -c. -i, -q, -v, -l and -L options that generate the same behavior
as grep(), so we're getting somewhere (and I have a man page !!!).

The issue that I have is what should ragrep()s behavior be when using
the -v option?  The -v option inverts the matching logic so it returns lines that do
not match the regular expression.    Now, ragrep() only matches regular
expressions from  the user data buffers in flow records.   How should we test
records that don't have user data buffers?   Do they get included in the test,
or should they be excluded?

So, lets say I have a file, /tmp/argus.file, with 100 argus records, where 80 have
user data buffers.  Lets say that I search for the bad domains list below, that
are specified in the bad.domains file,  and the result is that there are no matches.  

If I use the "-c" option, ragrep() will printout the same output as grep,  filename:count
on a single line for every file that has matches.

  % ragrep -c -f bad.domains -r /tmp/*
       /tmp/argus.file:0

Now if I invert the matching logic,  ragrep() could report on the inverse of the
test, which means we would only try to match on records that have user data buffers.

  % ragrep -c -f bad.domains -v -f argus.file
      argus.file:80

Is this reasonable, or should ragrep() report 100?
I guess the question is, "if I'm looking for all the records that don't have 'whatever' in
the user buffer", do records that don't have a user buffers match?

Carter

On Mar 15, 2010, at 2:02 PM, John Kennedy wrote:

> How many OR'd names until a seg fault occurs?  A program similar to rafilteraddr.pl would come in handy for domain names
>  
> There are thousands if not millions of malicious domains and auditing for these gets a little interesting at times.  Building and maintaining rules for outbreaks in an IDS can get quite cumbersome. I think using a tool similar to rafilteraddr.pl would help keep a list concise and could be fast enough for integration into other avenues.
> 
> I am just curious if others audit for known malicious domains in their environment?  Is it even worth the trouble?
> 
> John
> 
> On Sun, Mar 14, 2010 at 4:58 PM, Stéphane Peters <stephane.peters at forem.be> wrote:
> Hello Carter,
> 
> could I do case insensitive searches with ra* clients ?
> Maybe the "-i" option of ragrep could be added in the next release ?
> 
> Meanwhile, let me share a command used to look for evidences of some virus / malware infection.
> It works pretty well without parenthesis; are they needed in this case ?
> The RE is split on several lines to permit a nice posting, it seems limited to 4096 bytes, about 200 site names.
>  #!/bin/bash
>  s="PPIHelper.com"
>  s="$s|solfire.aljosaborkovic.com"
>  s="$s|kukutrustnet777.info"
>  s="$s|www.kjwre.*fqwieluoi.info"
>  s="$s|l33t.brand-clothes.net"
>  s="$s|pica.banjalucke-ljepotice.ru"
>  s="$s|maellisromance.com"
>  s="$s|217.32.75.74"
>  s="$s|pingaksh.com"
>  s="$s|radio.irib.ir"
>  s="$s|regal-mont.pondi.hr"
>  s="$s|sandra.prichaonica.com"
>  s="$s|sasgrowth.com"
>  s="$s|snowboard619.w.interia.pl"
>  s="$s|spargeunid.go.ro"
>  s="$s|stakrix.st.funpic.de"
>  s="$s|us516757.bizhostnet.com"
>  s="$s|www.abassiehmunicipality.com"
>  s="$s|www.polaris.ge"
>  s="$s|www.railwayservices.be"
>  s="$s|www.senaauto.ge"
>  s="$s|ziyagokalpilkogretim72.meb.k12.tr"
>  ragrep -s "+suser:50 -bytes" -i -e "$s" $* - udp port 53
> Here an example of how it is used:
> ragrep-sality.sh -nr $file -w /tmp/sality-traces.ra
> 
> More details on the wiki site  :
>  http://nsmwiki.org/Argus#ragrep_example:_Finding_Palevo_.2F_Sality_virus_activity
> 
> julien a écrit :
> 
> Hello,
> 
> I have a small problem with ragrep, that I recently begin to use
> following this paper
> http://www.rawpacket.org/anonymous/papers/Argus-PracticalBotNetDetection.pdf
> 
> When I launch the following request:
> 
> $ ragrep -z -i -e '(http|https|ftp|get|post|head)' -r $src_log - dst
> port 80 or dst port 443 or dst port 8080
> (same without filter)
> 
> I get nothing. $src_log is an argus file converted from a pcap (with argus)
> In Ethereal, there is some http traffic and ragrep doesn't get it. why ?
> bad command line, pcap conversion or something else ?
> 
> Thanks.
> Best regards,
> 
>        Julien
> 
>  
> Regards,
> 
> -- 
> Stephane.Peters at forem.be.
> 
> 

Carter Bullard
CEO/President
QoSient, LLC
150 E 57th Street Suite 12D
New York, New York  10022

+1 212 588-9133 Phone
+1 212 588-9134 Fax

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20100317/ea39986f/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3815 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20100317/ea39986f/attachment.bin>