feature request: ragrep

[CS Lee]

hi carter,

Since I'm not doing this in real time environment, so I can still deal with
the performance issue.
The reason I'm doing this is to identify certain file transfer for long flow
easily, and I can know what file it is transfer(whether it is doc, zip or
whatever file that can be identified using magic file metadata). I'm looking
into certain offset because for example http file transfer(it needs to go
through the http property set before the exact file transfer started so i
don't need to find at the beginning but instead only start matching after
certain byte offset.

Snort uses distance to limit certain range for better performance. Yes, for
offset search, I'm looking forward for implementation such as s[10]:"regex"
to search the regex within 10 bytes, or maybe we can jump to certain offset
and start searching from there.

For the moment, the previous example will do the trick. Normally as for now
performance is not my concern, so what i usually do is -

argus collects flow with user data bytes
racluster merges the flow in 30 minutes/hourly basis
ragrep certain flow and store it separately
rastrip the user data as I don't need user data anymore

So with that I can quickly explain why certain flow has high data bytes, and
what kind of file it transfers even without granular data as I can't afford
expensive disks for the current setup environment.

Thanks!

On Tue, Jun 17, 2008 at 10:18 PM, Carter Bullard <carter at qosient.com> wrote:

> Hey CS Lee,Now these regular expressions are not the highest performing
> things in the world.
> And if you write a program to test a flow against, say a couple hundred
> regex's, you
> will be hard pressed to maintain 10K records per second, so, ...., how
> important is
> this start the search x bytes in front of the data?
>
> Most of the snort filters that are looking at content do offsets for
> performance,
> not necessarily to have better filters.  We can do an offset search, by
> modifying
> the call to the -e option to have something like this"
>     -e s[10]:"regex"
>
> If you have more than one filter.
>
> Carter
>
> On Jun 17, 2008, at 12:51 AM, CS Lee wrote:
>
> Hi Carter,
>
> The repetition works, I try to apply .{5,} and it will do at least with
> minimum 5 character before the matching.
>
> Thanks for the clue, I do use repetition but haven't use it with ragrep, i
> will try out the s: and d: matching.
>
> Blame my brain damage ;]
>
>
>
> On Tue, Jun 17, 2008 at 1:57 AM, Carter Bullard <carter at qosient.com>
> wrote:
>
>> Yes, put a "s:" or a "d:" in front of the string.Carter
>>
>>
>> On Jun 16, 2008, at 1:19 PM, Nick Diel wrote:
>>
>> Hey guys,
>>
>> Is is possible to grep only the source or the destination user data?
>>
>> Thanks,
>> Nick
>>
>> On Mon, Jun 16, 2008 at 10:37 AM, Carter Bullard <carter at qosient.com>
>> wrote:
>>
>>> Hey CS Lee,So can't you specify  this using regular expression anchors
>>> and
>>> repetition?  So you want to find "root" anywhere after 11 characters
>>> from the front of the user data.
>>>
>>>    -e "^.{11}.*root"
>>>
>>> The '^' anchors the search at the start of the string.  the ".{11}"
>>> requires
>>> that there be 11 characters of something, and then anywhere after that,
>>> the regular expression will match 'root'.
>>>
>>> Does that do it?
>>>
>>> Carter
>>>
>>> On Jun 15, 2008, at 11:05 AM, CS Lee wrote:
>>>
>>> hi carter,
>>>
>>> I'm making a request about ragrep to add the search range offset. For
>>> example the matching only apply to first 10 bytes in user data, or between
>>> 25-30 bytes in user data. With the range specification it can reduce false
>>> positive to filter desired flows.
>>>
>>> Thanks.
>>>
>>> --
>>> Best Regards,
>>>
>>> CS Lee<geek00L[at]gmail.com>
>>>
>>> http://geek00l.blogspot.com
>>>
>>>
>>>
>>
>>
>
>
> --
> Best Regards,
>
> CS Lee<geek00L[at]gmail.com>
>
> http://geek00l.blogspot.com
>
>
>


-- 
Best Regards,

CS Lee<geek00L[at]gmail.com>

http://geek00l.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20080617/5b784e49/attachment.html>