feature request: ragrep

[CS Lee]

hi carter,

By the way, the hex matching doesn't seem to work, I think we have
discussion about this previously.

I try ragrep -e '\x4d\x5a'  and it doesn't seem to match anything. Though I
can see a lot of executable file transfer within the flows.

I remember you have mentioned that the shell may add \ in front of \x so it
becomes \\x and you have fixed it.

Thanks!


On Tue, Jun 17, 2008 at 11:51 PM, CS Lee <geek00l at gmail.com> wrote:

> hi carter,
>
> Since I'm not doing this in real time environment, so I can still deal with
> the performance issue.
> The reason I'm doing this is to identify certain file transfer for long
> flow easily, and I can know what file it is transfer(whether it is doc, zip
> or whatever file that can be identified using magic file metadata). I'm
> looking into certain offset because for example http file transfer(it needs
> to go through the http property set before the exact file transfer started
> so i don't need to find at the beginning but instead only start matching
> after certain byte offset.
>
> Snort uses distance to limit certain range for better performance. Yes, for
> offset search, I'm looking forward for implementation such as s[10]:"regex"
> to search the regex within 10 bytes, or maybe we can jump to certain offset
> and start searching from there.
>
> For the moment, the previous example will do the trick. Normally as for now
> performance is not my concern, so what i usually do is -
>
> argus collects flow with user data bytes
> racluster merges the flow in 30 minutes/hourly basis
> ragrep certain flow and store it separately
> rastrip the user data as I don't need user data anymore
>
> So with that I can quickly explain why certain flow has high data bytes,
> and what kind of file it transfers even without granular data as I can't
> afford expensive disks for the current setup environment.
>
> Thanks!
>
>
> On Tue, Jun 17, 2008 at 10:18 PM, Carter Bullard <carter at qosient.com>
> wrote:
>
>> Hey CS Lee,Now these regular expressions are not the highest performing
>> things in the world.
>> And if you write a program to test a flow against, say a couple hundred
>> regex's, you
>> will be hard pressed to maintain 10K records per second, so, ...., how
>> important is
>> this start the search x bytes in front of the data?
>>
>> Most of the snort filters that are looking at content do offsets for
>> performance,
>> not necessarily to have better filters.  We can do an offset search, by
>> modifying
>> the call to the -e option to have something like this"
>>     -e s[10]:"regex"
>>
>> If you have more than one filter.
>>
>> Carter
>>
>> On Jun 17, 2008, at 12:51 AM, CS Lee wrote:
>>
>> Hi Carter,
>>
>> The repetition works, I try to apply .{5,} and it will do at least with
>> minimum 5 character before the matching.
>>
>> Thanks for the clue, I do use repetition but haven't use it with ragrep, i
>> will try out the s: and d: matching.
>>
>> Blame my brain damage ;]
>>
>>
>>
>> On Tue, Jun 17, 2008 at 1:57 AM, Carter Bullard <carter at qosient.com>
>> wrote:
>>
>>> Yes, put a "s:" or a "d:" in front of the string.Carter
>>>
>>>
>>> On Jun 16, 2008, at 1:19 PM, Nick Diel wrote:
>>>
>>> Hey guys,
>>>
>>> Is is possible to grep only the source or the destination user data?
>>>
>>> Thanks,
>>> Nick
>>>
>>> On Mon, Jun 16, 2008 at 10:37 AM, Carter Bullard <carter at qosient.com>
>>> wrote:
>>>
>>>> Hey CS Lee,So can't you specify  this using regular expression anchors
>>>> and
>>>> repetition?  So you want to find "root" anywhere after 11 characters
>>>> from the front of the user data.
>>>>
>>>>    -e "^.{11}.*root"
>>>>
>>>> The '^' anchors the search at the start of the string.  the ".{11}"
>>>> requires
>>>> that there be 11 characters of something, and then anywhere after that,
>>>> the regular expression will match 'root'.
>>>>
>>>> Does that do it?
>>>>
>>>> Carter
>>>>
>>>> On Jun 15, 2008, at 11:05 AM, CS Lee wrote:
>>>>
>>>> hi carter,
>>>>
>>>> I'm making a request about ragrep to add the search range offset. For
>>>> example the matching only apply to first 10 bytes in user data, or between
>>>> 25-30 bytes in user data. With the range specification it can reduce false
>>>> positive to filter desired flows.
>>>>
>>>> Thanks.
>>>>
>>>> --
>>>> Best Regards,
>>>>
>>>> CS Lee<geek00L[at]gmail.com>
>>>>
>>>> http://geek00l.blogspot.com
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>> --
>> Best Regards,
>>
>> CS Lee<geek00L[at]gmail.com>
>>
>> http://geek00l.blogspot.com
>>
>>
>>
>
>
> --
> Best Regards,
>
> CS Lee<geek00L[at]gmail.com>
>
> http://geek00l.blogspot.com
>



-- 
Best Regards,

CS Lee<geek00L[at]gmail.com>

http://geek00l.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20080618/91374320/attachment.html>