radump more tshark-like?

[elof2 at sentor.se]

On Tue, 2 Jul 2013, Carter Bullard wrote:
> Hey Elof2,
>
> OK, more printers !!!!
> If you would like for me to work with these 3 printer types,
> strip-all-binary, decode-netbios-names, and decode-barred-smb,
> I'll implement them in ra(), which means every ra* program
> will be able to do it, except radump(), which has its own
> methods for printing user buffers.
> If you want to think about it for a bit, and come up with
> another scheme, that would be cool as well.  I have to
> go through the code to prepare for additional printers,
> etc…, anyway, so we've got some time to experiment.

Having it in all ra* tools sounds good. Then one can switch tool and still 
see the data in the same consistent manner.
Radump will be the exception.

To be honest, I've hardly ever used radump. What are the difference 
running radump vs ra? The only thing I've seen so far is some minor 
changes in the printed user data fields.
As long as radump is tcpdump-like and not tshark-like, I think I will 
stick with ra.
Tcpdump seldom add any new good information in its info-line. Contrary to 
tshark, which often display exactly the data you're looking for (Netbios 
name, SMB login/domain, SMB share, etc).
Making radump tshark-like is probably not a simple task though, since 
tshark use lots of dissectors to understand and parse all protocols.
Buf IF radump could be made tshark-like, it would be really awesome since 
you would then have even more human readable data to glance through (even 
binary protocols could be decoded).
One request still remain though: Is it possible to make radump use the 
strip-all-binary?


> If our goal is to find all information available about
> an IP address, then there are lots of places to look,
> IP flows, sub encapsulations, tunnel identifiers, ARP
> and RARP flows, DNS user buffer data, referrals in HTTP
> data, etc…  Do we want to think about all of that as well?

No, finding all that data is the job of the eye, skimming through the 
argus data on the screen.
Ra only need to deal with normalizing/decoding data into human readable 
form for common protocols that don't show up in grep:able cleartext by 
default (such as netbios names and barred text).



PS. I'd like the strip-all-binary printer to be a little bit smart, like 
when you feed a binary file to strings(). Then it will remove lots of 
garbage that you don't want to see anyhow.
That is, the printer should supress printing binary data completely and 
supress printing individual ordinary characters if they are surrounded by 
binary garbage and don't look like anything human readable.


PS. Regarding the user data:
Is there any tool to extract the user data to raw binary form? I would 
like two new options, one in ra and one in radump:

Raw output in ra tools)
I'd like to have a fourth user data printer, a "raw-printer", which will 
echo the user data in raw. Yes, spew out binary data to the terminal 
without any conversion.
When only specifying the suser or duser field, one can easily redirect the 
binary data into a file or pipe it to some command for post-processing.

Pcap output in radump)
I'd like radump to be able to output the data in pcap format. That is:
* first, output a pcap header
* now, for each flow, take the flow + suser data and create a packet out 
of it, and then another response packet for flow + duser data if such 
exist.
* continue until EOF

/Elof









> On Jul 2, 2013, at 10:47 AM, elof2 at sentor.se wrote:
>
>>
>> Since ra and radump are pretty simillar, and since people usually use ra prior to other ra-tools when browsing through data, I'd say both of them should have the new printer.
>>
>> What I want to do with this new functionality is to try to find the identity of an IP by looking at the argus data.
>>
>> Lets say that I just now got an alert from last week, telling me that IP 10.2.3.4 show traces of a malicious bot infection.
>> Lets say there are hundreds of different subnets, and no documentation of the network. The only thing I know is that 10.2.3.x is some office in India.
>> I then need to try to figure out as much as possible about 10.2.3.4 in order to understand which machine is infected ...to be able to tell the technician over which machine to re-install.
>>
>>
>> The following three extended-printers would be nice:
>>
>> strip-all-binary)
>>  For all data:
>>  When printing the user data, only echo printable characters. I.e.
>>  supress printing all the placeholder dots for binary data.
>> decode-netbios-names)
>>  For UDP data on ports 137 and 138: (or on all data?)
>>  find half-ASCII strings and convert them to cleartext
>>  (http://support.microsoft.com/kb/194203)
>> decode-barred-smb)
>>  For all data (or possibly only TCP data on port 445):
>>  find strings (paths, filenames, UNC paths, etc) that are barred with
>>  dots and remove the dots, leaving only the clean string.
>>  * \.\.E.U.R.S.T.H.L.M.D.C.0.1.\.I.P.C.$   ->   \\EURSTHLMDC01\IPC$
>>  * f.o.o...b.a.r  ->  foo.bar
>>  * S.E.L.E.C.T. .[.U.s.e.r.I.d.]. .F.R.O.M. .[.U.s.e.r.P.r.o.f.i.l.e.].
>>    -> SELECT [UserId] FROM [UserProfile]
>>
>>  The shortest string to look for imo should be six characters. Shorter
>>  than that matches too much random garbage:
>>  GREP_OPTIONS=--color=auto ra -nr argus.log -s suser:120 duser:120 - | grep "[a-zA-Z0-9_ $\\]\.[a-zA-Z0-9_ $\\]\.[a-zA-Z0-9_ $\\]\.[a-zA-Z0-9_ $\\]\.[a-zA-Z0-9_ $\\]\."
>>
>>
>> That way I could do a ra/radump search for 10.2.3.4, and skim through the data to see if any details help identify the machine (or the user behind it).
>>
>> By stripping off all the binary junk and only keeping human readable strings I can see stuff like the User Agent, document names, mail addresses, irc chats, dropbox-connections, logins to various systems, etc. (I can even grep for stuff if I want to)
>>
>> /Elof
>>
>>
>> On Tue, 2 Jul 2013, Carter Bullard wrote:
>>
>>> One other thing.  What do we want to do with this ?  Grep for a name?
>>> We grep on the printer's output buffer, we don't currently grep on radump()s ouput buffer, so putting the Netbios decode only in radump() will get us only so far.
>>>
>>> Carter
>>>
>>> On Jul 2, 2013, at 8:36 AM, Carter Bullard <carter at qosient.com> wrote:
>>>
>>>> Hey Elof2,
>>>> I don't have any problems making the change, just need to know when to do it.
>>>> Applying a strange decoding to non-Netbios traffic isn't going to do much positive.
>>>>
>>>> I think we should define a printer, call it "extended", which is where we implement
>>>> any of these protocol specific decoding capabilities?
>>>>
>>>> OR
>>>>
>>>> we just do it in radump(), and leave ra() alone?
>>>>
>>>> Carter
>>>>
>>>> On Jul 2, 2013, at 8:24 AM, elof2 at sentor.se wrote:
>>>>
>>>>>
>>>>> Hi Carter!
>>>>>
>>>>> I see in the manual for radump that it is tcpdump-like.
>>>>> Would it be lots of work to make it more tshark-like instead?
>>>>>
>>>>> tcpdump is not parsing Microsoft networking very well (ports 135, 137-139, 445). Tshark on the other hand usually manages to show what I'm interested in, i.e. the machine name, domain, login name, etc.
>>>>>
>>>>>
>>>>> It is mainly the Microsoft protocols I need decoded, but naturally other common protocols that can reveal the identity behind an IP address would be interesting.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> In my last email I was asking for a function to decode the NetBIOS half-ASCII.
>>>>> It would also be nice if data like this:
>>>>> ......H.&.\.\.E.U.R.S.T.H.L.M.D.C.0.1.\.I.P.C.$.....
>>>>> was decoded into strings:
>>>>> ......H.&.\\EURSTHLMDC01\IPC$.....
>>>>>
>>>>> /Elof
>>>>
>>>
>>
>
>