radump more tshark-like?

Wed Jul 10 11:48:11 EDT 2013

Hey Elof2,
Well, I like radecode(), so I'm not going to compete with it.  Since its Dave's
program, I'd like Dave to direct any changes that may need to occur to
improve his tool.

radump() is really an example program to show how you can build you're
own user data decode programs.  The concept is that if people wanted
better, they would contribute and make the changes to make it better.
That strategy doesn't always work...

Hey, probably more important, would you spend money for improvements 
to argus?  I have to start commercializing some of this project, would love
to hear comments on what someone would pay for. 

Carter

On Jul 10, 2013, at 11:40 AM, elof2 at sentor.se wrote:

> On Wed, 10 Jul 2013, Carter Bullard wrote:
> 
>> Hey Elof2,
>> The reason generating a pcap file from flow data is a hack, is because we
>> don't capture everything needed to generate the actual packet.  What we
>> don't have will generate some uncertainty, such as missing tunnel
>> encapsulations, some option values, etc.....
> 
> Yep.
> In my case I only capture the first 120 bytes too, so it won't be a complete packet.
> 
>> But ra* programs can
>> generate packets better than the text2pcap method, as there is more
>> info in argus records than can be conveyed to text2pcap with current
>> printing schemes.
> 
> Exactly my point and why I would like you to spend some time coding the system than improving David's 'radecode'.
> 
>> One thing we could do, if there are issues, is have a printer that is
>> specifically designed for text2pcap ?  generating a good ethernet header,
>> next header, next header, etc....
> 
> That sounds like a fantastic middle way. That is proably easier for you than to generate a true pcap from the argus data.
> 
> Some areas to consider when writing this:
> * for each flow, the new text2pcap-printer need to print two packets, one with suser data, sttl, etc and the other with duser data, etc
> * what to do with flows that lack any user data, like a "S_RA"? Should the printer generate a SYN packet and a RST+ACK-packet, or simply supress them? Since all of this is related to displaying the user data, my opinion is to supress empty packets and only generate packets where there are suser or duser data.
> 
>> I'm just not sure that this is the best place to spend a huge amount of time.
>> You are just not happy with tcpdump() and would rather have tshark()
>> decoding ?
> 
> Will it really be a huge amount of time to code this printer?
> I *really* would like this new functionality together with radecode, but I don't want to burden you with a new huge project.
> 
> No, tcpdump is pretty worthless at displaying the data you are actually interested in. Like the URL, the NetBIOS name, the domain and username:
> 
> 'tcpdump port 80' don't show me the "GET /icons/green.gif" request or the "HTTP/1.1 200 OK  (GIF89a) (GIF89a) (image/gif)" response as tshark do. Tcpdump only show "Flags [P.], seq 1:293, ack 158, win 6432, length 292".
> 
> 'tcpdump port 137' only show the uninteresting "NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST" while tshark show "NBNS 92 Name query NB DC01.ST.FOO.SE<20>"
> 
> 'tcpdump port 445' only show "SMB PACKET: SMBtrans2 (REQUEST)" while tshark show "Session Setup AndX Request, NTLMSSP_AUTH, User: FOO\elof"
> 
> I'm interested in all those human readable pieces of information from the user data: User: FOO\elof, DC01.ST.FOO.SE, URLs, etc.
> 
> /Elof
> 
> 
>> On Jul 10, 2013, at 10:34 AM, elof2 at sentor.se wrote:
>> 
>>> 
>>> Unfortunetly I'm no programmer. :-/
>>> 
>>> On the other hand, I'm totally fine with a hack!
>>> David's 'radecode' is a hack.
>>> I simply ask that this hack can be improved, since it is extremely useful.
>>> 
>>> 
>>> IMHO the best thing would be if radump could generate a pcap file.
>>> Then users like myself can use tshark, tcpdump or even open it in Wireshark to look for specific details.
>>> 
>>> The alternative would be for David (and the community) to improve the existing radecode (see my other thread).
>>> 
>>> 
>>> So my main question is if you Carter can, and would like to, code a pcap output capability into radump (or ra if you find that better)?
>>> 
>>> /Elof
>>> 
>>> 
>>> On Wed, 10 Jul 2013, Carter Bullard wrote:
>>> 
>>>> Hey Elof2,
>>>> We provide radump() as an example.  If you want more / different decoding,
>>>> you can easily add it to radump().  If you would rather have tshark buffer
>>>> decoding, if tshark() source code is any good, it shouldn't be difficult for
>>>> anyone to use it instead of tcpdump() routines.  The radump() source
>>>> shows how you would do that.
>>>> 
>>>> I personally think that manufacturing a pseudo packet to stuff into tshark is
>>>> a hack.  But I don't have any problems with you doing whatever, to do
>>>> what you want it to do.  Thats why we provide source code.
>>>> 
>>>> So is the tshark source code so difficult to work with that you guys can't
>>>> write rashark() ?
>>>> 
>>>> Carter
>>>> 
>>>> On Jul 10, 2013, at 9:17 AM, elof2 at sentor.se wrote:
>>>> 
>>>>> 
>>>>> Yes, radecode is just what I was looking for.
>>>>> 
>>>>> Regarding this thread:
>>>>> With radecode together with tshark and text2pcap I can do pretty much
>>>>> everything I asked for, so you don't have to do any changes at all if you don't want to.
>>>>> 
>>>>> 
>>>>> If you still WANT to do some changes, here what would still be useful:
>>>>> 
>>>>> * a smart strip-all-binary printer that only output human readable strings
>>>>> 
>>>>> * Pcap output in radump
>>>>> 
>>>>> 
>>>>> If you can create a pcap output in radump, then David's 'radecode' wouldn't need to do all the text2pcap conversion (which also limits it to tcp and upd packets only). Instead the 'radecode' (or 'rashark' as I would like to call it) would simply be a wrapper-script that re-use the ra commandline options to let radump create the pcap and tshark to view it.
>>>>> 
>>>>> The reason there's a need for a wrapper script at all:
>>>>> * tshark can't read from stdin, it has to read a pcap *file*
>>>>> * I'm lazy
>>>>> After I have a ra cmdline I'm satisfied with, I only want to rename "ra"
>>>>> to "rashark" in order to see the data decoded by tshark, not
>>>>> - manually run radump, specifying a pcap filename
>>>>> - manually run tshark, specifying this file
>>>>> - remember to remove the file when done
>>>>> 
>>>>> 
>>>>> So if you Carter can do a better job creating a pcap than David can using text2pcap, I think David should wait a bit before dealing with the bugs and requests for radecode, since this could be an easier and better solution.
>>>>> 
>>>>> What do you say?
>>>>> 
>>>>> /Elof
>>>>> 
>>>>> 
>>>>> 
>>>>> On Tue, 9 Jul 2013, Carter Bullard wrote:
>>>>> 
>>>>>> Hey Elof2,
>>>>>> Not sure where we are on this thread.
>>>>>> Started with new printers, and now we have radecode(), which is a good thing.
>>>>>> 
>>>>>> So,…, are we still wanting new printers ?
>>>>>> Do we have a good definition of what we want to do?
>>>>>> 
>>>>>> Carter
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Jul 2, 2013, at 2:44 PM, elof2 at sentor.se wrote:
>>>>>> 
>>>>>>> 
>>>>>>> On Tue, 2 Jul 2013, Carter Bullard wrote:
>>>>>>>> Hey Elof2,
>>>>>>>> 
>>>>>>>> OK, more printers !!!!
>>>>>>>> If you would like for me to work with these 3 printer types,
>>>>>>>> strip-all-binary, decode-netbios-names, and decode-barred-smb,
>>>>>>>> I'll implement them in ra(), which means every ra* program
>>>>>>>> will be able to do it, except radump(), which has its own
>>>>>>>> methods for printing user buffers.
>>>>>>>> If you want to think about it for a bit, and come up with
>>>>>>>> another scheme, that would be cool as well.  I have to
>>>>>>>> go through the code to prepare for additional printers,
>>>>>>>> etc…, anyway, so we've got some time to experiment.
>>>>>>> 
>>>>>>> Having it in all ra* tools sounds good. Then one can switch tool and still see the data in the same consistent manner.
>>>>>>> Radump will be the exception.
>>>>>>> 
>>>>>>> To be honest, I've hardly ever used radump. What are the difference running radump vs ra? The only thing I've seen so far is some minor changes in the printed user data fields.
>>>>>>> As long as radump is tcpdump-like and not tshark-like, I think I will stick with ra.
>>>>>>> Tcpdump seldom add any new good information in its info-line. Contrary to tshark, which often display exactly the data you're looking for (Netbios name, SMB login/domain, SMB share, etc).
>>>>>>> Making radump tshark-like is probably not a simple task though, since tshark use lots of dissectors to understand and parse all protocols.
>>>>>>> Buf IF radump could be made tshark-like, it would be really awesome since you would then have even more human readable data to glance through (even binary protocols could be decoded).
>>>>>>> One request still remain though: Is it possible to make radump use the strip-all-binary?
>>>>>>> 
>>>>>>> 
>>>>>>>> If our goal is to find all information available about
>>>>>>>> an IP address, then there are lots of places to look,
>>>>>>>> IP flows, sub encapsulations, tunnel identifiers, ARP
>>>>>>>> and RARP flows, DNS user buffer data, referrals in HTTP
>>>>>>>> data, etc…  Do we want to think about all of that as well?
>>>>>>> 
>>>>>>> No, finding all that data is the job of the eye, skimming through the argus data on the screen.
>>>>>>> Ra only need to deal with normalizing/decoding data into human readable form for common protocols that don't show up in grep:able cleartext by default (such as netbios names and barred text).
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> PS. I'd like the strip-all-binary printer to be a little bit smart, like when you feed a binary file to strings(). Then it will remove lots of garbage that you don't want to see anyhow.
>>>>>>> That is, the printer should supress printing binary data completely and supress printing individual ordinary characters if they are surrounded by binary garbage and don't look like anything human readable.
>>>>>>> 
>>>>>>> 
>>>>>>> PS. Regarding the user data:
>>>>>>> Is there any tool to extract the user data to raw binary form? I would like two new options, one in ra and one in radump:
>>>>>>> 
>>>>>>> Raw output in ra tools)
>>>>>>> I'd like to have a fourth user data printer, a "raw-printer", which will echo the user data in raw. Yes, spew out binary data to the terminal without any conversion.
>>>>>>> When only specifying the suser or duser field, one can easily redirect the binary data into a file or pipe it to some command for post-processing.
>>>>>>> 
>>>>>>> Pcap output in radump)
>>>>>>> I'd like radump to be able to output the data in pcap format. That is:
>>>>>>> * first, output a pcap header
>>>>>>> * now, for each flow, take the flow + suser data and create a packet out of it, and then another response packet for flow + duser data if such exist.
>>>>>>> * continue until EOF
>>>>>>> 
>>>>>>> /Elof
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> On Jul 2, 2013, at 10:47 AM, elof2 at sentor.se wrote:
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Since ra and radump are pretty simillar, and since people usually use ra prior to other ra-tools when browsing through data, I'd say both of them should have the new printer.
>>>>>>>>> 
>>>>>>>>> What I want to do with this new functionality is to try to find the identity of an IP by looking at the argus data.
>>>>>>>>> 
>>>>>>>>> Lets say that I just now got an alert from last week, telling me that IP 10.2.3.4 show traces of a malicious bot infection.
>>>>>>>>> Lets say there are hundreds of different subnets, and no documentation of the network. The only thing I know is that 10.2.3.x is some office in India.
>>>>>>>>> I then need to try to figure out as much as possible about 10.2.3.4 in order to understand which machine is infected ...to be able to tell the technician over which machine to re-install.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> The following three extended-printers would be nice:
>>>>>>>>> 
>>>>>>>>> strip-all-binary)
>>>>>>>>> For all data:
>>>>>>>>> When printing the user data, only echo printable characters. I.e.
>>>>>>>>> supress printing all the placeholder dots for binary data.
>>>>>>>>> decode-netbios-names)
>>>>>>>>> For UDP data on ports 137 and 138: (or on all data?)
>>>>>>>>> find half-ASCII strings and convert them to cleartext
>>>>>>>>> (http://support.microsoft.com/kb/194203)
>>>>>>>>> decode-barred-smb)
>>>>>>>>> For all data (or possibly only TCP data on port 445):
>>>>>>>>> find strings (paths, filenames, UNC paths, etc) that are barred with
>>>>>>>>> dots and remove the dots, leaving only the clean string.
>>>>>>>>> * \.\.E.U.R.S.T.H.L.M.D.C.0.1.\.I.P.C.$   ->   \\EURSTHLMDC01\IPC$
>>>>>>>>> * f.o.o...b.a.r  ->  foo.bar
>>>>>>>>> * S.E.L.E.C.T. .[.U.s.e.r.I.d.]. .F.R.O.M. .[.U.s.e.r.P.r.o.f.i.l.e.].
>>>>>>>>> -> SELECT [UserId] FROM [UserProfile]
>>>>>>>>> 
>>>>>>>>> The shortest string to look for imo should be six characters. Shorter
>>>>>>>>> than that matches too much random garbage:
>>>>>>>>> GREP_OPTIONS=--color=auto ra -nr argus.log -s suser:120 duser:120 - | grep "[a-zA-Z0-9_ $\\]\.[a-zA-Z0-9_ $\\]\.[a-zA-Z0-9_ $\\]\.[a-zA-Z0-9_ $\\]\.[a-zA-Z0-9_ $\\]\."
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> That way I could do a ra/radump search for 10.2.3.4, and skim through the data to see if any details help identify the machine (or the user behind it).
>>>>>>>>> 
>>>>>>>>> By stripping off all the binary junk and only keeping human readable strings I can see stuff like the User Agent, document names, mail addresses, irc chats, dropbox-connections, logins to various systems, etc. (I can even grep for stuff if I want to)
>>>>>>>>> 
>>>>>>>>> /Elof
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Tue, 2 Jul 2013, Carter Bullard wrote:
>>>>>>>>> 
>>>>>>>>>> One other thing.  What do we want to do with this ?  Grep for a name?
>>>>>>>>>> We grep on the printer's output buffer, we don't currently grep on radump()s ouput buffer, so putting the Netbios decode only in radump() will get us only so far.
>>>>>>>>>> 
>>>>>>>>>> Carter
>>>>>>>>>> 
>>>>>>>>>> On Jul 2, 2013, at 8:36 AM, Carter Bullard <carter at qosient.com> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hey Elof2,
>>>>>>>>>>> I don't have any problems making the change, just need to know when to do it.
>>>>>>>>>>> Applying a strange decoding to non-Netbios traffic isn't going to do much positive.
>>>>>>>>>>> 
>>>>>>>>>>> I think we should define a printer, call it "extended", which is where we implement
>>>>>>>>>>> any of these protocol specific decoding capabilities?
>>>>>>>>>>> 
>>>>>>>>>>> OR
>>>>>>>>>>> 
>>>>>>>>>>> we just do it in radump(), and leave ra() alone?
>>>>>>>>>>> 
>>>>>>>>>>> Carter
>>>>>>>>>>> 
>>>>>>>>>>> On Jul 2, 2013, at 8:24 AM, elof2 at sentor.se wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi Carter!
>>>>>>>>>>>> 
>>>>>>>>>>>> I see in the manual for radump that it is tcpdump-like.
>>>>>>>>>>>> Would it be lots of work to make it more tshark-like instead?
>>>>>>>>>>>> 
>>>>>>>>>>>> tcpdump is not parsing Microsoft networking very well (ports 135, 137-139, 445). Tshark on the other hand usually manages to show what I'm interested in, i.e. the machine name, domain, login name, etc.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> It is mainly the Microsoft protocols I need decoded, but naturally other common protocols that can reveal the identity behind an IP address would be interesting.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> In my last email I was asking for a function to decode the NetBIOS half-ASCII.
>>>>>>>>>>>> It would also be nice if data like this:
>>>>>>>>>>>> ......H.&.\.\.E.U.R.S.T.H.L.M.D.C.0.1.\.I.P.C.$.....
>>>>>>>>>>>> was decoded into strings:
>>>>>>>>>>>> ......H.&.\\EURSTHLMDC01\IPC$.....
>>>>>>>>>>>> 
>>>>>>>>>>>> /Elof
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20130710/049456e8/attachment.bin>