Do you know how to read a pcap file continuously?
Carter Bullard
carter at qosient.com
Fri Oct 25 11:06:56 EDT 2013
Hey Sebas,
That is great !!!!
There are other things to look for with regard to botnets,
especially with regard to Command and Control channel
detection. Traditional service protocols, like NTP for example,
have been used for advertisement,
The argus ABR metric is one that you can use to find esoteric
one way data transfer. Take all the records, and sort for " abr ",
rasort data -m abr bytes -w - - ipv4 and not ip multicast and not ether dst ff:ff:ff:ff:ff:ff | \
ra -s stime dur proto saddr sport dir daddr dport spkts dpkts sappbytes dappbytes abr - not tcp and abr gt 0.99
This will give you all the connection-less traffic that is transmitting data,
a few other filter choices will give you traffic with no responses (dst pkts 0).
Its a good list to check for leaking at low levels.
Let the list get involved, if you would like to.
Hope all is most excellent,
Carter
On Oct 25, 2013, at 4:19 AM, el draco <eldraco at gmail.com> wrote:
> Hi Carter, sorry for my late replay.
> Yes I would love to tell you about our setup.
>
> In the university we are researching about botnet behavior on the
> network. For that purpose we need to capture a lot of different and
> real botnet traffic. Also we need long-lived connections (more than 1
> month at least).
> We are capturing on different locations, but they are using the same
> setup in all of them. In one big machine running linux we execute 20
> or more Windows vms using Virtualbox.
>
> Because of the network configuration and some security policies all
> the vms are running NATed. We need the traffic of each machine
> separated and identified (i.e. before the NAT), but because of the NAT
> configuration we can not capture the traffic on the linux computers.
> The only solution we found was to use the packet trace feature of
> virtualbox to capture the traffic.
>
> So in the linux computers we can see the pcap files only.
>
> These pcap files are being analyzed continuously now and, thanks to
> your work, we can plot the main features of the traffic live. Right
> now we are plotting the amount of connections of: DNS, Web, SSL, SSH,
> TCP, UDP, IPV6 and SPAM. The idea is to have a nice plot of the botnet
> activity really quick. We developed a program to read this data from
> the pcap file+argus and output the information to a RRD file. Then we
> are using Cacti to show the plots of all the vms. This way we can
> monitor a lot of infections at the same time.
>
> After the infection is stopped we use ralabel to label the argus file.
> The rules to label the botnets were made by hand and, most of the
> time, we create ad-hoc rules for each capture. This is time consuming
> but is paramount for a proper research on malware and botnets. These
> labels are used later in our machine learning algorithms.
> Also we process the pcap file to create a bidirectional labeled argus
> file, a text-based netflow file, a web log file, a histogram of
> labels, and the a basic network analysis of what happened, along with
> information about the malware and links to virustotal, malwr, anubis,
> etc.
>
> All this information is being freely shared on our web server with
> some organizations working on malware or botnet research. Usually we
> have some sort of informal collaboration with them.
>
> Hope it helps!
>
> thanks
> sebas
>
>
>
>
> On Tue, Oct 8, 2013 at 12:42 PM, Carter Bullard <carter at qosient.com> wrote:
>> Hey sebas,
>> With it working for you, could you describe the environment that this is running in ???? There was a mention of VMs, Virtualbox, etc... Could you elaborate a bit on what you now have ???
>>
>> I've done quite a bit of cloud and SDN work with argus, and I'm curious about your approach.
>>
>> Hope all is most excellent !!!
>>
>> Carter
>>
>>> On Oct 8, 2013, at 4:53 AM, el draco <eldraco at gmail.com> wrote:
>>>
>>> Carter! It is working perfectly!!
>>> Thanks a lot for your time on this...
>>>
>>> I tried both a manual tcpdump without -U and the real virtualbox pcap
>>> capture, and both worked fine, here the commands just for the sake of
>>> completion:
>>>
>>> In one console:
>>> tcpdump -n -s0 -i eth0 -v -w /tmp/test.pcap
>>>
>>> In other console:
>>> ./bin/argus -f -r /tmp/test.pcap -w /tmp/test.argus -P 2040
>>>
>>> And now you can both read with ra the socket or the file continuously:
>>> ra -n -S localhost:2040
>>>
>>> It is great! We are going to start using it...
>>> thanks again
>>> sebas
>>>
>>>> On Mon, Oct 7, 2013 at 9:13 PM, Carter Bullard <carter at qosient.com> wrote:
>>>> Hey Sebas,
>>>> Try this version of ArgusSource.c and ArgusSource.h. These should fix trying to read a truncated file.
>>>> Replace the ones in the ./argus directory and remake.
>>>>
>>>> Carter
>>>>
>>>>
>>>>
>>>>> On Oct 7, 2013, at 4:17 PM, el draco <eldraco at gmail.com> wrote:
>>>>>
>>>>> Hi Carter
>>>>>
>>>>> The basic stuff we need is to read a common pcap file (no -U in
>>>>> tcpdump) continuously with argus.
>>>>>
>>>>> We are not expecting argus to survive a file deletion.
>>>>> If the pcap file is not there or it is deleted, we start over, no
>>>>> problem for us.
>>>>> If the file is not there when argus starts, argus can exit.
>>>>>
>>>>> Does this make it easier?
>>>>> thanks!
>>>>> sebas
>>>>>
>>>>>
>>>>>> On Mon, Oct 7, 2013 at 7:10 PM, Carter Bullard <carter at qosient.com> wrote:
>>>>>> Hey Sebas,
>>>>>> I can recover from the truncated file error condition, but
>>>>>> its a bit expensive, pcap_close(), look for a change in size
>>>>>> then pcap_open() lseek() …, but its really expensive.
>>>>>>
>>>>>> If you can describe the conditions that you expect, like
>>>>>> should survive file deletion, etc… Then we can wrap this
>>>>>> thing up pretty quick.
>>>>>>
>>>>>> Carter
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Oct 7, 2013, at 11:18 AM, el draco <eldraco at gmail.com> wrote:
>>>>>>>
>>>>>>> Hi list, we have been playing a little bit with reading a pcap file
>>>>>>> continuously. I'm resending our posts so they remain in the list.
>>>>>>>
>>>>>>> Sorry that you have to read the mails backwards in time to make sanse of this.
>>>>>>> In summary, you can read a pcap file continuously. Better if the
>>>>>>> tcpdump is using -U, but there is a work-around if it is not. Thanks
>>>>>>> Carter for all your work.
>>>>>>>
>>>>>>> sebas
>>>>>>>
>>>>>>>> On Oct 7, 2013, at 10:21 AM, el draco <eldraco at gmail.com> wrote:
>>>>>>>> Hi Carter!
>>>>>>>>
>>>>>>>> Ufff that's an issue. While I can run it with -U if I can choose to,
>>>>>>>> right now we are using virtualbox
>>>>>>>> -tracefile option to crate the pcap file. It means that when we start
>>>>>>>> a vm, virtualbox automatically creates the pcap file for us. And
>>>>>>>> unfortunately, virtualbox is not running the capture with -U.
>>>>>>>>
>>>>>>>> However I managed to find a workaround!! It is ugly... again.. but it works.
>>>>>>>>
>>>>>>>> So, first you have a non-buffered pcap file running (without -U), like
>>>>>>>> the one from virtualbox -tracefile
>>>>>>>> tcpdump -n -s0 -i eth0 -w /tmp/test.pcap -v
>>>>>>>>
>>>>>>>> Then you have a second tcpdump capture file, translating the packets
>>>>>>>> and adding the buffer...
>>>>>>>> tail -n 1000 -f /tmp/test.pcap | tcpdump -n -s0 -r - -U -w /tmp/test.pcap2
>>>>>>>>
>>>>>>>> This second pcap file is buffered, so now argus can read it without problems:
>>>>>>>> ./bin/argus -f -r /tmp/test.pcap2 -w /tmp/test.argus -P 2040
>>>>>>>>
>>>>>>>> And you can see the argus data with ra continuously:
>>>>>>>> ra -n -S localhost:2040
>>>>>>>>
>>>>>>>>
>>>>>>>> With these commands I'm now able to read the argus data continuously
>>>>>>>> from a pcap file.
>>>>>>>>
>>>>>>>> The only issue I had is this:
>>>>>>>> If I stop the first pcap, then the second pcap exits (the tail command
>>>>>>>> in fact), and the argus and ra commands keep running. However, if I
>>>>>>>> start the first pcap and second pcap again... the argus is NOT getting
>>>>>>>> the data, and the ra command shows nothing. If I restart the argus,
>>>>>>>> everything works again.
>>>>>>>> Not an issue for me, but I report it in case it helps.
>>>>>>>>
>>>>>>>> Thanks a lot for your work on this topic!
>>>>>>>> cheers
>>>>>>>> sebas
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Mon, Oct 7, 2013 at 1:07 PM, Carter Bullard <carter at qosient.com> wrote:
>>>>>>>>> Hey sebas,
>>>>>>>>> You MUST run tcpdump with the -U option to guarantee that the file
>>>>>>>>> contains data on packet boundaries.
>>>>>>>>>
>>>>>>>>> Carter
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Oct 7, 2013, at 5:25 AM, el draco <eldraco at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Verification of argus-3.0.7.5
>>>>>>>>>>
>>>>>>>>>> 1st experiment
>>>>>>>>>> ----------------------
>>>>>>>>>> In one root console: tcpdump -n -s0 -i eth0 -v -w /tmp/test.pcap
>>>>>>>>>>
>>>>>>>>>> In another root console:
>>>>>>>>>> ./bin/argus -h
>>>>>>>>>> Argus Version 3.0.7.5
>>>>>>>>>> ./bin/argus -f -r /tmp/test.pcap -w /tmp/test.argus
>>>>>>>>>>
>>>>>>>>>> After some reading, It exits with this error:
>>>>>>>>>> ArgusError: 07 Oct 13 08:02:18.126142 ArgusGetPackets: pcap_dispatch()
>>>>>>>>>> returned bogus savefile header
>>>>>>>>>>
>>>>>>>>>> The /tmp/test.argus file is created with the flows. And if I run it
>>>>>>>>>> again, the flows are updated.
>>>>>>>>>> However the argus process dies when it reaches the end of the file.
>>>>>>>>>>
>>>>>>>>>> 2nd experiment
>>>>>>>>>> ----------------------
>>>>>>>>>> I deleted the previous files.
>>>>>>>>>> In one root console: tcpdump -n -s0 -i eth0 -v -w /tmp/test.pcap
>>>>>>>>>> Then I stopped the tcpdump.
>>>>>>>>>>
>>>>>>>>>> In another root console: ./bin/argus -f -r /tmp/test.pcap -w /tmp/test.argus
>>>>>>>>>> The argus keeps running without dying.
>>>>>>>>>> The ra -r /tmp/test.argus reads the netfows and exits.
>>>>>>>>>>
>>>>>>>>>> If I start the tcpdump again, the argus runs for 10 seconds, and then
>>>>>>>>>> exits with this error:
>>>>>>>>>> ArgusError: 07 Oct 13 09:08:07.562844 ArgusGetPackets:
>>>>>>>>>> pcap_dispatch() returned bogus savefile header
>>>>>>>>>> The /tmp/test.argus file was updated with the new information and I
>>>>>>>>>> could read it with ra.
>>>>>>>>>>
>>>>>>>>>> 3rd experiment
>>>>>>>>>> ----------------------
>>>>>>>>>> I deleted the previous files.
>>>>>>>>>> In a root console: ./bin/argus -f -i eth0 -w /tmp/test.argus
>>>>>>>>>> I can read the file with ra.
>>>>>>>>>> This works fine.
>>>>>>>>>>
>>>>>>>>>> 4th experiment
>>>>>>>>>> ----------------------
>>>>>>>>>> I deleted the previous files.
>>>>>>>>>> In one NON-ROOT console: tcpdump -n -s0 -i eth0 -v -w /tmp/test.pcap
>>>>>>>>>> In another NON-ROOT console: ./bin/argus -r /home/sebas/test.pcap -f -P 2040
>>>>>>>>>>
>>>>>>>>>> After some reading, It exits with the same error
>>>>>>>>>> ArgusError: 07 Oct 13 09:14:45.628054 ArgusGetPackets: pcap_dispatch()
>>>>>>>>>> returned bogus savefile header
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Info about my setup:
>>>>>>>>>> ii libpcap-dev 1.4.0-2 all
>>>>>>>>>> development library for libpcap (transitional package)
>>>>>>>>>> ii libpcap0.8:i386 1.4.0-2 i386
>>>>>>>>>> system interface for user-level packet capture
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Tell me if you need me to test something else.
>>>>>>>>>> cheers
>>>>>>>>>> sebas
>>>>>>>>>>
>>>>>>>>>>> On Sun, Oct 6, 2013 at 5:50 PM, Carter Bullard <carter at qosient.com> wrote:
>>>>>>>>>>> So, this works, but still needs some of the exceptions worked out,
>>>>>>>>>>> such as if you rename or delete the packet file, but it should work....
>>>>>>>>>>> Please give this version of argus a trial, and send any feedback,
>>>>>>>>>>> bugs, additional features needed, etc soon !!!
>>>>>>>>>>>
>>>>>>>>>>> The man page for argus has been updated for this feature yet,
>>>>>>>>>>> so mostly done.
>>>>>>>>>>>
>>>>>>>>>>> This version has some other changes, so this will serve as general
>>>>>>>>>>> testing as well. The other changes are basically idle interface
>>>>>>>>>>> timer issues, and shouldn't be an issue.
>>>>>>>>>>>
>>>>>>>>>>> Use the -f option, along with the -r packetfile option, just like tail.
>>>>>>>>>>> This may work with pipes as well, but I didn't test.
>>>>>>>>>>>
>>>>>>>>>>> Carter
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> On Oct 6, 2013, at 1:29 PM, el draco <eldraco at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Carter, We are waiting for it! Yes!!!
>>>>>>>>>>>>
>>>>>>>>>>>> We are now using tshark and offline argus..., so if you manage to make it work,
>>>>>>>>>>>> we will be more than happy to test it.
>>>>>>>>>>>>
>>>>>>>>>>>> Just tell me what you need from us.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks a lot!
>>>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6837 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20131025/b563aa82/attachment.bin>
More information about the argus
mailing list