Do you know how to read a pcap file continuously?

Fri Oct 25 04:19:07 EDT 2013

Hi Carter, sorry for my late replay.
Yes I would love to tell you about our setup.

In the university we are researching about botnet behavior on the
network. For that purpose we need to capture a lot of different and
real botnet traffic. Also we need long-lived connections (more than 1
month at least).
We are capturing on different locations, but they are using the same
setup in all of them. In one big machine running linux we execute 20
or more Windows vms using Virtualbox.

Because of the network configuration and some security policies all
the vms are running NATed. We need the traffic of each machine
separated and identified (i.e. before the NAT), but because of the NAT
configuration we can not capture the traffic on the linux computers.
The only solution we found was to use the packet trace feature of
virtualbox to capture the traffic.

So in the linux computers we can see the pcap files only.

These pcap files are being analyzed continuously now and, thanks to
your work, we can plot the main features of the traffic live. Right
now we are plotting the amount of connections of: DNS, Web, SSL, SSH,
TCP, UDP, IPV6 and SPAM. The idea is to have a nice plot of the botnet
activity really quick. We developed a program to read this data from
the pcap file+argus and output the information to a RRD file. Then we
are using Cacti to show the plots of all the vms. This way we can
monitor a lot of infections at the same time.

After the infection is stopped we use ralabel to label the argus file.
The rules to label the botnets were made by hand and, most of the
time, we create ad-hoc rules for each capture. This is time consuming
but is paramount for a proper research on malware and botnets. These
labels are used later in our machine learning algorithms.
Also we process the pcap file to create a bidirectional labeled argus
file, a text-based netflow file, a web log file, a histogram of
labels, and the a basic network analysis of what happened, along with
information about the malware and links to virustotal, malwr, anubis,
etc.

All this information is being freely shared on our web server with
some organizations working on malware or botnet research. Usually we
have some sort of informal collaboration with them.

Hope it helps!

thanks
sebas

On Tue, Oct 8, 2013 at 12:42 PM, Carter Bullard <carter at qosient.com> wrote:
> Hey sebas,
> With it working for you, could you describe the environment that this is running in ????  There was a mention of VMs, Virtualbox, etc...  Could you elaborate a bit on what you now have ???
>
> I've done quite a bit of cloud and SDN work with argus, and I'm curious about your approach.
>
> Hope all is most excellent !!!
>
> Carter
>
>> On Oct 8, 2013, at 4:53 AM, el draco <eldraco at gmail.com> wrote:
>>
>> Carter! It is working perfectly!!
>> Thanks a lot for your time on this...
>>
>> I tried both a manual tcpdump without -U and the real virtualbox pcap
>> capture, and both worked fine, here the commands just for the sake of
>> completion:
>>
>> In one console:
>> tcpdump -n -s0 -i eth0 -v -w /tmp/test.pcap
>>
>> In other console:
>> ./bin/argus -f -r /tmp/test.pcap -w /tmp/test.argus -P 2040
>>
>> And now you can both read with ra the socket or the file continuously:
>> ra -n -S localhost:2040
>>
>> It is great! We are going to start using it...
>> thanks again
>> sebas
>>
>>> On Mon, Oct 7, 2013 at 9:13 PM, Carter Bullard <carter at qosient.com> wrote:
>>> Hey Sebas,
>>> Try this version of ArgusSource.c and ArgusSource.h. These should fix trying to read a truncated file.
>>> Replace the ones in the ./argus directory and remake.
>>>
>>> Carter
>>>
>>>
>>>
>>>> On Oct 7, 2013, at 4:17 PM, el draco <eldraco at gmail.com> wrote:
>>>>
>>>> Hi Carter
>>>>
>>>> The basic stuff we need is to read a common pcap file (no -U in
>>>> tcpdump) continuously with argus.
>>>>
>>>> We are not expecting argus to survive a file deletion.
>>>> If the pcap file is not there or it is deleted, we start over, no
>>>> problem for us.
>>>> If the file is not there when argus starts, argus can exit.
>>>>
>>>> Does this make it easier?
>>>> thanks!
>>>> sebas
>>>>
>>>>
>>>>> On Mon, Oct 7, 2013 at 7:10 PM, Carter Bullard <carter at qosient.com> wrote:
>>>>> Hey Sebas,
>>>>> I can recover from the truncated file error condition, but
>>>>> its a bit expensive, pcap_close(), look for a change in size
>>>>> then pcap_open() lseek() …, but its really expensive.
>>>>>
>>>>> If you can describe the conditions that you expect, like
>>>>> should survive file deletion, etc…  Then we can wrap this
>>>>> thing up pretty quick.
>>>>>
>>>>> Carter
>>>>>
>>>>>
>>>>>
>>>>>> On Oct 7, 2013, at 11:18 AM, el draco <eldraco at gmail.com> wrote:
>>>>>>
>>>>>> Hi list, we have been playing a little bit with reading a pcap file
>>>>>> continuously. I'm resending our posts so they remain in the list.
>>>>>>
>>>>>> Sorry that you have to read the mails backwards in time to make sanse of this.
>>>>>> In summary, you can read a pcap file continuously. Better if the
>>>>>> tcpdump is using -U, but there is a work-around if it is not. Thanks
>>>>>> Carter for all your work.
>>>>>>
>>>>>> sebas
>>>>>>
>>>>>>> On Oct 7, 2013, at 10:21 AM, el draco <eldraco at gmail.com> wrote:
>>>>>>> Hi Carter!
>>>>>>>
>>>>>>> Ufff that's an issue. While I can run it with -U if I can choose to,
>>>>>>> right now we are using virtualbox
>>>>>>> -tracefile option to crate the pcap file. It means that when we start
>>>>>>> a vm, virtualbox automatically creates the pcap file for us. And
>>>>>>> unfortunately, virtualbox is not running the capture with -U.
>>>>>>>
>>>>>>> However I managed to find a workaround!! It is ugly... again.. but it works.
>>>>>>>
>>>>>>> So, first you have a non-buffered pcap file running (without -U), like
>>>>>>> the one from virtualbox -tracefile
>>>>>>> tcpdump -n -s0 -i eth0 -w /tmp/test.pcap -v
>>>>>>>
>>>>>>> Then you have a second tcpdump capture file, translating the packets
>>>>>>> and adding the buffer...
>>>>>>> tail -n 1000 -f /tmp/test.pcap | tcpdump -n -s0 -r - -U -w /tmp/test.pcap2
>>>>>>>
>>>>>>> This second pcap file is buffered, so now argus can read it without problems:
>>>>>>> ./bin/argus -f -r /tmp/test.pcap2 -w /tmp/test.argus -P 2040
>>>>>>>
>>>>>>> And you can see the argus data with ra continuously:
>>>>>>> ra -n -S localhost:2040
>>>>>>>
>>>>>>>
>>>>>>> With these commands I'm now able to read the argus data continuously
>>>>>>> from a pcap file.
>>>>>>>
>>>>>>> The only issue I had is this:
>>>>>>> If I stop the first pcap, then the second pcap exits (the tail command
>>>>>>> in fact), and the argus and ra commands keep running. However, if I
>>>>>>> start the first pcap and second pcap again... the argus is NOT getting
>>>>>>> the data, and the ra command shows nothing. If I restart the argus,
>>>>>>> everything works again.
>>>>>>> Not an issue for me, but I report it in case it helps.
>>>>>>>
>>>>>>> Thanks a lot for your work on this topic!
>>>>>>> cheers
>>>>>>> sebas
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On Mon, Oct 7, 2013 at 1:07 PM, Carter Bullard <carter at qosient.com> wrote:
>>>>>>>> Hey sebas,
>>>>>>>> You MUST run tcpdump with the -U option to guarantee that the file
>>>>>>>> contains data on packet boundaries.
>>>>>>>>
>>>>>>>> Carter
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Oct 7, 2013, at 5:25 AM, el draco <eldraco at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Verification of argus-3.0.7.5
>>>>>>>>>
>>>>>>>>> 1st experiment
>>>>>>>>> ----------------------
>>>>>>>>> In one root console: tcpdump -n -s0 -i eth0 -v -w /tmp/test.pcap
>>>>>>>>>
>>>>>>>>> In another root console:
>>>>>>>>> ./bin/argus -h
>>>>>>>>> Argus Version 3.0.7.5
>>>>>>>>> ./bin/argus -f -r /tmp/test.pcap -w /tmp/test.argus
>>>>>>>>>
>>>>>>>>> After some reading, It exits with this error:
>>>>>>>>> ArgusError: 07 Oct 13 08:02:18.126142 ArgusGetPackets: pcap_dispatch()
>>>>>>>>> returned bogus savefile header
>>>>>>>>>
>>>>>>>>> The /tmp/test.argus file is created with the flows. And if I run it
>>>>>>>>> again, the flows are updated.
>>>>>>>>> However the argus process dies when it reaches the end of the file.
>>>>>>>>>
>>>>>>>>> 2nd experiment
>>>>>>>>> ----------------------
>>>>>>>>> I deleted the previous files.
>>>>>>>>> In one root console: tcpdump -n -s0 -i eth0 -v -w /tmp/test.pcap
>>>>>>>>> Then I stopped the tcpdump.
>>>>>>>>>
>>>>>>>>> In another root console: ./bin/argus -f -r /tmp/test.pcap -w /tmp/test.argus
>>>>>>>>> The argus keeps running without dying.
>>>>>>>>> The ra -r /tmp/test.argus reads the netfows and exits.
>>>>>>>>>
>>>>>>>>> If I start the tcpdump again, the argus runs for 10 seconds, and then
>>>>>>>>> exits with this error:
>>>>>>>>> ArgusError: 07 Oct 13 09:08:07.562844 ArgusGetPackets:
>>>>>>>>> pcap_dispatch() returned bogus savefile header
>>>>>>>>> The /tmp/test.argus file was updated with the new information and I
>>>>>>>>> could read it with ra.
>>>>>>>>>
>>>>>>>>> 3rd experiment
>>>>>>>>> ----------------------
>>>>>>>>> I deleted the previous files.
>>>>>>>>> In a root console: ./bin/argus -f -i eth0 -w /tmp/test.argus
>>>>>>>>> I can read the file with ra.
>>>>>>>>> This works fine.
>>>>>>>>>
>>>>>>>>> 4th experiment
>>>>>>>>> ----------------------
>>>>>>>>> I deleted the previous files.
>>>>>>>>> In one NON-ROOT console: tcpdump -n -s0 -i eth0 -v -w /tmp/test.pcap
>>>>>>>>> In another NON-ROOT console: ./bin/argus -r /home/sebas/test.pcap -f -P 2040
>>>>>>>>>
>>>>>>>>> After some reading, It exits with the same error
>>>>>>>>> ArgusError: 07 Oct 13 09:14:45.628054 ArgusGetPackets: pcap_dispatch()
>>>>>>>>> returned bogus savefile header
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Info about my setup:
>>>>>>>>> ii  libpcap-dev       1.4.0-2                        all
>>>>>>>>> development library for libpcap (transitional package)
>>>>>>>>> ii  libpcap0.8:i386  1.4.0-2                        i386
>>>>>>>>> system interface for user-level packet capture
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Tell me if you need me to test something else.
>>>>>>>>> cheers
>>>>>>>>> sebas
>>>>>>>>>
>>>>>>>>>> On Sun, Oct 6, 2013 at 5:50 PM, Carter Bullard <carter at qosient.com> wrote:
>>>>>>>>>> So, this works, but still needs some of the exceptions worked out,
>>>>>>>>>> such as if you rename or delete the packet file, but it should work....
>>>>>>>>>> Please give this version of argus a trial, and send any feedback,
>>>>>>>>>> bugs, additional features needed, etc soon !!!
>>>>>>>>>>
>>>>>>>>>> The man page for argus has been updated for this feature yet,
>>>>>>>>>> so mostly done.
>>>>>>>>>>
>>>>>>>>>> This version has some other changes, so this will serve as general
>>>>>>>>>> testing as well.  The other changes are basically idle interface
>>>>>>>>>> timer issues, and shouldn't be an issue.
>>>>>>>>>>
>>>>>>>>>> Use the -f option, along with the -r packetfile option, just like tail.
>>>>>>>>>> This may work with pipes as well, but I didn't test.
>>>>>>>>>>
>>>>>>>>>> Carter
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On Oct 6, 2013, at 1:29 PM, el draco <eldraco at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Carter, We are waiting for it! Yes!!!
>>>>>>>>>>>
>>>>>>>>>>> We are now using tshark and offline argus..., so if you manage to make it work,
>>>>>>>>>>> we will be more than happy to test it.
>>>>>>>>>>>
>>>>>>>>>>> Just tell me what you need from us.
>>>>>>>>>>>
>>>>>>>>>>> Thanks a lot!
>>