Do you know how to read a pcap file continuously?

Tue Oct 8 08:42:56 EDT 2013

Hey sebas,
With it working for you, could you describe the environment that this is running in ????  There was a mention of VMs, Virtualbox, etc...  Could you elaborate a bit on what you now have ???

I've done quite a bit of cloud and SDN work with argus, and I'm curious about your approach.

Hope all is most excellent !!!

Carter

> On Oct 8, 2013, at 4:53 AM, el draco <eldraco at gmail.com> wrote:
> 
> Carter! It is working perfectly!!
> Thanks a lot for your time on this...
> 
> I tried both a manual tcpdump without -U and the real virtualbox pcap
> capture, and both worked fine, here the commands just for the sake of
> completion:
> 
> In one console:
> tcpdump -n -s0 -i eth0 -v -w /tmp/test.pcap
> 
> In other console:
> ./bin/argus -f -r /tmp/test.pcap -w /tmp/test.argus -P 2040
> 
> And now you can both read with ra the socket or the file continuously:
> ra -n -S localhost:2040
> 
> It is great! We are going to start using it...
> thanks again
> sebas
> 
>> On Mon, Oct 7, 2013 at 9:13 PM, Carter Bullard <carter at qosient.com> wrote:
>> Hey Sebas,
>> Try this version of ArgusSource.c and ArgusSource.h. These should fix trying to read a truncated file.
>> Replace the ones in the ./argus directory and remake.
>> 
>> Carter
>> 
>> 
>> 
>>> On Oct 7, 2013, at 4:17 PM, el draco <eldraco at gmail.com> wrote:
>>> 
>>> Hi Carter
>>> 
>>> The basic stuff we need is to read a common pcap file (no -U in
>>> tcpdump) continuously with argus.
>>> 
>>> We are not expecting argus to survive a file deletion.
>>> If the pcap file is not there or it is deleted, we start over, no
>>> problem for us.
>>> If the file is not there when argus starts, argus can exit.
>>> 
>>> Does this make it easier?
>>> thanks!
>>> sebas
>>> 
>>> 
>>>> On Mon, Oct 7, 2013 at 7:10 PM, Carter Bullard <carter at qosient.com> wrote:
>>>> Hey Sebas,
>>>> I can recover from the truncated file error condition, but
>>>> its a bit expensive, pcap_close(), look for a change in size
>>>> then pcap_open() lseek() …, but its really expensive.
>>>> 
>>>> If you can describe the conditions that you expect, like
>>>> should survive file deletion, etc…  Then we can wrap this
>>>> thing up pretty quick.
>>>> 
>>>> Carter
>>>> 
>>>> 
>>>> 
>>>>> On Oct 7, 2013, at 11:18 AM, el draco <eldraco at gmail.com> wrote:
>>>>> 
>>>>> Hi list, we have been playing a little bit with reading a pcap file
>>>>> continuously. I'm resending our posts so they remain in the list.
>>>>> 
>>>>> Sorry that you have to read the mails backwards in time to make sanse of this.
>>>>> In summary, you can read a pcap file continuously. Better if the
>>>>> tcpdump is using -U, but there is a work-around if it is not. Thanks
>>>>> Carter for all your work.
>>>>> 
>>>>> sebas
>>>>> 
>>>>>> On Oct 7, 2013, at 10:21 AM, el draco <eldraco at gmail.com> wrote:
>>>>>> Hi Carter!
>>>>>> 
>>>>>> Ufff that's an issue. While I can run it with -U if I can choose to,
>>>>>> right now we are using virtualbox
>>>>>> -tracefile option to crate the pcap file. It means that when we start
>>>>>> a vm, virtualbox automatically creates the pcap file for us. And
>>>>>> unfortunately, virtualbox is not running the capture with -U.
>>>>>> 
>>>>>> However I managed to find a workaround!! It is ugly... again.. but it works.
>>>>>> 
>>>>>> So, first you have a non-buffered pcap file running (without -U), like
>>>>>> the one from virtualbox -tracefile
>>>>>> tcpdump -n -s0 -i eth0 -w /tmp/test.pcap -v
>>>>>> 
>>>>>> Then you have a second tcpdump capture file, translating the packets
>>>>>> and adding the buffer...
>>>>>> tail -n 1000 -f /tmp/test.pcap | tcpdump -n -s0 -r - -U -w /tmp/test.pcap2
>>>>>> 
>>>>>> This second pcap file is buffered, so now argus can read it without problems:
>>>>>> ./bin/argus -f -r /tmp/test.pcap2 -w /tmp/test.argus -P 2040
>>>>>> 
>>>>>> And you can see the argus data with ra continuously:
>>>>>> ra -n -S localhost:2040
>>>>>> 
>>>>>> 
>>>>>> With these commands I'm now able to read the argus data continuously
>>>>>> from a pcap file.
>>>>>> 
>>>>>> The only issue I had is this:
>>>>>> If I stop the first pcap, then the second pcap exits (the tail command
>>>>>> in fact), and the argus and ra commands keep running. However, if I
>>>>>> start the first pcap and second pcap again... the argus is NOT getting
>>>>>> the data, and the ra command shows nothing. If I restart the argus,
>>>>>> everything works again.
>>>>>> Not an issue for me, but I report it in case it helps.
>>>>>> 
>>>>>> Thanks a lot for your work on this topic!
>>>>>> cheers
>>>>>> sebas
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> On Mon, Oct 7, 2013 at 1:07 PM, Carter Bullard <carter at qosient.com> wrote:
>>>>>>> Hey sebas,
>>>>>>> You MUST run tcpdump with the -U option to guarantee that the file
>>>>>>> contains data on packet boundaries.
>>>>>>> 
>>>>>>> Carter
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> On Oct 7, 2013, at 5:25 AM, el draco <eldraco at gmail.com> wrote:
>>>>>>>> 
>>>>>>>> Verification of argus-3.0.7.5
>>>>>>>> 
>>>>>>>> 1st experiment
>>>>>>>> ----------------------
>>>>>>>> In one root console: tcpdump -n -s0 -i eth0 -v -w /tmp/test.pcap
>>>>>>>> 
>>>>>>>> In another root console:
>>>>>>>> ./bin/argus -h
>>>>>>>> Argus Version 3.0.7.5
>>>>>>>> ./bin/argus -f -r /tmp/test.pcap -w /tmp/test.argus
>>>>>>>> 
>>>>>>>> After some reading, It exits with this error:
>>>>>>>> ArgusError: 07 Oct 13 08:02:18.126142 ArgusGetPackets: pcap_dispatch()
>>>>>>>> returned bogus savefile header
>>>>>>>> 
>>>>>>>> The /tmp/test.argus file is created with the flows. And if I run it
>>>>>>>> again, the flows are updated.
>>>>>>>> However the argus process dies when it reaches the end of the file.
>>>>>>>> 
>>>>>>>> 2nd experiment
>>>>>>>> ----------------------
>>>>>>>> I deleted the previous files.
>>>>>>>> In one root console: tcpdump -n -s0 -i eth0 -v -w /tmp/test.pcap
>>>>>>>> Then I stopped the tcpdump.
>>>>>>>> 
>>>>>>>> In another root console: ./bin/argus -f -r /tmp/test.pcap -w /tmp/test.argus
>>>>>>>> The argus keeps running without dying.
>>>>>>>> The ra -r /tmp/test.argus reads the netfows and exits.
>>>>>>>> 
>>>>>>>> If I start the tcpdump again, the argus runs for 10 seconds, and then
>>>>>>>> exits with this error:
>>>>>>>> ArgusError: 07 Oct 13 09:08:07.562844 ArgusGetPackets:
>>>>>>>> pcap_dispatch() returned bogus savefile header
>>>>>>>> The /tmp/test.argus file was updated with the new information and I
>>>>>>>> could read it with ra.
>>>>>>>> 
>>>>>>>> 3rd experiment
>>>>>>>> ----------------------
>>>>>>>> I deleted the previous files.
>>>>>>>> In a root console: ./bin/argus -f -i eth0 -w /tmp/test.argus
>>>>>>>> I can read the file with ra.
>>>>>>>> This works fine.
>>>>>>>> 
>>>>>>>> 4th experiment
>>>>>>>> ----------------------
>>>>>>>> I deleted the previous files.
>>>>>>>> In one NON-ROOT console: tcpdump -n -s0 -i eth0 -v -w /tmp/test.pcap
>>>>>>>> In another NON-ROOT console: ./bin/argus -r /home/sebas/test.pcap -f -P 2040
>>>>>>>> 
>>>>>>>> After some reading, It exits with the same error
>>>>>>>> ArgusError: 07 Oct 13 09:14:45.628054 ArgusGetPackets: pcap_dispatch()
>>>>>>>> returned bogus savefile header
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Info about my setup:
>>>>>>>> ii  libpcap-dev       1.4.0-2                        all
>>>>>>>> development library for libpcap (transitional package)
>>>>>>>> ii  libpcap0.8:i386  1.4.0-2                        i386
>>>>>>>> system interface for user-level packet capture
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Tell me if you need me to test something else.
>>>>>>>> cheers
>>>>>>>> sebas
>>>>>>>> 
>>>>>>>>> On Sun, Oct 6, 2013 at 5:50 PM, Carter Bullard <carter at qosient.com> wrote:
>>>>>>>>> So, this works, but still needs some of the exceptions worked out,
>>>>>>>>> such as if you rename or delete the packet file, but it should work....
>>>>>>>>> Please give this version of argus a trial, and send any feedback,
>>>>>>>>> bugs, additional features needed, etc soon !!!
>>>>>>>>> 
>>>>>>>>> The man page for argus has been updated for this feature yet,
>>>>>>>>> so mostly done.
>>>>>>>>> 
>>>>>>>>> This version has some other changes, so this will serve as general
>>>>>>>>> testing as well.  The other changes are basically idle interface
>>>>>>>>> timer issues, and shouldn't be an issue.
>>>>>>>>> 
>>>>>>>>> Use the -f option, along with the -r packetfile option, just like tail.
>>>>>>>>> This may work with pipes as well, but I didn't test.
>>>>>>>>> 
>>>>>>>>> Carter
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On Oct 6, 2013, at 1:29 PM, el draco <eldraco at gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hi Carter, We are waiting for it! Yes!!!
>>>>>>>>>> 
>>>>>>>>>> We are now using tshark and offline argus..., so if you manage to make it work,
>>>>>>>>>> we will be more than happy to test it.
>>>>>>>>>> 
>>>>>>>>>> Just tell me what you need from us.
>>>>>>>>>> 
>>>>>>>>>> Thanks a lot!
> 