Do you know how to read a pcap file continuously?

[el draco]

Carter! It is working perfectly!!
Thanks a lot for your time on this...

I tried both a manual tcpdump without -U and the real virtualbox pcap
capture, and both worked fine, here the commands just for the sake of
completion:

In one console:
tcpdump -n -s0 -i eth0 -v -w /tmp/test.pcap

In other console:
./bin/argus -f -r /tmp/test.pcap -w /tmp/test.argus -P 2040

And now you can both read with ra the socket or the file continuously:
ra -n -S localhost:2040

It is great! We are going to start using it...
thanks again
sebas

On Mon, Oct 7, 2013 at 9:13 PM, Carter Bullard <carter at qosient.com> wrote:
> Hey Sebas,
> Try this version of ArgusSource.c and ArgusSource.h. These should fix trying to read a truncated file.
> Replace the ones in the ./argus directory and remake.
>
> Carter
>
>
>
> On Oct 7, 2013, at 4:17 PM, el draco <eldraco at gmail.com> wrote:
>
>> Hi Carter
>>
>> The basic stuff we need is to read a common pcap file (no -U in
>> tcpdump) continuously with argus.
>>
>> We are not expecting argus to survive a file deletion.
>> If the pcap file is not there or it is deleted, we start over, no
>> problem for us.
>> If the file is not there when argus starts, argus can exit.
>>
>> Does this make it easier?
>> thanks!
>> sebas
>>
>>
>> On Mon, Oct 7, 2013 at 7:10 PM, Carter Bullard <carter at qosient.com> wrote:
>>> Hey Sebas,
>>> I can recover from the truncated file error condition, but
>>> its a bit expensive, pcap_close(), look for a change in size
>>> then pcap_open() lseek() …, but its really expensive.
>>>
>>> If you can describe the conditions that you expect, like
>>> should survive file deletion, etc…  Then we can wrap this
>>> thing up pretty quick.
>>>
>>> Carter
>>>
>>>
>>>
>>> On Oct 7, 2013, at 11:18 AM, el draco <eldraco at gmail.com> wrote:
>>>
>>>> Hi list, we have been playing a little bit with reading a pcap file
>>>> continuously. I'm resending our posts so they remain in the list.
>>>>
>>>> Sorry that you have to read the mails backwards in time to make sanse of this.
>>>> In summary, you can read a pcap file continuously. Better if the
>>>> tcpdump is using -U, but there is a work-around if it is not. Thanks
>>>> Carter for all your work.
>>>>
>>>> sebas
>>>>
>>>> On Oct 7, 2013, at 10:21 AM, el draco <eldraco at gmail.com> wrote:
>>>>> Hi Carter!
>>>>>
>>>>> Ufff that's an issue. While I can run it with -U if I can choose to,
>>>>> right now we are using virtualbox
>>>>> -tracefile option to crate the pcap file. It means that when we start
>>>>> a vm, virtualbox automatically creates the pcap file for us. And
>>>>> unfortunately, virtualbox is not running the capture with -U.
>>>>>
>>>>> However I managed to find a workaround!! It is ugly... again.. but it works.
>>>>>
>>>>> So, first you have a non-buffered pcap file running (without -U), like
>>>>> the one from virtualbox -tracefile
>>>>> tcpdump -n -s0 -i eth0 -w /tmp/test.pcap -v
>>>>>
>>>>> Then you have a second tcpdump capture file, translating the packets
>>>>> and adding the buffer...
>>>>> tail -n 1000 -f /tmp/test.pcap | tcpdump -n -s0 -r - -U -w /tmp/test.pcap2
>>>>>
>>>>> This second pcap file is buffered, so now argus can read it without problems:
>>>>> ./bin/argus -f -r /tmp/test.pcap2 -w /tmp/test.argus -P 2040
>>>>>
>>>>> And you can see the argus data with ra continuously:
>>>>> ra -n -S localhost:2040
>>>>>
>>>>>
>>>>> With these commands I'm now able to read the argus data continuously
>>>>> from a pcap file.
>>>>>
>>>>> The only issue I had is this:
>>>>> If I stop the first pcap, then the second pcap exits (the tail command
>>>>> in fact), and the argus and ra commands keep running. However, if I
>>>>> start the first pcap and second pcap again... the argus is NOT getting
>>>>> the data, and the ra command shows nothing. If I restart the argus,
>>>>> everything works again.
>>>>> Not an issue for me, but I report it in case it helps.
>>>>>
>>>>> Thanks a lot for your work on this topic!
>>>>> cheers
>>>>> sebas
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Oct 7, 2013 at 1:07 PM, Carter Bullard <carter at qosient.com> wrote:
>>>>>> Hey sebas,
>>>>>> You MUST run tcpdump with the -U option to guarantee that the file
>>>>>> contains data on packet boundaries.
>>>>>>
>>>>>> Carter
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Oct 7, 2013, at 5:25 AM, el draco <eldraco at gmail.com> wrote:
>>>>>>
>>>>>>> Verification of argus-3.0.7.5
>>>>>>>
>>>>>>> 1st experiment
>>>>>>> ----------------------
>>>>>>> In one root console: tcpdump -n -s0 -i eth0 -v -w /tmp/test.pcap
>>>>>>>
>>>>>>> In another root console:
>>>>>>> ./bin/argus -h
>>>>>>> Argus Version 3.0.7.5
>>>>>>> ./bin/argus -f -r /tmp/test.pcap -w /tmp/test.argus
>>>>>>>
>>>>>>> After some reading, It exits with this error:
>>>>>>> ArgusError: 07 Oct 13 08:02:18.126142 ArgusGetPackets: pcap_dispatch()
>>>>>>> returned bogus savefile header
>>>>>>>
>>>>>>> The /tmp/test.argus file is created with the flows. And if I run it
>>>>>>> again, the flows are updated.
>>>>>>> However the argus process dies when it reaches the end of the file.
>>>>>>>
>>>>>>> 2nd experiment
>>>>>>> ----------------------
>>>>>>> I deleted the previous files.
>>>>>>> In one root console: tcpdump -n -s0 -i eth0 -v -w /tmp/test.pcap
>>>>>>> Then I stopped the tcpdump.
>>>>>>>
>>>>>>> In another root console: ./bin/argus -f -r /tmp/test.pcap -w /tmp/test.argus
>>>>>>> The argus keeps running without dying.
>>>>>>> The ra -r /tmp/test.argus reads the netfows and exits.
>>>>>>>
>>>>>>> If I start the tcpdump again, the argus runs for 10 seconds, and then
>>>>>>> exits with this error:
>>>>>>> ArgusError: 07 Oct 13 09:08:07.562844 ArgusGetPackets:
>>>>>>> pcap_dispatch() returned bogus savefile header
>>>>>>> The /tmp/test.argus file was updated with the new information and I
>>>>>>> could read it with ra.
>>>>>>>
>>>>>>> 3rd experiment
>>>>>>> ----------------------
>>>>>>> I deleted the previous files.
>>>>>>> In a root console: ./bin/argus -f -i eth0 -w /tmp/test.argus
>>>>>>> I can read the file with ra.
>>>>>>> This works fine.
>>>>>>>
>>>>>>> 4th experiment
>>>>>>> ----------------------
>>>>>>> I deleted the previous files.
>>>>>>> In one NON-ROOT console: tcpdump -n -s0 -i eth0 -v -w /tmp/test.pcap
>>>>>>> In another NON-ROOT console: ./bin/argus -r /home/sebas/test.pcap -f -P 2040
>>>>>>>
>>>>>>> After some reading, It exits with the same error
>>>>>>> ArgusError: 07 Oct 13 09:14:45.628054 ArgusGetPackets: pcap_dispatch()
>>>>>>> returned bogus savefile header
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Info about my setup:
>>>>>>> ii  libpcap-dev       1.4.0-2                        all
>>>>>>> development library for libpcap (transitional package)
>>>>>>> ii  libpcap0.8:i386  1.4.0-2                        i386
>>>>>>> system interface for user-level packet capture
>>>>>>>
>>>>>>>
>>>>>>> Tell me if you need me to test something else.
>>>>>>> cheers
>>>>>>> sebas
>>>>>>>
>>>>>>> On Sun, Oct 6, 2013 at 5:50 PM, Carter Bullard <carter at qosient.com> wrote:
>>>>>>>> So, this works, but still needs some of the exceptions worked out,
>>>>>>>> such as if you rename or delete the packet file, but it should work....
>>>>>>>> Please give this version of argus a trial, and send any feedback,
>>>>>>>> bugs, additional features needed, etc soon !!!
>>>>>>>>
>>>>>>>> The man page for argus has been updated for this feature yet,
>>>>>>>> so mostly done.
>>>>>>>>
>>>>>>>> This version has some other changes, so this will serve as general
>>>>>>>> testing as well.  The other changes are basically idle interface
>>>>>>>> timer issues, and shouldn't be an issue.
>>>>>>>>
>>>>>>>> Use the -f option, along with the -r packetfile option, just like tail.
>>>>>>>> This may work with pipes as well, but I didn't test.
>>>>>>>>
>>>>>>>> Carter
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Oct 6, 2013, at 1:29 PM, el draco <eldraco at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Carter, We are waiting for it! Yes!!!
>>>>>>>>>
>>>>>>>>> We are now using tshark and offline argus..., so if you manage to make it work,
>>>>>>>>> we will be more than happy to test it.
>>>>>>>>>
>>>>>>>>> Just tell me what you need from us.
>>>>>>>>>
>>>>>>>>> Thanks a lot!
>>>>>>>>>
>>>>
>>>
>>
>
>