Do you know how to read a pcap file continuously?

[el draco]

Hi Carter

The basic stuff we need is to read a common pcap file (no -U in
tcpdump) continuously with argus.

We are not expecting argus to survive a file deletion.
If the pcap file is not there or it is deleted, we start over, no
problem for us.
If the file is not there when argus starts, argus can exit.

Does this make it easier?
thanks!
sebas


On Mon, Oct 7, 2013 at 7:10 PM, Carter Bullard <carter at qosient.com> wrote:
> Hey Sebas,
> I can recover from the truncated file error condition, but
> its a bit expensive, pcap_close(), look for a change in size
> then pcap_open() lseek() …, but its really expensive.
>
> If you can describe the conditions that you expect, like
> should survive file deletion, etc…  Then we can wrap this
> thing up pretty quick.
>
> Carter
>
>
>
> On Oct 7, 2013, at 11:18 AM, el draco <eldraco at gmail.com> wrote:
>
>> Hi list, we have been playing a little bit with reading a pcap file
>> continuously. I'm resending our posts so they remain in the list.
>>
>> Sorry that you have to read the mails backwards in time to make sanse of this.
>> In summary, you can read a pcap file continuously. Better if the
>> tcpdump is using -U, but there is a work-around if it is not. Thanks
>> Carter for all your work.
>>
>> sebas
>>
>> On Oct 7, 2013, at 10:21 AM, el draco <eldraco at gmail.com> wrote:
>>> Hi Carter!
>>>
>>> Ufff that's an issue. While I can run it with -U if I can choose to,
>>> right now we are using virtualbox
>>> -tracefile option to crate the pcap file. It means that when we start
>>> a vm, virtualbox automatically creates the pcap file for us. And
>>> unfortunately, virtualbox is not running the capture with -U.
>>>
>>> However I managed to find a workaround!! It is ugly... again.. but it works.
>>>
>>> So, first you have a non-buffered pcap file running (without -U), like
>>> the one from virtualbox -tracefile
>>> tcpdump -n -s0 -i eth0 -w /tmp/test.pcap -v
>>>
>>> Then you have a second tcpdump capture file, translating the packets
>>> and adding the buffer...
>>> tail -n 1000 -f /tmp/test.pcap | tcpdump -n -s0 -r - -U -w /tmp/test.pcap2
>>>
>>> This second pcap file is buffered, so now argus can read it without problems:
>>> ./bin/argus -f -r /tmp/test.pcap2 -w /tmp/test.argus -P 2040
>>>
>>> And you can see the argus data with ra continuously:
>>> ra -n -S localhost:2040
>>>
>>>
>>> With these commands I'm now able to read the argus data continuously
>>> from a pcap file.
>>>
>>> The only issue I had is this:
>>> If I stop the first pcap, then the second pcap exits (the tail command
>>> in fact), and the argus and ra commands keep running. However, if I
>>> start the first pcap and second pcap again... the argus is NOT getting
>>> the data, and the ra command shows nothing. If I restart the argus,
>>> everything works again.
>>> Not an issue for me, but I report it in case it helps.
>>>
>>> Thanks a lot for your work on this topic!
>>> cheers
>>> sebas
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Oct 7, 2013 at 1:07 PM, Carter Bullard <carter at qosient.com> wrote:
>>>> Hey sebas,
>>>> You MUST run tcpdump with the -U option to guarantee that the file
>>>> contains data on packet boundaries.
>>>>
>>>> Carter
>>>>
>>>>
>>>>
>>>> On Oct 7, 2013, at 5:25 AM, el draco <eldraco at gmail.com> wrote:
>>>>
>>>>> Verification of argus-3.0.7.5
>>>>>
>>>>> 1st experiment
>>>>> ----------------------
>>>>> In one root console: tcpdump -n -s0 -i eth0 -v -w /tmp/test.pcap
>>>>>
>>>>> In another root console:
>>>>> ./bin/argus -h
>>>>> Argus Version 3.0.7.5
>>>>> ./bin/argus -f -r /tmp/test.pcap -w /tmp/test.argus
>>>>>
>>>>> After some reading, It exits with this error:
>>>>> ArgusError: 07 Oct 13 08:02:18.126142 ArgusGetPackets: pcap_dispatch()
>>>>> returned bogus savefile header
>>>>>
>>>>> The /tmp/test.argus file is created with the flows. And if I run it
>>>>> again, the flows are updated.
>>>>> However the argus process dies when it reaches the end of the file.
>>>>>
>>>>> 2nd experiment
>>>>> ----------------------
>>>>> I deleted the previous files.
>>>>> In one root console: tcpdump -n -s0 -i eth0 -v -w /tmp/test.pcap
>>>>> Then I stopped the tcpdump.
>>>>>
>>>>> In another root console: ./bin/argus -f -r /tmp/test.pcap -w /tmp/test.argus
>>>>> The argus keeps running without dying.
>>>>> The ra -r /tmp/test.argus reads the netfows and exits.
>>>>>
>>>>> If I start the tcpdump again, the argus runs for 10 seconds, and then
>>>>> exits with this error:
>>>>>  ArgusError: 07 Oct 13 09:08:07.562844 ArgusGetPackets:
>>>>> pcap_dispatch() returned bogus savefile header
>>>>> The /tmp/test.argus file was updated with the new information and I
>>>>> could read it with ra.
>>>>>
>>>>> 3rd experiment
>>>>> ----------------------
>>>>> I deleted the previous files.
>>>>> In a root console: ./bin/argus -f -i eth0 -w /tmp/test.argus
>>>>> I can read the file with ra.
>>>>> This works fine.
>>>>>
>>>>> 4th experiment
>>>>> ----------------------
>>>>> I deleted the previous files.
>>>>> In one NON-ROOT console: tcpdump -n -s0 -i eth0 -v -w /tmp/test.pcap
>>>>> In another NON-ROOT console: ./bin/argus -r /home/sebas/test.pcap -f -P 2040
>>>>>
>>>>> After some reading, It exits with the same error
>>>>> ArgusError: 07 Oct 13 09:14:45.628054 ArgusGetPackets: pcap_dispatch()
>>>>> returned bogus savefile header
>>>>>
>>>>>
>>>>>
>>>>> Info about my setup:
>>>>> ii  libpcap-dev       1.4.0-2                        all
>>>>> development library for libpcap (transitional package)
>>>>> ii  libpcap0.8:i386  1.4.0-2                        i386
>>>>> system interface for user-level packet capture
>>>>>
>>>>>
>>>>> Tell me if you need me to test something else.
>>>>> cheers
>>>>> sebas
>>>>>
>>>>> On Sun, Oct 6, 2013 at 5:50 PM, Carter Bullard <carter at qosient.com> wrote:
>>>>>> So, this works, but still needs some of the exceptions worked out,
>>>>>> such as if you rename or delete the packet file, but it should work....
>>>>>> Please give this version of argus a trial, and send any feedback,
>>>>>> bugs, additional features needed, etc soon !!!
>>>>>>
>>>>>> The man page for argus has been updated for this feature yet,
>>>>>> so mostly done.
>>>>>>
>>>>>> This version has some other changes, so this will serve as general
>>>>>> testing as well.  The other changes are basically idle interface
>>>>>> timer issues, and shouldn't be an issue.
>>>>>>
>>>>>> Use the -f option, along with the -r packetfile option, just like tail.
>>>>>> This may work with pipes as well, but I didn't test.
>>>>>>
>>>>>> Carter
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Oct 6, 2013, at 1:29 PM, el draco <eldraco at gmail.com> wrote:
>>>>>>
>>>>>>> Hi Carter, We are waiting for it! Yes!!!
>>>>>>>
>>>>>>> We are now using tshark and offline argus..., so if you manage to make it work,
>>>>>>> we will be more than happy to test it.
>>>>>>>
>>>>>>> Just tell me what you need from us.
>>>>>>>
>>>>>>> Thanks a lot!
>>>>>>>
>>
>