Argus 3.0 / 32-bit overflow / backward compatibility
Carter Bullard
carter at qosient.com
Thu Sep 22 22:52:36 EDT 2011
Hey Ilija,
I've checked every version of argus-2.x that I have, and non of the client programs
I have can read the argus-2 file that you sent. The starting MAR record in your
file is too big. Its 144 bytes long when its suppose to be 128 bytes.
Did you compile this on a 64-bit machine? Reason I ask, argus-2.x was not
64-bit ready, and compiling on 64-bit would cause struct size problems.
Clients compiled on the same architecture would be able to read flow data
generated by a 'broken' argus, as they would have the same size problems,
but the resulting files would not be portable, even to 32-bit argus-2.x programs.
That is my best guess as to why your argus-2.x files aren't working with argus-3.x,
but its just a guess.
One thing you could do is write out the argus-2.x data in ascii, using your existing
argus-2.x clients, and then read convert that ascii back to argus-3.0 flow data
using raconvert()?
Carter
On Sep 22, 2011, at 3:36 PM, Ilija Baniski wrote:
> Hey Carter,
>
> Thanks for the quick response and provided insight. You are right, with the correct configuration 32-bit counters should be sufficient for the good majority of the cases. However, setting the ARGUS_FLOW_STATUS_INTERVAL to a very low value would be an unnecessary overhead for most flows, I think. So, how does an alternative configuration feature (or perhaps this is something that already exists and I have missed?) where argus will produce "flow status" on its own when it has a byte counter that is ready to overflow? Such an option should allow for less aggressively outputting to disk while making it harder to loose data by having a poorly configured argus.
>
> As for the backwards compatibility, unfortunately I did not have much luck even with the newest development release. I also ran it across a number of files in the argus archive I have available. I am attaching one of the smallest argus 2.x files that I have (unsuccessfully) tested against.
>
> Thanks,
> Ilija
>
> On 11-09-22 02:29 PM, Carter Bullard wrote:
>> Hey Ilija,
>> Argus itself doesn't support 64-bit counters, but the clients do. It is the difference between flow
>> data generation and flow data processing.
>>
>> We do not recommend running argus such that it holds flow information for very large periods of time.
>> I run with ARGUS_FLOW_STATUS_INTERVAL set to 1s or 5s in every installation I do, and as
>> a result the 32-bit counter strategy has worked quite well. For 10-100 Gbps interface monitoring,
>> I think the 32-bit value is still the way to go, with the clients aggregating the status reports to generate the
>> final flow record, where we do support 64-bit reporting.
>>
>> If you think this is a flaw, then lets discuss. Adding 64-bit counters will increase the flow cache
>> for every flow tracked by 48 bytes (src and dst pkt, byte, and appbyte counters) and for TCP we'll need to
>> add another 64 bytes or so, as we have a lot of other counters. Can be a huge deal if we're tracking
>> 1M flows, but our current cache size is pretty big, when you add jitter and other features.
>>
>> Not sure what is up with your argus-2.0 data, as we should be able to read it with out issue.
>> argus-clients-3.0.5.19, in the developers section, may be able to read it fine, if there was a but.
>> If you can share the 2.x file, I'll check out what might be the issue.
>>
>> Carter
>>
>> On Sep 22, 2011, at 1:52 PM, Ilija Baniski wrote:
>>
>>> Hello all,
>>>
>>> I am doing some testing prior to upgrading to 3.0. 64-bit byte counters are an important reason for the upgrade, while the ability of the 3.0 clients to read argus 2.0 data make the switch that much sweeter. However, I haven't been able to get either of these two features to work. I have the latest argus-3.0.4 and argus-clients-3.0.4.1 built. I know these two are not very closely related, but I'll ask both questions here.
>>>
>>> I read that these two features should be supported here:
>>> http://permalink.gmane.org/gmane.network.argus/3601
>>> http://article.gmane.org/gmane.network.argus/3782
>>>
>>> 1.
>>> To test the 4GB byte overflow I performed a ~5GB file transfer twice: once with STATUS_INTERVAL=60 and then with STATUS_INTERVAL=600. These were the results:
>>>
>>> # ./ra -Xnr /opt/data/argus/var/log/argus.out - host 192.168.111.39
>>> 11:45:51.241371 e d tcp 192.168.111.137.49460 -> 192.168.111.39.22 674666 691670398 CON
>>> 11:46:51.241496 e d tcp 192.168.111.137.49460 -> 192.168.111.39.22 728665 746638654 CON
>>> 11:47:51.241563 e d tcp 192.168.111.137.49460 -> 192.168.111.39.22 729780 748390504 CON
>>> 11:48:51.241597 e d tcp 192.168.111.137.49460 -> 192.168.111.39.22 733293 752190798 CON
>>> 11:49:51.241631 e d tcp 192.168.111.137.49460 -> 192.168.111.39.22 733537 752519850 CON
>>> 11:50:51.241659 e d tcp 192.168.111.137.49460 -> 192.168.111.39.22 734255 752514878 CON
>>> 11:51:51.241729 e d tcp 192.168.111.137.49460 -> 192.168.111.39.22 732836 751412980 CON
>>> 11:52:51.241770 e d tcp 192.168.111.137.49460 -> 192.168.111.39.22 374739 384106534 FIN
>>> # ./racluster -Xnr /opt/data/argus/var/log/argus.out - port 49460
>>> 11:45:51.241371 e d tcp 192.168.111.137.49460 -> 192.168.111.39.22 5441771 5579444596 FIN
>>>
>>> Which shows the transfer of the ~5GB file. Now with only a single interval for the transfer:
>>>
>>> # ./ra -Xnr /opt/data/argus/var/log/argus.out - host 192.168.111.39
>>> 11:33:23.249123 e d tcp 192.168.111.137.47119 -> 192.168.111.39.22 5443943 1284774452 FIN
>>> # ./racluster -Xnr /opt/data/argus/var/log/argus.out - port 47119
>>> 11:33:23.249123 e d tcp 192.168.111.137.47119 -> 192.168.111.39.22 5443943 1284774452 FIN
>>>
>>> Which shows that the byte count has wrapped after 4GB (1284774452 + 4x1024x1024x1024 = 5579741748).
>>>
>>> So I guess the question is, what am I doing wrong? Do I need to compile argus in a certain way or configure it differently? Or perhaps this is related to the clients (unlikely since racluster seems to have no problem showing the 5GB if the data is there).
>>>
>>>
>>> 2.
>>> To check the backward compatibility I used the ra tool from the argus-clients-3.0.4.1 to try and read a file written by argus 2.0. Here are some output snippets of me trying to read the file with both, an old client and new one:
>>>
>>> The old:
>>> # ra -h
>>> Ra Version 2.0.6.fixes.1
>>> ...
>>> # ra -r argus.2011-09-13-07-00-01.gz | head -2
>>> 09-07-11 11:18:07.535387 man 10.2.1.37 v2.0 1 0 0 0 0 0 STA
>>> 09-13-11 02:49:01.914562 tcp 10.2.2.86.8280 -> 69.184.252.19.8292 3 6 384 435 CON
>>>
>>> and the new:
>>> # ./ra -Xh
>>> Ra Version 3.0.4.1
>>> ...
>>> # ./ra -Xr argus.2011-09-13-07-00-01.gz
>>> <no output at all>
>>>
>>> So back to the same questions: what am I doing wrong/how can I use the new clients to read the old data? In my random attempts to make this happen I did try to unzip the file first, but that made no difference.
>>>
>>>
>>> Thanks for any help,
>>> Ilija
>>>
>
> <argus-sample.out>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4367 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20110922/02bff049/attachment.bin>
More information about the argus
mailing list