Argus 3.0 / 32-bit overflow / backward compatibility
Ilija Baniski
ilija.baniski at esentire.com
Thu Sep 22 15:36:52 EDT 2011
Hey Carter,
Thanks for the quick response and provided insight. You are right, with
the correct configuration 32-bit counters should be sufficient for the
good majority of the cases. However, setting the
ARGUS_FLOW_STATUS_INTERVAL to a very low value would be an unnecessary
overhead for most flows, I think. So, how does an alternative
configuration feature (or perhaps this is something that already exists
and I have missed?) where argus will produce "flow status" on its own
when it has a byte counter that is ready to overflow? Such an option
should allow for less aggressively outputting to disk while making it
harder to loose data by having a poorly configured argus.
As for the backwards compatibility, unfortunately I did not have much
luck even with the newest development release. I also ran it across a
number of files in the argus archive I have available. I am attaching
one of the smallest argus 2.x files that I have (unsuccessfully) tested
against.
Thanks,
Ilija
On 11-09-22 02:29 PM, Carter Bullard wrote:
> Hey Ilija,
> Argus itself doesn't support 64-bit counters, but the clients do. It is the difference between flow
> data generation and flow data processing.
>
> We do not recommend running argus such that it holds flow information for very large periods of time.
> I run with ARGUS_FLOW_STATUS_INTERVAL set to 1s or 5s in every installation I do, and as
> a result the 32-bit counter strategy has worked quite well. For 10-100 Gbps interface monitoring,
> I think the 32-bit value is still the way to go, with the clients aggregating the status reports to generate the
> final flow record, where we do support 64-bit reporting.
>
> If you think this is a flaw, then lets discuss. Adding 64-bit counters will increase the flow cache
> for every flow tracked by 48 bytes (src and dst pkt, byte, and appbyte counters) and for TCP we'll need to
> add another 64 bytes or so, as we have a lot of other counters. Can be a huge deal if we're tracking
> 1M flows, but our current cache size is pretty big, when you add jitter and other features.
>
> Not sure what is up with your argus-2.0 data, as we should be able to read it with out issue.
> argus-clients-3.0.5.19, in the developers section, may be able to read it fine, if there was a but.
> If you can share the 2.x file, I'll check out what might be the issue.
>
> Carter
>
> On Sep 22, 2011, at 1:52 PM, Ilija Baniski wrote:
>
>> Hello all,
>>
>> I am doing some testing prior to upgrading to 3.0. 64-bit byte counters are an important reason for the upgrade, while the ability of the 3.0 clients to read argus 2.0 data make the switch that much sweeter. However, I haven't been able to get either of these two features to work. I have the latest argus-3.0.4 and argus-clients-3.0.4.1 built. I know these two are not very closely related, but I'll ask both questions here.
>>
>> I read that these two features should be supported here:
>> http://permalink.gmane.org/gmane.network.argus/3601
>> http://article.gmane.org/gmane.network.argus/3782
>>
>> 1.
>> To test the 4GB byte overflow I performed a ~5GB file transfer twice: once with STATUS_INTERVAL=60 and then with STATUS_INTERVAL=600. These were the results:
>>
>> # ./ra -Xnr /opt/data/argus/var/log/argus.out - host 192.168.111.39
>> 11:45:51.241371 e d tcp 192.168.111.137.49460 -> 192.168.111.39.22 674666 691670398 CON
>> 11:46:51.241496 e d tcp 192.168.111.137.49460 -> 192.168.111.39.22 728665 746638654 CON
>> 11:47:51.241563 e d tcp 192.168.111.137.49460 -> 192.168.111.39.22 729780 748390504 CON
>> 11:48:51.241597 e d tcp 192.168.111.137.49460 -> 192.168.111.39.22 733293 752190798 CON
>> 11:49:51.241631 e d tcp 192.168.111.137.49460 -> 192.168.111.39.22 733537 752519850 CON
>> 11:50:51.241659 e d tcp 192.168.111.137.49460 -> 192.168.111.39.22 734255 752514878 CON
>> 11:51:51.241729 e d tcp 192.168.111.137.49460 -> 192.168.111.39.22 732836 751412980 CON
>> 11:52:51.241770 e d tcp 192.168.111.137.49460 -> 192.168.111.39.22 374739 384106534 FIN
>> # ./racluster -Xnr /opt/data/argus/var/log/argus.out - port 49460
>> 11:45:51.241371 e d tcp 192.168.111.137.49460 -> 192.168.111.39.22 5441771 5579444596 FIN
>>
>> Which shows the transfer of the ~5GB file. Now with only a single interval for the transfer:
>>
>> # ./ra -Xnr /opt/data/argus/var/log/argus.out - host 192.168.111.39
>> 11:33:23.249123 e d tcp 192.168.111.137.47119 -> 192.168.111.39.22 5443943 1284774452 FIN
>> # ./racluster -Xnr /opt/data/argus/var/log/argus.out - port 47119
>> 11:33:23.249123 e d tcp 192.168.111.137.47119 -> 192.168.111.39.22 5443943 1284774452 FIN
>>
>> Which shows that the byte count has wrapped after 4GB (1284774452 + 4x1024x1024x1024 = 5579741748).
>>
>> So I guess the question is, what am I doing wrong? Do I need to compile argus in a certain way or configure it differently? Or perhaps this is related to the clients (unlikely since racluster seems to have no problem showing the 5GB if the data is there).
>>
>>
>> 2.
>> To check the backward compatibility I used the ra tool from the argus-clients-3.0.4.1 to try and read a file written by argus 2.0. Here are some output snippets of me trying to read the file with both, an old client and new one:
>>
>> The old:
>> # ra -h
>> Ra Version 2.0.6.fixes.1
>> ...
>> # ra -r argus.2011-09-13-07-00-01.gz | head -2
>> 09-07-11 11:18:07.535387 man 10.2.1.37 v2.0 1 0 0 0 0 0 STA
>> 09-13-11 02:49:01.914562 tcp 10.2.2.86.8280 -> 69.184.252.19.8292 3 6 384 435 CON
>>
>> and the new:
>> # ./ra -Xh
>> Ra Version 3.0.4.1
>> ...
>> # ./ra -Xr argus.2011-09-13-07-00-01.gz
>> <no output at all>
>>
>> So back to the same questions: what am I doing wrong/how can I use the new clients to read the old data? In my random attempts to make this happen I did try to unzip the file first, but that made no difference.
>>
>>
>> Thanks for any help,
>> Ilija
>>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: argus-sample.out
Type: application/octet-stream
Size: 4784 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20110922/3c7f442b/attachment.obj>
More information about the argus
mailing list