Argus 3.0 / 32-bit overflow / backward compatibility
Carter Bullard
carter at qosient.com
Thu Sep 22 22:38:15 EDT 2011
Hey Llija,
Not so sure that about the unnecessary overhead. Just depends on what you're after.
With long status intervals, argus can't provide near real-time information about flow
presence, so if getting indications of flows quickly is important, then the overhead is required.
The deal is that 93% of all Internet flows, or at least up until about 2 years ago, lived
about 1.73 seconds. So, having a status timer much much longer than that really is a
waste of argus memory. But a 300 or 600 second status timer, puts such a memory load
on argus, in the general case, so I suggest having shorter status timers than longer ones.
I can put in a "flush on overflow" state in the modeler, but ….., one of the properties that
we put in the later argus's, was that the output flow data stream MUST be sorted in start time
order. This becomes an important property when you're doing what is called "streaming analytics".
So I'll have to put in some cache management exceptions to handle when a flow counter
overflows. The problem is that the flow cache needs to be flushed, but its before its status timer,
and there are probably many other flows that are ahead if this flow cache, so we'll need to deal
with that.
Thanks for the argus-2.x file. I'll look at it tonight.
Carter
On Sep 22, 2011, at 3:36 PM, Ilija Baniski wrote:
> Hey Carter,
>
> Thanks for the quick response and provided insight. You are right, with the correct configuration 32-bit counters should be sufficient for the good majority of the cases. However, setting the ARGUS_FLOW_STATUS_INTERVAL to a very low value would be an unnecessary overhead for most flows, I think. So, how does an alternative configuration feature (or perhaps this is something that already exists and I have missed?) where argus will produce "flow status" on its own when it has a byte counter that is ready to overflow? Such an option should allow for less aggressively outputting to disk while making it harder to loose data by having a poorly configured argus.
>
> As for the backwards compatibility, unfortunately I did not have much luck even with the newest development release. I also ran it across a number of files in the argus archive I have available. I am attaching one of the smallest argus 2.x files that I have (unsuccessfully) tested against.
>
> Thanks,
> Ilija
>
> On 11-09-22 02:29 PM, Carter Bullard wrote:
>> Hey Ilija,
>> Argus itself doesn't support 64-bit counters, but the clients do. It is the difference between flow
>> data generation and flow data processing.
>>
>> We do not recommend running argus such that it holds flow information for very large periods of time.
>> I run with ARGUS_FLOW_STATUS_INTERVAL set to 1s or 5s in every installation I do, and as
>> a result the 32-bit counter strategy has worked quite well. For 10-100 Gbps interface monitoring,
>> I think the 32-bit value is still the way to go, with the clients aggregating the status reports to generate the
>> final flow record, where we do support 64-bit reporting.
>>
>> If you think this is a flaw, then lets discuss. Adding 64-bit counters will increase the flow cache
>> for every flow tracked by 48 bytes (src and dst pkt, byte, and appbyte counters) and for TCP we'll need to
>> add another 64 bytes or so, as we have a lot of other counters. Can be a huge deal if we're tracking
>> 1M flows, but our current cache size is pretty big, when you add jitter and other features.
>>
>> Not sure what is up with your argus-2.0 data, as we should be able to read it with out issue.
>> argus-clients-3.0.5.19, in the developers section, may be able to read it fine, if there was a but.
>> If you can share the 2.x file, I'll check out what might be the issue.
>>
>> Carter
>>
>> On Sep 22, 2011, at 1:52 PM, Ilija Baniski wrote:
>>
>>> Hello all,
>>>
>>> I am doing some testing prior to upgrading to 3.0. 64-bit byte counters are an important reason for the upgrade, while the ability of the 3.0 clients to read argus 2.0 data make the switch that much sweeter. However, I haven't been able to get either of these two features to work. I have the latest argus-3.0.4 and argus-clients-3.0.4.1 built. I know these two are not very closely related, but I'll ask both questions here.
>>>
>>> I read that these two features should be supported here:
>>> http://permalink.gmane.org/gmane.network.argus/3601
>>> http://article.gmane.org/gmane.network.argus/3782
>>>
>>> 1.
>>> To test the 4GB byte overflow I performed a ~5GB file transfer twice: once with STATUS_INTERVAL=60 and then with STATUS_INTERVAL=600. These were the results:
>>>
>>> # ./ra -Xnr /opt/data/argus/var/log/argus.out - host 192.168.111.39
>>> 11:45:51.241371 e d tcp 192.168.111.137.49460 -> 192.168.111.39.22 674666 691670398 CON
>>> 11:46:51.241496 e d tcp 192.168.111.137.49460 -> 192.168.111.39.22 728665 746638654 CON
>>> 11:47:51.241563 e d tcp 192.168.111.137.49460 -> 192.168.111.39.22 729780 748390504 CON
>>> 11:48:51.241597 e d tcp 192.168.111.137.49460 -> 192.168.111.39.22 733293 752190798 CON
>>> 11:49:51.241631 e d tcp 192.168.111.137.49460 -> 192.168.111.39.22 733537 752519850 CON
>>> 11:50:51.241659 e d tcp 192.168.111.137.49460 -> 192.168.111.39.22 734255 752514878 CON
>>> 11:51:51.241729 e d tcp 192.168.111.137.49460 -> 192.168.111.39.22 732836 751412980 CON
>>> 11:52:51.241770 e d tcp 192.168.111.137.49460 -> 192.168.111.39.22 374739 384106534 FIN
>>> # ./racluster -Xnr /opt/data/argus/var/log/argus.out - port 49460
>>> 11:45:51.241371 e d tcp 192.168.111.137.49460 -> 192.168.111.39.22 5441771 5579444596 FIN
>>>
>>> Which shows the transfer of the ~5GB file. Now with only a single interval for the transfer:
>>>
>>> # ./ra -Xnr /opt/data/argus/var/log/argus.out - host 192.168.111.39
>>> 11:33:23.249123 e d tcp 192.168.111.137.47119 -> 192.168.111.39.22 5443943 1284774452 FIN
>>> # ./racluster -Xnr /opt/data/argus/var/log/argus.out - port 47119
>>> 11:33:23.249123 e d tcp 192.168.111.137.47119 -> 192.168.111.39.22 5443943 1284774452 FIN
>>>
>>> Which shows that the byte count has wrapped after 4GB (1284774452 + 4x1024x1024x1024 = 5579741748).
>>>
>>> So I guess the question is, what am I doing wrong? Do I need to compile argus in a certain way or configure it differently? Or perhaps this is related to the clients (unlikely since racluster seems to have no problem showing the 5GB if the data is there).
>>>
>>>
>>> 2.
>>> To check the backward compatibility I used the ra tool from the argus-clients-3.0.4.1 to try and read a file written by argus 2.0. Here are some output snippets of me trying to read the file with both, an old client and new one:
>>>
>>> The old:
>>> # ra -h
>>> Ra Version 2.0.6.fixes.1
>>> ...
>>> # ra -r argus.2011-09-13-07-00-01.gz | head -2
>>> 09-07-11 11:18:07.535387 man 10.2.1.37 v2.0 1 0 0 0 0 0 STA
>>> 09-13-11 02:49:01.914562 tcp 10.2.2.86.8280 -> 69.184.252.19.8292 3 6 384 435 CON
>>>
>>> and the new:
>>> # ./ra -Xh
>>> Ra Version 3.0.4.1
>>> ...
>>> # ./ra -Xr argus.2011-09-13-07-00-01.gz
>>> <no output at all>
>>>
>>> So back to the same questions: what am I doing wrong/how can I use the new clients to read the old data? In my random attempts to make this happen I did try to unzip the file first, but that made no difference.
>>>
>>>
>>> Thanks for any help,
>>> Ilija
>>>
>
> <argus-sample.out>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4367 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20110922/1c20ef49/attachment.bin>
More information about the argus
mailing list