Argus 3.0 / 32-bit overflow / backward compatibility

Thu Sep 22 15:36:52 EDT 2011

Hey Carter,

Thanks for the quick response and provided insight. You are right, with 
the correct configuration 32-bit counters should be sufficient for the 
good majority of the cases. However, setting the 
ARGUS_FLOW_STATUS_INTERVAL to a very low value would be an unnecessary 
overhead for most flows, I think. So, how does an alternative 
configuration feature (or perhaps this is something that already exists 
and I have missed?) where argus will produce "flow status" on its own 
when it has a byte counter that is ready to overflow? Such an option 
should allow for less aggressively outputting to disk while making it 
harder to loose data by having a poorly configured argus.

As for the backwards compatibility, unfortunately I did not have much 
luck even with the newest development release. I also ran it across a 
number of files in the argus archive I have available. I am attaching 
one of the smallest argus 2.x files that I have (unsuccessfully) tested 
against.

Thanks,
Ilija

On 11-09-22 02:29 PM, Carter Bullard wrote:
> Hey Ilija,
> Argus itself doesn't support 64-bit counters, but the clients do.  It is the difference between flow
> data generation and flow data processing.
>
> We do not recommend running argus such that it holds flow information for very large periods of time.
> I run with ARGUS_FLOW_STATUS_INTERVAL set to 1s or 5s in every installation I do, and as
> a result the 32-bit counter strategy has worked quite well.  For 10-100 Gbps interface monitoring,
> I think the 32-bit value is still the way to go, with the clients aggregating the status reports to generate the
> final flow record, where we do support 64-bit reporting.
>
> If you think this is a flaw, then lets discuss.  Adding 64-bit counters will increase the flow cache
> for every flow tracked by 48 bytes (src and dst pkt, byte, and appbyte counters) and for TCP we'll need to
> add another 64 bytes or so, as we have a lot of other counters.  Can be a huge deal if we're tracking
> 1M flows, but our current cache size is pretty big, when you add jitter and other features.
>
> Not sure what is up with your argus-2.0 data, as we should be able to read it with out issue.
> argus-clients-3.0.5.19, in the developers section, may be able to read it fine, if there was a but.
> If you can share the 2.x file, I'll check out what might be the issue.
>
> Carter
>
> On Sep 22, 2011, at 1:52 PM, Ilija Baniski wrote:
>
>> Hello all,
>>
>> I am doing some testing prior to upgrading to 3.0. 64-bit byte counters are an important reason for the upgrade, while the ability of the 3.0 clients to read argus 2.0 data make the switch that much sweeter. However, I haven't been able to get either of these two features to work. I have the latest argus-3.0.4 and argus-clients-3.0.4.1 built. I know these two are not very closely related, but I'll ask both questions here.
>>
>> I read that these two features should be supported here:
>> http://permalink.gmane.org/gmane.network.argus/3601
>> http://article.gmane.org/gmane.network.argus/3782
>>
>> 1.
>> To test the 4GB byte overflow I performed a ~5GB file transfer twice: once with STATUS_INTERVAL=60 and then with STATUS_INTERVAL=600. These were the results:
>>
>> # ./ra -Xnr /opt/data/argus/var/log/argus.out - host 192.168.111.39
>>    11:45:51.241371  e d       tcp    192.168.111.137.49460     ->      192.168.111.39.22       674666  691670398   CON
>>    11:46:51.241496  e d       tcp    192.168.111.137.49460     ->      192.168.111.39.22       728665  746638654   CON
>>    11:47:51.241563  e d       tcp    192.168.111.137.49460     ->      192.168.111.39.22       729780  748390504   CON
>>    11:48:51.241597  e d       tcp    192.168.111.137.49460     ->      192.168.111.39.22       733293  752190798   CON
>>    11:49:51.241631  e d       tcp    192.168.111.137.49460     ->      192.168.111.39.22       733537  752519850   CON
>>    11:50:51.241659  e d       tcp    192.168.111.137.49460     ->      192.168.111.39.22       734255  752514878   CON
>>    11:51:51.241729  e d       tcp    192.168.111.137.49460     ->      192.168.111.39.22       732836  751412980   CON
>>    11:52:51.241770  e d       tcp    192.168.111.137.49460     ->      192.168.111.39.22       374739  384106534   FIN
>> # ./racluster -Xnr /opt/data/argus/var/log/argus.out - port 49460
>>    11:45:51.241371  e d       tcp    192.168.111.137.49460     ->      192.168.111.39.22      5441771 5579444596   FIN
>>
>> Which shows the transfer of the ~5GB file. Now with only a single interval for the transfer:
>>
>> # ./ra -Xnr /opt/data/argus/var/log/argus.out - host 192.168.111.39
>>    11:33:23.249123  e d       tcp    192.168.111.137.47119     ->      192.168.111.39.22      5443943 1284774452   FIN
>> # ./racluster -Xnr /opt/data/argus/var/log/argus.out - port 47119
>>    11:33:23.249123  e d       tcp    192.168.111.137.47119     ->      192.168.111.39.22      5443943 1284774452   FIN
>>
>> Which shows that the byte count has wrapped after 4GB (1284774452 + 4x1024x1024x1024 = 5579741748).
>>
>> So I guess the question is, what am I doing wrong? Do I need to compile argus in a certain way or configure it differently? Or perhaps this is related to the clients (unlikely since racluster seems to have no problem showing the 5GB if the data is there).
>>
>>
>> 2.
>> To check the backward compatibility I used the ra tool from the argus-clients-3.0.4.1 to try and read a file written by argus 2.0. Here are some output snippets of me trying to read the file with both, an old client and new one:
>>
>> The old:
>> # ra -h
>> Ra Version 2.0.6.fixes.1
>> ...
>> # ra -r argus.2011-09-13-07-00-01.gz | head -2
>> 09-07-11 11:18:07.535387           man                    10.2.1.37  v2.0                                     1 0          0        0         0            0           STA
>> 09-13-11 02:49:01.914562           tcp                    10.2.2.86.8280        ->                 69.184.252.19.8292       3        6         384          435         CON
>>
>> and the new:
>> # ./ra -Xh
>> Ra Version 3.0.4.1
>> ...
>> # ./ra -Xr argus.2011-09-13-07-00-01.gz
>> <no output at all>
>>
>> So back to the same questions: what am I doing wrong/how can I use the new clients to read the old data? In my random attempts to make this happen I did try to unzip the file first, but that made no difference.
>>
>>
>> Thanks for any help,
>> Ilija
>>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: argus-sample.out
Type: application/octet-stream
Size: 4784 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20110922/3c7f442b/attachment.obj>