[ARGUS] ARGUSBug Argus server occasionally generates an invalid meter DSR in a flow record
Carter Bullard
carter at qosient.com
Mon Mar 18 15:17:06 EDT 2019
Hey Joel,
There are a lot of opportunities for us to race a flow output record … we mark it to write out, we write it out a flow record, we mark it as written, with it in a number of queues, the active queue a timeout queue, a pending queue … that type of thing.
So I’m thinking that we wrote out the record, and zero'd it out (no metrics), and put it in a timeout queue … now, if a packet for it shows up, we move it to the status queue, and then update the record. If somehow the queue needs to be flushed, we may not check if there are metrics, we just shove it to the output processor, so its pending, but the packets didn’t get tallied by the time we prepare the output record.
Just a guess …. I’m going to compare with the commercial code later today, which is really different on this front, to see if we just don’t generate an output record in this case and return it to a timeout queue.
If you get records with zero src and dst pkts, look to see if there were flows with the same flow ids, and lets see if there was an output record just seconds before this one was generated.
Carter
> On Mar 18, 2019, at 3:07 PM, Reed, Joel <reedjw at ornl.gov> wrote:
>
> Hi Carter,
>
> Sorry for the delay in responding. I have deployed the new ArgusModeler file and will check the logs in a few hours. I do see other flow records for this same connection before the problem record occurs. The example I attached earlier is from a connection that was open for several days, idle most of the time, but occasionally moving large chunks of data.
>
> The ARGUS_MAR_STATUS_INTERVAL is not set (using default) and the ARGUS_FLOW_STATUS_INTERVAL is set to 60 seconds.
>
> Fragments??? The sensor is collecting data outside our firewall, so I’m guessing Argus seeing a lot of interesting things! In addition Argus is collecting some internal to internal data, which is where the earlier example data was collected. I would estimate that Argus is getting 8 Gbps peak.
>
> Sounds like you may be on to something with the no packets to report. That would fit with the example being a long running connection with lots of idle time.
>
> Thanks,
> Joel
>
>
>
>> On Mar 18, 2019, at 9:58 AM, Carter Bullard <carter at qosient.com <mailto:carter at qosient.com>> wrote:
>>
>> Hey Joel,
>> Just a few questions about your bug. If you have a flow that is wrong, are there other flow records from the same flow (same 5-tuple) prior to this record ??
>>
>> What is your ARGUS_MAR_STATUS_INTERVAL set to ??? ( your flow record Is 31.85 secs long … is that reasonable ??)
>>
>> Do you have any Fragments flying around in your network ???
>>
>> Is this argus heavily loaded or lightly loaded ??
>>
>> There is one condition that can generate this problem, where we have a flow record that argus wants to export, but there aren’t any packets to report. This shouldn’t happen, but you never know ….
>>
>> Can you try this modified ArgusModeler.c, which will not generate the METRICS DSR if there are no packets to report. Should generate records with zero’s, without the carryover bytes from the previous record. I would think that this is better, but not finished ...
>>
>> Carter
>>
>>
>> <ArgusModeler.c.new>
>>
>>> On Mar 6, 2019, at 1:50 PM, Carter Bullard <carter at qosient.com <mailto:carter at qosient.com>> wrote:
>>>
>>> Sorry for the delayed response …. Yes I can confirm that the dsr is well formed but the qualifier is not set. You said that this record came from Argus …. Argus generated the file ??? If so, I’ll know where to look …
>>>
>>> Carter
>>>
>>>
>>
>>>> On Feb 27, 2019, at 3:56 PM, Reed, Joel <reedjw at ornl.gov <mailto:reedjw at ornl.gov>> wrote:
>>>>
>>>> Hey Carter,
>>>>
>>>> I have attached a binary argus file with the problem record. It was captured directly from Argus.
>>>>
>>>> When the record has been put in a file by itself, the bytes and packets are 0 like below. When the record is in the file with another record before it, the client prints the bytes and packet values from the previous record.
>>>>
>>>> $ ra -r prob_rec.argus
>>>> StartTime Flgs Proto SrcAddr Sport SrcPkts SrcBytes Dir DstAddr Dport DstPkts DstBytes State
>>>> 2019-02-27T10:33:36.493615 EST * * tcp 160.91.94.169.2049 0 0 ? 160.91.86.74.1023 0 0 CON
>>>>
>>>>
>>>> All is well here. Hope all is well with you!
>>>>
>>>> Thanks,
>>>> Joel
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> On Feb 27, 2019, at 2:14 PM, carter at qosient.com <mailto:carter at qosient.com> wrote:
>>>>>
>>>>> Hey Joel,
>>>>> Can you send a binary argus file that has one of the records in it ??? Is this record coming directly from argus or is it coming from a client … radium.1 ???
>>>>>
>>>>> There is a lot of data compression going on with the Meter DSR, and the client library will attempt to pack the metrics into as little space as possible, such that, if all the values are less than 256, then all the metrics are reported as a char array. I suspect that the Meter DSR has been compressed, but the final type description is getting dropped.
>>>>>
>>>>> What do the argus-clients do with this record … do they print out anything at all or do they jump past this DSR ???
>>>>> Hope all is going well down in Tennesseeeeeeeeeee land !!!!
>>>>>
>>>>> Carter
>>>>>
>>>>>
>>>>>> On Feb 27, 2019, at 2:01 PM, Reed, Joel via Argus-info <argus-info at lists.andrew.cmu.edu <mailto:argus-info at lists.andrew.cmu.edu>> wrote:
>>>>>>
>>>>>> >Description:
>>>>>> The Argus server occasionally generates a flow record with a meter DSR that is not properly parsed by the ra client. This causes the packet, byte, and appbytes counts to be incorrect, usually containing at least some count values from the previous flow record. Below I have included a partial dump of the of the problematic flow record. The meter DSR subtype 0x04 (includes app bytes) has a qualifier of 0x00. The meter DSR parser (common/argus_client.c:~2223) does not have a case to process a meter DSR subtype 0x04, qualifier of 0x00.
>>>>>>
>>>>>> Partial hex dump of the flow record:
>>>>>>
>>>>>> 01: 13 20 00 39 -- Type 0x10 (FAR), version 3, 0x20 continuation, length 0x39
>>>>>> 02: 01 03 00 03 -- Transport DSR
>>>>>> 03: 00 00 00 00 |
>>>>>> 04: e1 12 61 ff |
>>>>>> 05: 02 01 01 05 -- Flow DSR
>>>>>> 06: a0 5b 56 4a |
>>>>>> 07: a0 5b 5e a9 |
>>>>>> 08: 06 00 03 ff |
>>>>>> 09: 08 01 00 00 |
>>>>>> 0a: 03 02 18 05 -- Time DSR
>>>>>> 0b: 5c 76 ad d0 |
>>>>>> 0c: 00 07 88 2f |
>>>>>> 0d: 5c 76 ad f0 |
>>>>>> 0e: 00 05 41 67 |
>>>>>> 0f: 10 04 00 05 -- Meter DSR, subtype 0x04, qualifier 0x00, length 0x05
>>>>>> 10: 30 00 00 01 |
>>>>>> 11: 40 00 01 02 |
>>>>>> 12: 01 f4 00 00 |
>>>>>> 13: 48 00 01 02 |
>>>>>> 14: 30 05 00 1e -- Network DSR
>>>>>> 15: ...
>>>>>>
>>>>>> >How-To-Repeat:
>>>>>> Unknown. We see approximately one of these per hour.
>>>>>>
>>>>>> >Fix:
>>>>>> Unknown.
>>>>>>
>>>>>> >Originator: Joel Reed <reedjw at ornl.gov <mailto:reedjw at ornl.gov>>
>>>>>> >Organization:
>>>>>> ORNL
>>>>>> >ARGUS support: none
>>>>>> >Release: argus-3.0
>>>>>> >Product: argus
>>>>>> >Synopsis: Argus server occasionally generates an invalid meter DSR in a flow record
>>>>>> >Class: sw-bug
>>>>>> >Severity: non-critical
>>>>>> >Priority: low
>>>>>>
>>>>>> >Environment:
>>>>>>
>>>>>> ARGUS: Argus Version 3.0.8.2
>>>>>> RA: Ra Version 3.0.8.2
>>>>>>
>>>>>
>>>>
>>>> <prob_rec.argus>
>>>
>>> _______________________________________________
>>> argus mailing list
>>> argus at qosient.com <mailto:argus at qosient.com>
>>> https://pairlist1.pair.net/mailman/listinfo/argus <https://pairlist1.pair.net/mailman/listinfo/argus>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20190318/9b032a6d/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4065 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20190318/9b032a6d/attachment-0001.bin>
More information about the argus
mailing list