Ra - filter syntax error

[elof2 at sentor.se]

Hi Carter!

Making it a configurable timeout is up to you. It could be hard-coded for 
all I care.

The important bit is to have an error message that states clearly what is 
wrong, and not fool you into thinking something is suddenly wrong with the 
filter syntax.

/Elof



On Tue, 2 Jun 2015, elof2 at sentor.se wrote:

>
> Hi Carter!
>
> Sure, a configurable timeout sounds like the most professional solution.
> A ra.conf option is enough in my opinion. No need to also add a new 
> commandline option to 'ra'.
>
> Perhaps the errormessage should also hint about this ra.conf option.
>
> /Elof
>
>
> On Mon, 1 Jun 2015, Carter Bullard wrote:
>
>> Hey /Elof,
>> There are 2 parts to this ... Length of compiler timeout and error message 
>> format.
>> Do we want a configurable timer ???
>> Carter
>> 
>> 
>>> On Jan 26, 2015, at 5:22 AM, elof2 at sentor.se wrote:
>>> 
>>> 
>>> Actually, my error is not a syntax error - I use the same filter in a 
>>> cronjob every 5 minutes. Only sometimes, like once in three weeks, I get 
>>> the error below.
>>> 
>>> So there's nothing wrong with the syntax, but I suspect that the filter 
>>> compilation sometimes takes too long, and that this somehow triggers the 
>>> "filter syntax error" when it should trigger a timeout error instead.
>>> (I haven't had time to recompile and test yet, to verify)
>>> 
>>> 
>>> If you put the errormessage first, no truncation is needed I think.
>>> If syslogd has a maximum messagesize, then *it* will truncate instead but 
>>> we will still see the actual error message.
>>> 
>>> /Elof
>>> 
>>> 
>>>> On Thu, 22 Jan 2015, Carter Bullard wrote:
>>>> 
>>>> Hey /Elof,
>>>> Error messages where the filter comes before the error type,
>>>> are all syntax errors, and are found in the ./common/argus_util.c
>>>> file.
>>>> 
>>>> Try this patch to move the filter to the end of the error messages …
>>>> 
>>>> ==== //depot/argus/clients/common/argus_util.c#390 - 
>>>> /Volumes/Users/carter/argus/clients/common/argus_util.c ====
>>>> 1553c1553
>>>> <          ArgusLog (LOG_ERR, "%s filter syntax error", 
>>>> parser->ArgusRemoteFilter);
>>>> ---
>>>>>         ArgusLog (LOG_ERR, "filter syntax error: %s", 
>>>>> parser->ArgusRemoteFilter);
>>>> 1557c1557
>>>> <          ArgusLog (LOG_ERR, "%s filter syntax error", 
>>>> parser->ArgusLocalFilter);
>>>> ---
>>>>>         ArgusLog (LOG_ERR, "filter syntax error: %s", 
>>>>> parser->ArgusLocalFilter);
>>>> 1561c1561
>>>> <          ArgusLog (LOG_ERR, "%s filter syntax error", 
>>>> parser->ArgusDisplayFilter);
>>>> ---
>>>>>         ArgusLog (LOG_ERR, "filter syntax error: %s", 
>>>>> parser->ArgusDisplayFilter);
>>>> 
>>>> 
>>>> If we truncated the string length, say using %.xxxs where xxx is the 
>>>> number of bytes,
>>>> what do you think would be a good length ???
>>>> 
>>>> Carter
>>>> 
>>>>> On Jan 22, 2015, at 12:14 PM, elof2 at sentor.se wrote:
>>>>> 
>>>>> 
>>>>> Apropos this old thread.
>>>>> 
>>>>> If ra 3.0.8 still take too long to compile the filter, the logged 
>>>>> message should be cropped.
>>>>> Currently, my *long* filter is more than 992 characters, and apparently 
>>>>> that is some limit. Either in ra or in syslog, because the syslog 
>>>>> message
>>>>> looks like this:
>>>>> 
>>>>> "2015-01-22 15:54:33 +01:00 foobar ra[36131]: 1421938473.058216 src net 
>>>>> (10.0.0.0/8 or ... or ... ... ..  . . . . . . . . . and not host 
>>>>> 10.123.123.100 and not host 10.123.123.123 and not host"
>>>>> 
>>>>> Where the message is 992 characters long.
>>>>> 
>>>>> In reality it should continue with more "and not host" and after that 
>>>>> probably give some error-message about the filer compilation taking too 
>>>>> long. However, this information is not shown due to the length of my 
>>>>> *long* filterstring.
>>>>> So, either crop the filterstring, or put the error-message in front of 
>>>>> the filterstring so that people understand why ra is complaining.
>>>>> 
>>>>> /Elof
>>>>> 
>>>>> 
>>>>>> On Mon, 20 May 2013, Carter Bullard wrote:
>>>>>> 
>>>>>> Hey Elof,
>>>>>> Great, and thanks for the info.  While the clients will be blocked 
>>>>>> waiting for the filter to be compiled, because this is structured as a 
>>>>>> deadman timer, if the compiler returns quickly, there won't be any 
>>>>>> delay.
>>>>>> 
>>>>>> For remote accesses, we don't connect until we've compiled the filter. 
>>>>>> The logic is if we can't compile the filter, the remote won't be able 
>>>>>> to either...so a long time wait for the local compiler won't over run 
>>>>>> us with data.
>>>>>> 
>>>>>> I'll make it 1sec in the code base until I hear otherwise.
>>>>>> Carter
>>>>>> 
>>>>>> 
>>>>>>> On May 20, 2013, at 7:41 AM, elof2 at sentor.se wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> Carter and I have discussed a very unusual error message; "filter 
>>>>>>> syntax error", which may show up if the machine is HEAVILY burdened, 
>>>>>>> like swapping a lot and/or receiving tons of interrupts, while feeding 
>>>>>>> a long and complex filter expression to a ra* process.
>>>>>>> 
>>>>>>>> I haven't seen this problem before, but there maybe a simple fix.
>>>>>>>> In order to get around a problem with using flex and bison to compile
>>>>>>>> multiple filters at a time, we fork a process to compile a filter, 
>>>>>>>> and wait
>>>>>>>> for that process to write the filter back to argus through a pipe. 
>>>>>>>> We
>>>>>>>> have specific error messages for the pipe creation, but we use a
>>>>>>>> generic syntax error message if the forked process doesn't
>>>>>>>> return a binary filter back to us in a timeout period, which is now 
>>>>>>>> set
>>>>>>>> at 200 milliseconds.
>>>>>>> 
>>>>>>> I suggest that Carter add a specific error message for this particular 
>>>>>>> scenario, logging the message "filter compilation timeout" instead of 
>>>>>>> the generic "filter syntax error".
>>>>>>> 
>>>>>>>> In the argus-clients file ./common/argus_code.c, in the routine 
>>>>>>>> ArgusFilterCompile(),
>>>>>>>> on line 389, we set the timer value.  Increase the time to, what, a 
>>>>>>>> second ?
>>>>>>>> Here is a patch for the 3.0.6.2 code…..
>>>>>>>> thoth:common carter$ diff argus_code.c.orig argus_code.c
>>>>>>>> 389,390c389,390
>>>>>>>> <       wait.tv_sec  = 0;
>>>>>>>> <       wait.tv_usec = 200000;
>>>>>>>> ---
>>>>>>>>>     wait.tv_sec  = 1;
>>>>>>>>>     wait.tv_usec = 0;
>>>>>>> 
>>>>>>> 
>>>>>>> During the weekend I've had cron start ra with a long and complex 
>>>>>>> filter string every 5 minutes.
>>>>>>> With a timeout of 500ms, I had 9 "filter syntax error" in 34 hours.
>>>>>>> (on an old Intel Xeon 3050 @ 2.13GHz machine from 2008)
>>>>>>> (with 200ms I suspect I would have had approx 300 warnings)
>>>>>>> 
>>>>>>> 
>>>>>>> So to prevent this message to appear "too often" on choked machines, 
>>>>>>> you could increase the wait.tv_usec to 750000  ...or even higher if a 
>>>>>>> higher value doesn't introduce any overall negative impact on ra*.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> I now upped the timeout to 900000 just to see if that is enough to 
>>>>>>> quell even those 9 warnings.
>>>>>>> 
>>>>>>> /Elof
>>>> 
>