ARGUS Binary Size
Carter Bullard
carter at qosient.com
Wed Sep 24 13:16:33 EDT 2014
Hey James,
That’s great !!
Sometimes those broadcasts provide some really good information,
as broadcast can be used in attacks, or can reveal a bad acting
L2 device that is confused, etc…
When you get a lot of junky traffic like that, you use racluster.1
with a pretty interesting aggregation rule set, to squelch the
broadcasts down, something like
filter="ip multicast" model=‘saddr daddr’ status=60
filter=“arp” status=60
status=1
So this will aggregate multicast, and arp, and report on
those flows every 60s, but it will not aggregate any other
flow records. The output will not be stime ordered, but
that is pretty easy to deal with.
Carter
On Sep 24, 2014, at 12:21 PM, James Grace <jgrac002 at fiu.edu> wrote:
> Carter,
>
> I wasn't seeing any errors on the argus side of the equation, but your suggestion has worked.
>
> In regards to the origin of this thread, it turns out one of the VLANs that runs through our exchange point contains a rather large flat network -- somewhere in the area of /16 I believe, and it is rather busy. This explains the myriad of single-packet flow records -- they were all broadcasts! I removed this vlan using an ra* filter and now everything is running smoothly.
>
> Binary sizes have tumbled down to 10MBytes per 5-minute record. A vast improvement!
>
> Thanks to everyone that has contributed!
> -james
>
>
> On Wed, Sep 24, 2014 at 10:46 AM, Carter Bullard <carter at qosient.com> wrote:
> Hey James,
> When you send bug/problem reports, you should send some
> output of the command run, so we all can see what’s going
> on.
>
> The manpage is a good place to start, and you can use an argus.conf file.
>
> -i dup:dag0,dag1/2.3.4.5
>
> If its not working, then send some email, and we can try
> to figure it out.
>
> You should see a printed statement as to whether the interfaces
> are up, etc… And you can always use a -D2 or -D3 option to
> see what it thinks is going on with the packet readers.
>
> Carter
>
>
> On Sep 24, 2014, at 10:37 AM, James Grace <jgrac002 at fiu.edu> wrote:
>
>> Carter,
>> How does one go about attach to two cards using one argus? I've tried:
>>
>> argus -d —i dag0 dag1 -P 561
>>
>>
>>
>> But this doesn't seem to attach to dag0 as expected -- the drops on the Dag0 are increasing as if no process was attached to it.
>>
>>
>>
>> Thanks,
>>
>> James
>>
>>
>>
>>
>> On Tue, Sep 16, 2014 at 2:38 PM, Carter Bullard <carter at qosient.com> wrote:
>> Definitely should be running 1 argus, opening both interfaces.
>> You should set a little bit more in your /etc/argus.conf file,
>> like the ARGUS_MONITOR_ID.
>>
>> Many tests for whether your getting the data you want is to
>> inspect the actual ‘primitive’ data from argus. Do any of the
>> flows in your collection files look like they have bi-directional
>> counts, more than 1 packet in each one, that type of stuff.
>>
>> racluster() will merge records together, which will correct for
>> any problems, such as the status interval setting, which is the
>> value that gets you the most data reduction.
>>
>> Carter
>>
>> On Sep 16, 2014, at 11:19 AM, James Grace <jgrac002 at fiu.edu> wrote:
>>
>>> Hi Carter,
>>>
>>> In regards to our flow records, when I run
>>> racluster -m matrix -r argus.12.35 -w - | rasort -m pkts
>>>
>>> During this 5 minutes period, there were ~500 flows with large packet counts, but then these packets/per flow dropped off severely — most being around 7 or 8pkts/flow.
>>>
>>> These 2 DAG cards are measuring the egress and ingress on a single circuit, and if I run the DAG tools (dagsnap, etc) I am seeing unique traffic over the wire on each card.
>>>
>>> This may just be chalked up to the way traffic if flowing over this circuit rather than anything to do with Argus.
>>>
>>> Thoughts?
>>>
>>> -james
>>>
>>> On Sep 15, 2014, at 4:01 PM, Carter Bullard <carter at qosient.com> wrote:
>>>
>>>> Hey James,
>>>> The src and dst labels for traffic in argus records does not
>>>> equate to the number of packets we receive on the 2 DAG interfaces.
>>>> These src and dst are flow src and dst, and refer to the flow
>>>> source (the initiator of the flow transaction) and flow destination
>>>> (the target of the flow transaction).
>>>>
>>>> I suspect that if there is a problem, its that argus is getting
>>>> packets from only one interface.
>>>>
>>>> Carter
>>>>
>>>>
>>>> On Sep 15, 2014, at 1:42 PM, James Grace <jgrac002 at fiu.edu> wrote:
>>>>
>>>>> David, Carter
>>>>>
>>>>> Thanks for the help so far. I’ve changed some things around and the new scheme is:
>>>>>
>>>>> Two argus instances listening to a DAG card:
>>>>>
>>>>> argus -d -i dag0 -P 561
>>>>> argus -d -i dag1 -P 562
>>>>>
>>>>> I have only one option in my /etc/argus.conf:
>>>>>
>>>>> # cat /etc/argus.confg
>>>>> ARGUS_CAPTURE_DATA_LEN=88
>>>>>
>>>>> I have also configured the DAG cards to slen=88.
>>>>>
>>>>> I have radium connecting to the argi thusly:
>>>>> radium -S localhost:561 -S localhost:562 -d -M dsrs=-net -encaps -P 563
>>>>>
>>>>> And I have rastream connecting to radium to store and sort the argus binaries.
>>>>> rastream -s 48 -S localhost:563 -M dsrs=-net -M time 5m -w /flows/argus/%Y/%m/%d/argus.%H.%M -D3
>>>>>
>>>>> I understand that each of these binaries now contain two streams of traffic. Is that correct? Even with all these flags and slen parameters, I am seeing massive binary sizes:
>>>>>
>>>>> ls -l
>>>>> total 57231320
>>>>> -rw-r--r--. 1 root root 5790698448 Sep 15 13:13 argus.12.45
>>>>> -rw-r--r--. 1 root root 8052120672 Sep 15 13:13 argus.12.50
>>>>> -rw-r--r--. 1 root root 8416875168 Sep 15 13:13 argus.12.55
>>>>> -rw-r--r--. 1 root root 7613404132 Sep 15 13:13 argus.13.00
>>>>> -rw-r--r--. 1 root root 6660266172 Sep 15 13:13 argus.13.05
>>>>> -rw-r--r--. 1 root root 6112649344 Sep 15 13:19 argus.13.15
>>>>> -rw-r--r--. 1 root root 7716601984 Sep 15 13:24 argus.13.20
>>>>> -rw-r--r--. 1 root root 8098926720 Sep 15 13:29 argus.13.25
>>>>> -rw-r--r--. 1 root root 143270016 Sep 15 13:30 argus.13.30
>>>>>
>>>>>
>>>>> If I run recount I’m seeing 5million records per 5 minute binary:
>>>>>
>>>>> racount -r argus.13.25
>>>>> racount records total_pkts src_pkts dst_pkts total_bytes src_bytes dst_bytes
>>>>> sum 53181673 53191299 37622951 15568348 49381504197 28748639718 20632864479
>>>>>
>>>>> I’m thinking there’s got to be some miscounts. The link I’m measuring, both ingress and egress traffic sum up to about 2Gb/s with a peak of 4Gb/s (attached is a graph).
>>>>>
>>>>> I would expect the binary size to be roughly equivalent to an SFlow record (~3MB)
>>>>> <chart2.png>
>>>>> Any suggestions?
>>>>>
>>>>> Thanks
>>>>> -james
>>>>>
>>>>> On Aug 7, 2014, at 11:50 PM, David Edelman <dedelman at iname.com> wrote:
>>>>>
>>>>>> There are strange things done in the midnight sun by config files that hang around waiting for a trusting soul.
>>>>>>
>>>>>> I suggest that you try adding –X as the first item in your command line just to be sure that someone, sometime didn’t configure /etc/argus to capture 2048 bytes of user data or something else that takes up space.
>>>>>>
>>>>>> —Dave
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> From: James Grace <jgrac002 at fiu.edu>
>>>>>> Date: Monday, August 4, 2014 at 8:02 PM
>>>>>> To: Carter Bullard <carter at qosient.com>
>>>>>> Cc: Argus <argus-info at lists.andrew.cmu.edu>
>>>>>> Subject: Re: [ARGUS] ARGUS Binary Size
>>>>>>
>>>>>> Carter,
>>>>>>
>>>>>> Here is the flow records for a 5 minute trace:
>>>>>>
>>>>>> /flows/South/2014/07/31$ racount -r argus.13.50
>>>>>> racount records total_pkts src_pkts dst_pkts total_bytes src_bytes dst_bytes
>>>>>>
>>>>>> sum 1764263 35097518 24751551 10345967 43920187649 30233499247 13686688402
>>>>>> I'm currently not running argus off of an argus.conf -- I imagine it's using default values.
>>>>>>
>>>>>> Thanks,
>>>>>> -james
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Aug 4, 2014 at 3:58 PM, Carter Bullard <carter at qosient.com> wrote:
>>>>>>> Hey James,
>>>>>>> Could be you’re seeing a lot of flows. How many flow records are being stored?
>>>>>>>
>>>>>>> % racount -r argus.data.file.5m
>>>>>>>
>>>>>>> Depending on the configuration, a flow record can be anywhere from 100-1K bytes
>>>>>>> per record, on the average. Argus output size is not sensitive to packet size,
>>>>>>> except when its capturing user data. This is controlled by the
>>>>>>> ARGUS_CAPTURE_DATA_LEN variable in the /etc/argus.conf file, if you have one...
>>>>>>>
>>>>>>> What is that set to ????
>>>>>>>
>>>>>>> Carter
>>>>>>>
>>>>>>> On Aug 4, 2014, at 3:50 PM, James Grace <jgrac002 at fiu.edu> wrote:
>>>>>>>
>>>>>>> > Good afternoon List,
>>>>>>> >
>>>>>>> > I've been collecting traces using argus and rastream off of a DAG8.1SX. The link is running right around 2.3Gb/s. I've been looking into the argus.conf manpage to see if there is a way to limit the packet length stored by rastream or argus.
>>>>>>> >
>>>>>>> > Right now, if I run argus and rastream with these flags"
>>>>>>> >
>>>>>>> > #argus -d -i dag0 -P 561
>>>>>>> >
>>>>>>> > #rastream -S localhost -M time 5m -w /flows/South/%Y/%m/%d/argus.%H.%M -D 3&
>>>>>>> >
>>>>>>> > I'm getting rather large binaries for 5 minutes -- right around 1GB. I have a feeling its grabbing all 9000bytes of the jumbo frame. Is there a workaround for this?
>>>>>>> >
>>>>>>> > Thanks a bunch!
>>>>>>> >
>>>>>>> > James
>>>>>>> >
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20140924/0d3a6089/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 455 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20140924/0d3a6089/attachment.sig>
More information about the argus
mailing list