[ARGUS] Argus importing zeek data

Thu Mar 31 08:49:00 EDT 2022

Morning Carter,

Apparently security onion does not work. I was able to install and run
successfully on a plain Centos 7 running zeek 4.0.5.

Thanks
Monah

On Wed, Mar 30, 2022 at 5:32 PM Carter Bullard <carter at qosient.com> wrote:

> Hey Monah,
> I made some more changes and its working for me with your data ...  please
> refetch and pull ... and all should be good ... if not ...  try a '-X' to
> clear out any conf that maybe in the way ...
>
> if no joy, run with '-D 5' and send the output !!!
>
> Carter
>
> On Mar 30, 2022, at 5:22 PM, Monah Baki <monahbaki at gmail.com> wrote:
>
> 
> Hi Carter,
>
> Downloaded the new clients-master, and still having the same issue.
>
> No matter what RACONVERT_TIME_FORMAT I use, the output looks like:
> 00:00:00.14370365*
>
> Monah
>
> On Wed, Mar 30, 2022 at 12:10 PM Carter Bullard <carter at qosient.com>
> wrote:
>
>> Hey Monah,
>> I fixed the issue so that your data and the default conf file will work.
>> I noticed that your timestamps were not parsed correctly, as you had not
>> set the correct time parsing method in the conf file.
>> I fixed that as well.
>>
>> Fetch the new master to get the config and the new code …
>>
>> Carter
>>
>> On Mar 30, 2022, at 10:42 AM, Carter Bullard <carter at qosient.com> wrote:
>>
>> Hey Monah,
>> Its the community_id key … its not in the raconvert.zeek.conf file …
>> raconvert should ignore fields that it doesn’t know, so I’ll fix that ...
>> Try this configuration file … I added ‘community_id’ to
>> the RACONVERT_FIELD_SPECIFIER variable, and mapped the value to the
>> label …
>>
>> Carter
>>
>> <raconvert.zeek.conf>
>>
>>
>> On Mar 30, 2022, at 10:27 AM, Monah Baki <monahbaki at gmail.com> wrote:
>>
>> Morning Carter
>>
>> We are running security onion with zeek. Here is a sample
>>
>> cat /nsm/zeek/logs/current/conn.log
>>
>> {"ts":1648648794.073199,"uid":"C9v77B3hIQWKghwEc2","id.orig_h":"172.16.100.149","id.orig_p":55909,"id.resp_h":"172.16.86.65","id.resp_p":6027,"proto":"tcp","conn_state":"S0","local_orig":true,"local_resp":tru
>>
>> e,"missed_bytes":0,"history":"S","orig_pkts":1,"orig_ip_bytes":52,"resp_pkts":0,"resp_ip_bytes":0,"community_id":"1:xacMILtZUoqN7bOt5I2J4n3s6UU="}
>>
>> {"ts":1648648794.108026,"uid":"CVYLk43Gi0NzjmuUj6","id.orig_h":"172.16.245.73","id.orig_p":53536,"id.resp_h":"172.16.86.14","id.resp_p":8530,"proto":"tcp","duration":0.015297889709472657,"orig_bytes":104,"res
>>
>> p_bytes":505,"conn_state":"SF","local_orig":true,"local_resp":true,"missed_bytes":0,"history":"ShADfFa","orig_pkts":5,"orig_ip_bytes":316,"resp_pkts":3,"resp_ip_bytes":637,"community_id":"1:waj0R3ypJ68ox0zi78
>> BBw07KepY="}
>>
>> If I use  /usr/local/bin/raconvert -f /etc/raconvert.zeek.conf -r
>> /nsm/zeek/logs/current/conn.log -w <filename>, the filename does not get
>> created
>>
>> Had to run the command without the -w, as you can see below
>>
>> [root at sosensor admin]# /usr/local/bin/raconvert -f
>> /etc/raconvert.zeek.conf -r /nsm/zeek/logs/current/conn.log
>>          StartTime      Flgs  Proto            SrcAddr  Sport   Dir
>>      DstAddr  Dport  TotPkts   TotBytes State
>>   06:53:52.4313291
>> 00:00:00.13203538*
>> 00:00:00.13203538*
>> 00:00:00.13203538*
>> 00:00:00.13203538*
>> 00:00:00.13203538*
>> 00:00:00.13203538*
>> 00:00:00.13203538*
>>
>> Am I missing anything?
>>
>>
>> Monah
>>
>>
>> On Sun, Mar 27, 2022 at 11:52 AM Carter Bullard <carter at qosient.com>
>> wrote:
>>
>>> Gentle persons,
>>> I’ve pushed new code to the GitHub openargus clients repo,
>>> https://github.com/openargus/clients , and bumped the version to
>>> 3.0.8.4.
>>> This specific push adds foreign flow data imports using raconvert.1, and
>>> I’ve added a raconvert.zeek.conf that enables raconvert.1 to read zeek conn
>>> logs.
>>>
>>> To convert a zeek conn log to argus binary:
>>>    % raconvert -f raconvert.zeek.conf -r zeek.conn.log -w argus.file
>>>
>>> The provided raconvert.zeek.conf conversion map, maps all of the zeek
>>> fields that I had available.  The conversion map is pretty chunky, as you
>>> need to specify the allowed key, value pairs, specify types, and identify
>>> where in an argus record the data will go …  If there isn’t a native argus
>>> attribute for the zeek field, say the “uid”, raconvert.1 can map the key,
>>> value pair it to the argus label, which you can filter, search, etc ….
>>>
>>> Converting zeek json data to argus binary reduces the size of the files
>>> by 2.5:1 or about 1/3rd.  And gzip’d argus binaries are smaller than gzip’d
>>> json zeek.conn.logs, by around 1.3:1 … not much but it is not worse :O)
>>>
>>> Converting the zeek logs to argus enables processing zeek data with
>>> argus clients programs like racluster.1, say if you want to generate
>>> baselines, or to generate different views.  The clients enable you to add
>>> country codes, and ASNs for addresses simply, and I like using
>>> rafilteraddr.1 with address lists from 3rd party intelligence like Firehol
>>> to check to see if the zeek data has any reputation hits ... and you can of
>>> course view the data with ratop.1 …
>>>
>>> Please grab this client code and test it out, if interested …
>>>
>>> I would love any opinions about the new GitHub software approach.  If
>>> you have any suggestions, please email the list or to me ...
>>> Hope all is most excellent,
>>>
>>> Carter
>>>
>>>
>> _______________________________________________
>> argus mailing list
>> argus at qosient.com
>> https://pairlist1.pair.net/mailman/listinfo/argus
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20220331/c62777a6/attachment.htm>