[ARGUS] Argus importing zeek data

Wed Mar 30 17:32:56 EDT 2022

Hey Monah,
I made some more changes and its working for me with your data ...  please refetch and pull ... and all should be good ... if not ...  try a '-X' to clear out any conf that maybe in the way ...

if no joy, run with '-D 5' and send the output !!!

Carter

> On Mar 30, 2022, at 5:22 PM, Monah Baki <monahbaki at gmail.com> wrote:
> 
> 
> Hi Carter,
> 
> Downloaded the new clients-master, and still having the same issue. 
> 
> No matter what RACONVERT_TIME_FORMAT I use, the output looks like:
> 00:00:00.14370365*
> 
> Monah
> 
>> On Wed, Mar 30, 2022 at 12:10 PM Carter Bullard <carter at qosient.com> wrote:
>> Hey Monah,
>> I fixed the issue so that your data and the default conf file will work.
>> I noticed that your timestamps were not parsed correctly, as you had not set the correct time parsing method in the conf file.
>> I fixed that as well.
>> 
>> Fetch the new master to get the config and the new code …
>> 
>> Carter
>> 
>>> On Mar 30, 2022, at 10:42 AM, Carter Bullard <carter at qosient.com> wrote:
>>> 
>>> Hey Monah,
>>> Its the community_id key … its not in the raconvert.zeek.conf file … raconvert should ignore fields that it doesn’t know, so I’ll fix that ...
>>> Try this configuration file … I added ‘community_id’ to the RACONVERT_FIELD_SPECIFIER variable, and mapped the value to the label …
>>> 
>>> Carter
>>> 
>>> <raconvert.zeek.conf>
>>> 
>>> 
>>>> On Mar 30, 2022, at 10:27 AM, Monah Baki <monahbaki at gmail.com> wrote:
>>>> 
>>>> Morning Carter
>>>> 
>>>> We are running security onion with zeek. Here is a sample 
>>>> 
>>>> cat /nsm/zeek/logs/current/conn.log 
>>>> {"ts":1648648794.073199,"uid":"C9v77B3hIQWKghwEc2","id.orig_h":"172.16.100.149","id.orig_p":55909,"id.resp_h":"172.16.86.65","id.resp_p":6027,"proto":"tcp","conn_state":"S0","local_orig":true,"local_resp":tru
>>>> e,"missed_bytes":0,"history":"S","orig_pkts":1,"orig_ip_bytes":52,"resp_pkts":0,"resp_ip_bytes":0,"community_id":"1:xacMILtZUoqN7bOt5I2J4n3s6UU="}
>>>> {"ts":1648648794.108026,"uid":"CVYLk43Gi0NzjmuUj6","id.orig_h":"172.16.245.73","id.orig_p":53536,"id.resp_h":"172.16.86.14","id.resp_p":8530,"proto":"tcp","duration":0.015297889709472657,"orig_bytes":104,"res
>>>> p_bytes":505,"conn_state":"SF","local_orig":true,"local_resp":true,"missed_bytes":0,"history":"ShADfFa","orig_pkts":5,"orig_ip_bytes":316,"resp_pkts":3,"resp_ip_bytes":637,"community_id":"1:waj0R3ypJ68ox0zi78
>>>> BBw07KepY="}
>>>> 
>>>> If I use  /usr/local/bin/raconvert -f /etc/raconvert.zeek.conf -r /nsm/zeek/logs/current/conn.log -w <filename>, the filename does not get created
>>>> 
>>>> Had to run the command without the -w, as you can see below
>>>> 
>>>> [root at sosensor admin]# /usr/local/bin/raconvert -f /etc/raconvert.zeek.conf -r /nsm/zeek/logs/current/conn.log
>>>>          StartTime      Flgs  Proto            SrcAddr  Sport   Dir            DstAddr  Dport  TotPkts   TotBytes State
>>>>   06:53:52.4313291
>>>> 00:00:00.13203538*
>>>> 00:00:00.13203538*
>>>> 00:00:00.13203538*
>>>> 00:00:00.13203538*
>>>> 00:00:00.13203538*
>>>> 00:00:00.13203538*
>>>> 00:00:00.13203538*
>>>> 
>>>> Am I missing anything?
>>>> 
>>>> 
>>>> Monah
>>>> 
>>>> 
>>>>> On Sun, Mar 27, 2022 at 11:52 AM Carter Bullard <carter at qosient.com> wrote:
>>>>> Gentle persons,
>>>>> I’ve pushed new code to the GitHub openargus clients repo, https://github.com/openargus/clients , and bumped the version to 3.0.8.4.
>>>>> This specific push adds foreign flow data imports using raconvert.1, and I’ve added a raconvert.zeek.conf that enables raconvert.1 to read zeek conn logs.
>>>>> 
>>>>> To convert a zeek conn log to argus binary:
>>>>>    % raconvert -f raconvert.zeek.conf -r zeek.conn.log -w argus.file
>>>>> 
>>>>> The provided raconvert.zeek.conf conversion map, maps all of the zeek fields that I had available.  The conversion map is pretty chunky, as you need to specify the allowed key, value pairs, specify types, and identify where in an argus record the data will go …  If there isn’t a native argus attribute for the zeek field, say the “uid”, raconvert.1 can map the key, value pair it to the argus label, which you can filter, search, etc ….
>>>>> 
>>>>> Converting zeek json data to argus binary reduces the size of the files by 2.5:1 or about 1/3rd.  And gzip’d argus binaries are smaller than gzip’d json zeek.conn.logs, by around 1.3:1 … not much but it is not worse :O)
>>>>> 
>>>>> Converting the zeek logs to argus enables processing zeek data with argus clients programs like racluster.1, say if you want to generate baselines, or to generate different views.  The clients enable you to add country codes, and ASNs for addresses simply, and I like using rafilteraddr.1 with address lists from 3rd party intelligence like Firehol to check to see if the zeek data has any reputation hits ... and you can of course view the data with ratop.1 …
>>>>> 
>>>>> Please grab this client code and test it out, if interested …
>>>>> 
>>>>> I would love any opinions about the new GitHub software approach.  If you have any suggestions, please email the list or to me ...
>>>>> Hope all is most excellent,
>>>>> 
>>>>> Carter
>>>>> 
>>> 
>>> _______________________________________________
>>> argus mailing list
>>> argus at qosient.com
>>> https://pairlist1.pair.net/mailman/listinfo/argus
>> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20220330/c83c0275/attachment.htm>