[ARGUS] Argus importing zeek data

[Carter Bullard]

Hey Monah,
Its the community_id key … its not in the raconvert.zeek.conf file … raconvert should ignore fields that it doesn’t know, so I’ll fix that ...
Try this configuration file … I added ‘community_id’ to the RACONVERT_FIELD_SPECIFIER variable, and mapped the value to the label …

Carter



> On Mar 30, 2022, at 10:27 AM, Monah Baki <monahbaki at gmail.com> wrote:
> 
> Morning Carter
> 
> We are running security onion with zeek. Here is a sample 
> 
> cat /nsm/zeek/logs/current/conn.log 
> {"ts":1648648794.073199,"uid":"C9v77B3hIQWKghwEc2","id.orig_h":"172.16.100.149","id.orig_p":55909,"id.resp_h":"172.16.86.65","id.resp_p":6027,"proto":"tcp","conn_state":"S0","local_orig":true,"local_resp":tru
> e,"missed_bytes":0,"history":"S","orig_pkts":1,"orig_ip_bytes":52,"resp_pkts":0,"resp_ip_bytes":0,"community_id":"1:xacMILtZUoqN7bOt5I2J4n3s6UU="}
> {"ts":1648648794.108026,"uid":"CVYLk43Gi0NzjmuUj6","id.orig_h":"172.16.245.73","id.orig_p":53536,"id.resp_h":"172.16.86.14","id.resp_p":8530,"proto":"tcp","duration":0.015297889709472657,"orig_bytes":104,"res
> p_bytes":505,"conn_state":"SF","local_orig":true,"local_resp":true,"missed_bytes":0,"history":"ShADfFa","orig_pkts":5,"orig_ip_bytes":316,"resp_pkts":3,"resp_ip_bytes":637,"community_id":"1:waj0R3ypJ68ox0zi78
> BBw07KepY="}
> 
> If I use  /usr/local/bin/raconvert -f /etc/raconvert.zeek.conf -r /nsm/zeek/logs/current/conn.log -w <filename>, the filename does not get created
> 
> Had to run the command without the -w, as you can see below
> 
> [root at sosensor admin]# /usr/local/bin/raconvert -f /etc/raconvert.zeek.conf -r /nsm/zeek/logs/current/conn.log
>          StartTime      Flgs  Proto            SrcAddr  Sport   Dir            DstAddr  Dport  TotPkts   TotBytes State
>   06:53:52.4313291
> 00:00:00.13203538*
> 00:00:00.13203538*
> 00:00:00.13203538*
> 00:00:00.13203538*
> 00:00:00.13203538*
> 00:00:00.13203538*
> 00:00:00.13203538*
> 
> Am I missing anything?
> 
> 
> Monah
> 
> 
> On Sun, Mar 27, 2022 at 11:52 AM Carter Bullard <carter at qosient.com <mailto:carter at qosient.com>> wrote:
> Gentle persons,
> I’ve pushed new code to the GitHub openargus clients repo, https://github.com/openargus/clients <https://github.com/openargus/clients> , and bumped the version to 3.0.8.4.
> This specific push adds foreign flow data imports using raconvert.1, and I’ve added a raconvert.zeek.conf that enables raconvert.1 to read zeek conn logs.
> 
> To convert a zeek conn log to argus binary:
>    % raconvert -f raconvert.zeek.conf -r zeek.conn.log -w argus.file
> 
> The provided raconvert.zeek.conf conversion map, maps all of the zeek fields that I had available.  The conversion map is pretty chunky, as you need to specify the allowed key, value pairs, specify types, and identify where in an argus record the data will go …  If there isn’t a native argus attribute for the zeek field, say the “uid”, raconvert.1 can map the key, value pair it to the argus label, which you can filter, search, etc ….
> 
> Converting zeek json data to argus binary reduces the size of the files by 2.5:1 or about 1/3rd.  And gzip’d argus binaries are smaller than gzip’d json zeek.conn.logs, by around 1.3:1 … not much but it is not worse :O)
> 
> Converting the zeek logs to argus enables processing zeek data with argus clients programs like racluster.1, say if you want to generate baselines, or to generate different views.  The clients enable you to add country codes, and ASNs for addresses simply, and I like using rafilteraddr.1 with address lists from 3rd party intelligence like Firehol to check to see if the zeek data has any reputation hits ... and you can of course view the data with ratop.1 …
> 
> Please grab this client code and test it out, if interested …
> 
> I would love any opinions about the new GitHub software approach.  If you have any suggestions, please email the list or to me ...
> Hope all is most excellent,
> 
> Carter
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20220330/de4ad85a/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: raconvert.zeek.conf
Type: application/octet-stream
Size: 7307 bytes
Desc: not available
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20220330/de4ad85a/attachment-0001.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20220330/de4ad85a/attachment-0003.htm>