[ARGUS] Argus importing zeek data

[Monah Baki]

Hi Carter,

Downloaded the new clients-master, and still having the same issue.

No matter what RACONVERT_TIME_FORMAT I use, the output looks like:
00:00:00.14370365*

Monah

On Wed, Mar 30, 2022 at 12:10 PM Carter Bullard <carter at qosient.com> wrote:

> Hey Monah,
> I fixed the issue so that your data and the default conf file will work.
> I noticed that your timestamps were not parsed correctly, as you had not
> set the correct time parsing method in the conf file.
> I fixed that as well.
>
> Fetch the new master to get the config and the new code …
>
> Carter
>
> On Mar 30, 2022, at 10:42 AM, Carter Bullard <carter at qosient.com> wrote:
>
> Hey Monah,
> Its the community_id key … its not in the raconvert.zeek.conf file …
> raconvert should ignore fields that it doesn’t know, so I’ll fix that ...
> Try this configuration file … I added ‘community_id’ to
> the RACONVERT_FIELD_SPECIFIER variable, and mapped the value to the
> label …
>
> Carter
>
> <raconvert.zeek.conf>
>
>
> On Mar 30, 2022, at 10:27 AM, Monah Baki <monahbaki at gmail.com> wrote:
>
> Morning Carter
>
> We are running security onion with zeek. Here is a sample
>
> cat /nsm/zeek/logs/current/conn.log
>
> {"ts":1648648794.073199,"uid":"C9v77B3hIQWKghwEc2","id.orig_h":"172.16.100.149","id.orig_p":55909,"id.resp_h":"172.16.86.65","id.resp_p":6027,"proto":"tcp","conn_state":"S0","local_orig":true,"local_resp":tru
>
> e,"missed_bytes":0,"history":"S","orig_pkts":1,"orig_ip_bytes":52,"resp_pkts":0,"resp_ip_bytes":0,"community_id":"1:xacMILtZUoqN7bOt5I2J4n3s6UU="}
>
> {"ts":1648648794.108026,"uid":"CVYLk43Gi0NzjmuUj6","id.orig_h":"172.16.245.73","id.orig_p":53536,"id.resp_h":"172.16.86.14","id.resp_p":8530,"proto":"tcp","duration":0.015297889709472657,"orig_bytes":104,"res
>
> p_bytes":505,"conn_state":"SF","local_orig":true,"local_resp":true,"missed_bytes":0,"history":"ShADfFa","orig_pkts":5,"orig_ip_bytes":316,"resp_pkts":3,"resp_ip_bytes":637,"community_id":"1:waj0R3ypJ68ox0zi78
> BBw07KepY="}
>
> If I use  /usr/local/bin/raconvert -f /etc/raconvert.zeek.conf -r
> /nsm/zeek/logs/current/conn.log -w <filename>, the filename does not get
> created
>
> Had to run the command without the -w, as you can see below
>
> [root at sosensor admin]# /usr/local/bin/raconvert -f
> /etc/raconvert.zeek.conf -r /nsm/zeek/logs/current/conn.log
>          StartTime      Flgs  Proto            SrcAddr  Sport   Dir
>      DstAddr  Dport  TotPkts   TotBytes State
>   06:53:52.4313291
> 00:00:00.13203538*
> 00:00:00.13203538*
> 00:00:00.13203538*
> 00:00:00.13203538*
> 00:00:00.13203538*
> 00:00:00.13203538*
> 00:00:00.13203538*
>
> Am I missing anything?
>
>
> Monah
>
>
> On Sun, Mar 27, 2022 at 11:52 AM Carter Bullard <carter at qosient.com>
> wrote:
>
>> Gentle persons,
>> I’ve pushed new code to the GitHub openargus clients repo,
>> https://github.com/openargus/clients , and bumped the version to 3.0.8.4.
>> This specific push adds foreign flow data imports using raconvert.1, and
>> I’ve added a raconvert.zeek.conf that enables raconvert.1 to read zeek conn
>> logs.
>>
>> To convert a zeek conn log to argus binary:
>>    % raconvert -f raconvert.zeek.conf -r zeek.conn.log -w argus.file
>>
>> The provided raconvert.zeek.conf conversion map, maps all of the zeek
>> fields that I had available.  The conversion map is pretty chunky, as you
>> need to specify the allowed key, value pairs, specify types, and identify
>> where in an argus record the data will go …  If there isn’t a native argus
>> attribute for the zeek field, say the “uid”, raconvert.1 can map the key,
>> value pair it to the argus label, which you can filter, search, etc ….
>>
>> Converting zeek json data to argus binary reduces the size of the files
>> by 2.5:1 or about 1/3rd.  And gzip’d argus binaries are smaller than gzip’d
>> json zeek.conn.logs, by around 1.3:1 … not much but it is not worse :O)
>>
>> Converting the zeek logs to argus enables processing zeek data with argus
>> clients programs like racluster.1, say if you want to generate baselines,
>> or to generate different views.  The clients enable you to add country
>> codes, and ASNs for addresses simply, and I like using rafilteraddr.1 with
>> address lists from 3rd party intelligence like Firehol to check to see if
>> the zeek data has any reputation hits ... and you can of course view the
>> data with ratop.1 …
>>
>> Please grab this client code and test it out, if interested …
>>
>> I would love any opinions about the new GitHub software approach.  If you
>> have any suggestions, please email the list or to me ...
>> Hope all is most excellent,
>>
>> Carter
>>
>>
> _______________________________________________
> argus mailing list
> argus at qosient.com
> https://pairlist1.pair.net/mailman/listinfo/argus
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist1.pair.net/pipermail/argus/attachments/20220330/ba735435/attachment-0001.htm>