From carter at qosient.com Mon Oct 7 11:36:50 2024 From: carter at qosient.com (Carter Bullard) Date: Mon, 7 Oct 2024 11:36:50 -0400 Subject: [ARGUS] Extensions to Argus packet capture support Message-ID: <1BED7ABD-A783-498E-984F-FC9C5B2DF862@qosient.com> Gentle people, I?ve recently been working with a site that is running 100G argus sensors, and we ran into an issue trying to debug some non-conventional tunneling. They were seeing foreign IP addresses in their flow data, which tripped their IDS, once every 4-5 days, and they couldn?t figure out where the data was coming from. Argus was generating flows with unexpected encapsulations and foreign L3 addresses ? Seems natural that people want to question the sensor first, so we needed to validate that the flows were real, the encapsulation parsing was correct, and that the data was captured and reported correctly ? Debugging this type of sensor can be difficult when there are issues, as you typically need to collect packets that can reproduce the condition / problem. Unfortunately, traditional packet capture at 100G is tricky ? In this particular situation, we were getting a packet that was in an unknown tunnel once every 4 or 5 days, which means we needed to figure out how to capture 1 out of 10^12 packets, without killing the sensor or having to buy a bunch of disks. Generally, when we can, we leverage argus itself to do packet capture for debugging. In argus-5.0, there is support for total packet capture (dump packet before processing) and selective packet capture on error (dump packet if there is an error in parsing) which can be configured using the argus.conf file. These two schemes help a lot in developing and debugging argus when there are issues. These schemes proved impractical for this problem, as using full packet capture to get 1 out of 10^12 is not really helpful, and there weren?t any errors when parsing out the packets. As a result, we engineered a ?capture on protocol? feature, which solved the problem in this case. The crazy flow was a GRE tunnel inside a VxLan encapsulation, but with VLANs and unknown potential UDP encapsulations, in a population of VxLAN / GRE / IPnIP , we couldn?t formulate a Wireshark or tcpdump filter that could reliably match the packet condition ? So we put a small bit of logic in Argus?s header parsing routine to check if a protocol mix existed, and enabled any of the argus threads to generate a libpcap dump file for those packets. This worked right out of the gate, and we grabbed 10 packets over a period of a month ? A month of packets at 100G, full line rate, is about 5.18x10^12 packets, so mission accomplished. I believe that this is a great new feature but wanted to poll the group to see if they have a need or see benefit to adding this support to argus-5.0. Hope all is most excellent, Carter -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 1385 bytes Desc: not available URL: From argus-info at lists.andrew.cmu.edu Mon Oct 7 12:06:10 2024 From: argus-info at lists.andrew.cmu.edu (Dave Edelman via Argus-info) Date: Mon, 7 Oct 2024 12:06:10 -0400 Subject: [ARGUS] Extensions to Argus packet capture support In-Reply-To: <1BED7ABD-A783-498E-984F-FC9C5B2DF862@qosient.com> References: <1BED7ABD-A783-498E-984F-FC9C5B2DF862@qosient.com> Message-ID: <03532743-F0C0-4C25-8A87-22026AE18705@iname.com> It sounds like a fantastic addition. Dave Regards, David Edelman who is responsible for the concept of this message. Unfortunately, autocorrect is responsible for the content > On Oct 7, 2024, at 11:44, Carter Bullard wrote: > > ?Gentle people, > I?ve recently been working with a site that is running 100G argus sensors, and we ran into an issue trying to debug some non-conventional tunneling. They were seeing foreign IP addresses in their flow data, which tripped their IDS, once every 4-5 days, and they couldn?t figure out where the data was coming from. > > Argus was generating flows with unexpected encapsulations and foreign L3 addresses ? Seems natural that people want to question the sensor first, so we needed to validate that the flows were real, the encapsulation parsing was correct, and that the data was captured and reported correctly ? Debugging this type of sensor can be difficult when there are issues, as you typically need to collect packets that can reproduce the condition / problem. Unfortunately, traditional packet capture at 100G is tricky ? In this particular situation, we were getting a packet that was in an unknown tunnel once every 4 or 5 days, which means we needed to figure out how to capture 1 out of 10^12 packets, without killing the sensor or having to buy a bunch of disks. > > Generally, when we can, we leverage argus itself to do packet capture for debugging. In argus-5.0, there is support for total packet capture (dump packet before processing) and selective packet capture on error (dump packet if there is an error in parsing) which can be configured using the argus.conf file. These two schemes help a lot in developing and debugging argus when there are issues. These schemes proved impractical for this problem, as using full packet capture to get 1 out of 10^12 is not really helpful, and there weren?t any errors when parsing out the packets. > > As a result, we engineered a ?capture on protocol? feature, which solved the problem in this case. The crazy flow was a GRE tunnel inside a VxLan encapsulation, but with VLANs and unknown potential UDP encapsulations, in a population of VxLAN / GRE / IPnIP , we couldn?t formulate a Wireshark or tcpdump filter that could reliably match the packet condition ? So we put a small bit of logic in Argus?s header parsing routine to check if a protocol mix existed, and enabled any of the argus threads to generate a libpcap dump file for those packets. This worked right out of the gate, and we grabbed 10 packets over a period of a month ? A month of packets at 100G, full line rate, is about 5.18x10^12 packets, so mission accomplished. > > I believe that this is a great new feature but wanted to poll the group to see if they have a need or see benefit to adding this support to argus-5.0. > > Hope all is most excellent, > > Carter > -------------- next part -------------- An HTML attachment was scrubbed... URL: From argus-info at lists.andrew.cmu.edu Thu Oct 17 21:14:05 2024 From: argus-info at lists.andrew.cmu.edu (Ming Fu via Argus-info) Date: Fri, 18 Oct 2024 01:14:05 +0000 Subject: [ARGUS] argus 5 ethernet parse error Message-ID: Hi, I noticed that the argus 5.0.0 can crash with an odd Ethernet head. I tried the argus from the head of repo as of today, the code crash at the same location. (gdb) where #0 0x000055555556baad in ArgusCreateIPv4Flow () #1 0x000055555556c6bd in ArgusCreateFlow () #2 0x000055555556c985 in ArgusProcessPacket () #3 0x0000555555570c64 in ArgusEtherPacket () #4 0x00007ffff7f7cb95 in ?? () from /lib/x86_64-linux-gnu/libpcap.so.0.8 #5 0x00007ffff7f7d004 in ?? () from /lib/x86_64-linux-gnu/libpcap.so.0.8 #6 0x00005555555752cd in ArgusGetPackets () #7 0x00007ffff7f56609 in start_thread (arg=) at pthread_create.c:477 #8 0x00007ffff7d04353 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Looks like the odd ether type caused argus to not fill the IP address when calling into the in ArgusCreateIPv4Flow() I attached a small pcap file, the second packet is the one that can crash the argus. It has an ether type of 0x0056 rather than the usual 0x8000. Regards, Ming -------------- next part -------------- A non-text attachment was scrubbed... Name: twopacket.pcap Type: application/octet-stream Size: 248 bytes Desc: twopacket.pcap URL: From carter at qosient.com Thu Oct 17 21:17:24 2024 From: carter at qosient.com (Carter Bullard) Date: Thu, 17 Oct 2024 21:17:24 -0400 Subject: [ARGUS] argus 5 ethernet parse error In-Reply-To: References: Message-ID: Hey Ming, I'll look into it tomorrow !! Carter > On Oct 17, 2024, at 9:15?PM, Ming Fu via Argus-info wrote: > > ?Hi, > > I noticed that the argus 5.0.0 can crash with an odd Ethernet head. > I tried the argus from the head of repo as of today, the code crash at the same location. > > (gdb) where > #0 0x000055555556baad in ArgusCreateIPv4Flow () > #1 0x000055555556c6bd in ArgusCreateFlow () > #2 0x000055555556c985 in ArgusProcessPacket () > #3 0x0000555555570c64 in ArgusEtherPacket () > #4 0x00007ffff7f7cb95 in ?? () from /lib/x86_64-linux-gnu/libpcap.so.0.8 > #5 0x00007ffff7f7d004 in ?? () from /lib/x86_64-linux-gnu/libpcap.so.0.8 > #6 0x00005555555752cd in ArgusGetPackets () > #7 0x00007ffff7f56609 in start_thread (arg=) at pthread_create.c:477 > #8 0x00007ffff7d04353 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 > > Looks like the odd ether type caused argus to not fill the IP address when calling into the in ArgusCreateIPv4Flow() > > I attached a small pcap file, the second packet is the one that can crash the argus. It has an ether type of 0x0056 rather than the usual 0x8000. > > Regards, > Ming > > > > > > > From argus-info at lists.andrew.cmu.edu Thu Oct 17 21:21:52 2024 From: argus-info at lists.andrew.cmu.edu (Ming Fu via Argus-info) Date: Fri, 18 Oct 2024 01:21:52 +0000 Subject: [ARGUS] argus 5 ethernet parse error In-Reply-To: References: Message-ID: Hi Carter, Thank you -----Original Message----- From: Carter Bullard Sent: Thursday, October 17, 2024 9:17 PM To: Ming Fu Cc: argus-info at lists.andrew.cmu.edu Subject: Re: [ARGUS] argus 5 ethernet parse error Hey Ming, I'll look into it tomorrow !! Carter > On Oct 17, 2024, at 9:15?PM, Ming Fu via Argus-info wrote: > > ?Hi, > > I noticed that the argus 5.0.0 can crash with an odd Ethernet head. > I tried the argus from the head of repo as of today, the code crash at the same location. > > (gdb) where > #0 0x000055555556baad in ArgusCreateIPv4Flow () > #1 0x000055555556c6bd in ArgusCreateFlow () > #2 0x000055555556c985 in ArgusProcessPacket () > #3 0x0000555555570c64 in ArgusEtherPacket () > #4 0x00007ffff7f7cb95 in ?? () from /lib/x86_64-linux-gnu/libpcap.so.0.8 > #5 0x00007ffff7f7d004 in ?? () from /lib/x86_64-linux-gnu/libpcap.so.0.8 > #6 0x00005555555752cd in ArgusGetPackets () > #7 0x00007ffff7f56609 in start_thread (arg=) at pthread_create.c:477 > #8 0x00007ffff7d04353 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 > > Looks like the odd ether type caused argus to not fill the IP address when calling into the in ArgusCreateIPv4Flow() > > I attached a small pcap file, the second packet is the one that can crash the argus. It has an ether type of 0x0056 rather than the usual 0x8000. > > Regards, > Ming > > > > > > > From carter at qosient.com Fri Oct 18 12:05:39 2024 From: carter at qosient.com (Carter Bullard) Date: Fri, 18 Oct 2024 12:05:39 -0400 Subject: [ARGUS] argus 5 ethernet parse error In-Reply-To: References: Message-ID: Hey Ming, Problem found, fixed and committed to main branch of argus on GitHub/openargus/argus The errant packet was an 802.3 LLC SNAP encapsulated packet ? In 802.3 the next protocol field (ether_type) is dependent on the type ? the field can either be the packet length or the next protocol (or a vlan tag) ... We parsed the packet correctly and updated all the needed data structs, but in this particular case, we reported the next protocol using the length ... This length was slightly special and caused us to say that the packet is an ethernet packet, but we setup to process the IP header that was there, so ? boom !!! Should work fine now ? We will report the 2 packets within the same flow, even though one uses Ethernet II and the other is SNAP encapsulated ?. This is correct behavior ... Carter > On Oct 17, 2024, at 9:21?PM, Ming Fu wrote: > > Hi Carter, > > Thank you > > -----Original Message----- > From: Carter Bullard > Sent: Thursday, October 17, 2024 9:17 PM > To: Ming Fu > Cc: argus-info at lists.andrew.cmu.edu > Subject: Re: [ARGUS] argus 5 ethernet parse error > > Hey Ming, > I'll look into it tomorrow !! > Carter > >> On Oct 17, 2024, at 9:15?PM, Ming Fu via Argus-info wrote: >> >> ?Hi, >> >> I noticed that the argus 5.0.0 can crash with an odd Ethernet head. >> I tried the argus from the head of repo as of today, the code crash at the same location. >> >> (gdb) where >> #0 0x000055555556baad in ArgusCreateIPv4Flow () >> #1 0x000055555556c6bd in ArgusCreateFlow () >> #2 0x000055555556c985 in ArgusProcessPacket () >> #3 0x0000555555570c64 in ArgusEtherPacket () >> #4 0x00007ffff7f7cb95 in ?? () from /lib/x86_64-linux-gnu/libpcap.so.0.8 >> #5 0x00007ffff7f7d004 in ?? () from /lib/x86_64-linux-gnu/libpcap.so.0.8 >> #6 0x00005555555752cd in ArgusGetPackets () >> #7 0x00007ffff7f56609 in start_thread (arg=) at pthread_create.c:477 >> #8 0x00007ffff7d04353 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 >> >> Looks like the odd ether type caused argus to not fill the IP address when calling into the in ArgusCreateIPv4Flow() >> >> I attached a small pcap file, the second packet is the one that can crash the argus. It has an ether type of 0x0056 rather than the usual 0x8000. >> >> Regards, >> Ming >> >> >> >> >> >> >> -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 1385 bytes Desc: not available URL: From argus-info at lists.andrew.cmu.edu Wed Oct 23 08:38:33 2024 From: argus-info at lists.andrew.cmu.edu (Ming Fu via Argus-info) Date: Wed, 23 Oct 2024 12:38:33 +0000 Subject: [ARGUS] argus 5 ethernet parse error In-Reply-To: References: Message-ID: Hi Carter, I tried the main branch; the problem is fixed. I applied the diff to the 5.0.0 branch. The problem persists, Is there plan to release an update to 5.0.0 branch? Regards, Ming -----Original Message----- From: Carter Bullard Sent: Friday, October 18, 2024 12:06 PM To: Ming Fu Cc: Argus Subject: Re: [ARGUS] argus 5 ethernet parse error Hey Ming, Problem found, fixed and committed to main branch of argus on GitHub/openargus/argus The errant packet was an 802.3 LLC SNAP encapsulated packet ? In 802.3 the next protocol field (ether_type) is dependent on the type ? the field can either be the packet length or the next protocol (or a vlan tag) ... We parsed the packet correctly and updated all the needed data structs, but in this particular case, we reported the next protocol using the length ... This length was slightly special and caused us to say that the packet is an ethernet packet, but we setup to process the IP header that was there, so ? boom !!! Should work fine now ? We will report the 2 packets within the same flow, even though one uses Ethernet II and the other is SNAP encapsulated ?. This is correct behavior ... Carter > On Oct 17, 2024, at 9:21?PM, Ming Fu wrote: > > Hi Carter, > > Thank you > > -----Original Message----- > From: Carter Bullard > Sent: Thursday, October 17, 2024 9:17 PM > To: Ming Fu > Cc: argus-info at lists.andrew.cmu.edu > Subject: Re: [ARGUS] argus 5 ethernet parse error > > Hey Ming, > I'll look into it tomorrow !! > Carter > >> On Oct 17, 2024, at 9:15?PM, Ming Fu via Argus-info wrote: >> >> ?Hi, >> >> I noticed that the argus 5.0.0 can crash with an odd Ethernet head. >> I tried the argus from the head of repo as of today, the code crash at the same location. >> >> (gdb) where >> #0 0x000055555556baad in ArgusCreateIPv4Flow () >> #1 0x000055555556c6bd in ArgusCreateFlow () >> #2 0x000055555556c985 in ArgusProcessPacket () >> #3 0x0000555555570c64 in ArgusEtherPacket () >> #4 0x00007ffff7f7cb95 in ?? () from /lib/x86_64-linux-gnu/libpcap.so.0.8 >> #5 0x00007ffff7f7d004 in ?? () from /lib/x86_64-linux-gnu/libpcap.so.0.8 >> #6 0x00005555555752cd in ArgusGetPackets () >> #7 0x00007ffff7f56609 in start_thread (arg=) at pthread_create.c:477 >> #8 0x00007ffff7d04353 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 >> >> Looks like the odd ether type caused argus to not fill the IP address when calling into the in ArgusCreateIPv4Flow() >> >> I attached a small pcap file, the second packet is the one that can crash the argus. It has an ether type of 0x0056 rather than the usual 0x8000. >> >> Regards, >> Ming >> >> >> >> >> >> >> From carter at qosient.com Wed Oct 23 08:50:41 2024 From: carter at qosient.com (Carter Bullard) Date: Wed, 23 Oct 2024 08:50:41 -0400 Subject: [ARGUS] argus 5 ethernet parse error In-Reply-To: References: Message-ID: <0590C51C-B9B8-441F-8BA8-0FCE3ABEA0F7@qosient.com> Hey Ming, Coooool ... yes we're targeting 3-4 releases per year ... so we should release in a few weeks !! Carter > On Oct 23, 2024, at 8:38?AM, Ming Fu wrote: > > ?Hi Carter, > > I tried the main branch; the problem is fixed. > I applied the diff to the 5.0.0 branch. The problem persists, Is there plan to release an update to 5.0.0 branch? > > Regards, > Ming > > -----Original Message----- > From: Carter Bullard > Sent: Friday, October 18, 2024 12:06 PM > To: Ming Fu > Cc: Argus > Subject: Re: [ARGUS] argus 5 ethernet parse error > > Hey Ming, > Problem found, fixed and committed to main branch of argus on GitHub/openargus/argus > The errant packet was an 802.3 LLC SNAP encapsulated packet ? > In 802.3 the next protocol field (ether_type) is dependent on the type ? the field can either be the packet length or the next protocol (or a vlan tag) ... > > We parsed the packet correctly and updated all the needed data structs, but in this particular case, we reported the next protocol using the length ... > This length was slightly special and caused us to say that the packet is an ethernet packet, but we setup to process the IP header that was there, so ? boom !!! > > Should work fine now ? > We will report the 2 packets within the same flow, even though one uses Ethernet II and the other is SNAP encapsulated ?. This is correct behavior ... > > Carter > >> On Oct 17, 2024, at 9:21?PM, Ming Fu wrote: >> >> Hi Carter, >> >> Thank you >> >> -----Original Message----- >> From: Carter Bullard >> Sent: Thursday, October 17, 2024 9:17 PM >> To: Ming Fu >> Cc: argus-info at lists.andrew.cmu.edu >> Subject: Re: [ARGUS] argus 5 ethernet parse error >> >> Hey Ming, >> I'll look into it tomorrow !! >> Carter >> >>>> On Oct 17, 2024, at 9:15?PM, Ming Fu via Argus-info wrote: >>> >>> ?Hi, >>> >>> I noticed that the argus 5.0.0 can crash with an odd Ethernet head. >>> I tried the argus from the head of repo as of today, the code crash at the same location. >>> >>> (gdb) where >>> #0 0x000055555556baad in ArgusCreateIPv4Flow () >>> #1 0x000055555556c6bd in ArgusCreateFlow () >>> #2 0x000055555556c985 in ArgusProcessPacket () >>> #3 0x0000555555570c64 in ArgusEtherPacket () >>> #4 0x00007ffff7f7cb95 in ?? () from /lib/x86_64-linux-gnu/libpcap.so.0.8 >>> #5 0x00007ffff7f7d004 in ?? () from /lib/x86_64-linux-gnu/libpcap.so.0.8 >>> #6 0x00005555555752cd in ArgusGetPackets () >>> #7 0x00007ffff7f56609 in start_thread (arg=) at pthread_create.c:477 >>> #8 0x00007ffff7d04353 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 >>> >>> Looks like the odd ether type caused argus to not fill the IP address when calling into the in ArgusCreateIPv4Flow() >>> >>> I attached a small pcap file, the second packet is the one that can crash the argus. It has an ether type of 0x0056 rather than the usual 0x8000. >>> >>> Regards, >>> Ming >>> >>> >>> >>> >>> >>> >>> > From carter at qosient.com Thu Oct 31 14:29:50 2024 From: carter at qosient.com (Carter Bullard) Date: Thu, 31 Oct 2024 14:29:50 -0400 Subject: [ARGUS] Argus-5.0.2 release Nov 14 Message-ID: Gentle persons, We?re planning on releasing argus-5.0.2 on or just before Nov 14th ?. All the mods, which include new features and bug fixes are now in the main GitHub branches: https://github.com/openargus/argus https://github.com/openargus/clients Any testing that you may be able to do before Nov 14th would be greatly appreciated. We?re having good luck on all target Linux, Mac OS and Windows (10,11) OS, if there is a specific OS you would like tested, please holler !!! The important changes from argus-5.0.0 include: 1. Argus segfault on specific ethernet 802.3 LLC headers 2. Bug fix for IPv6 flows in Ethernet II based tunnels not being reported. This is an important bug fix, as the bug resulted in argus not generating records for key IPv6 traffic. 3. Fixes for parsing specific tunnel headers (Geneve, GRE, VxLan) 4. Argus client filter parsing issues for ?proto? keyword. There are also new features that have been introduced for 5.0.2 1. Packet "capture on protocol" - enhance encapsulation debugging for high performance sensors 2. Encapsulation capture - capture encapsulation headers on a flow record basis 3. Flow spec capture for key tunnel protocols And new argus-clients support for argus?s new features 1. Print tunnel flow spec identifiers - gresaddr, gredaddr, greproto ? 2. Shift to libmaxminddb as the default geoip library (libgeoip as fallback). Some of the new features are intended for tunnel accountability and debugging support, but have a lot of uses for forensics and hunting. And of course lots of bug fixes and tweaking to mature the example programs. Hope all is most excellent, Carter -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 1385 bytes Desc: not available URL: