<\/p>\n
For those that know me, it’s no secret I’m a huge Vyos<\/a> fan.\u00a0 I moved away from pfSense after they released a version that had issues with more than 1 vCPU<\/a>, instead trying Vyos.\u00a0 Once I’d seen how much better it performed under Proxmox I stayed on it and have never looked back. I recently upgraded from Vyos 1.3 to Vyos 1.4<\/a>. 1.4 is a huge step forward for the project as it moves from iptables to nftables.\u00a0 It also brings some great new features, like Flowtable<\/a> (software\/hardware flow offload).\u00a0 This means that once a flow has created a conntrack entry, all future packets that match this flow are fastpath’d through the conntrack service, assuming you have a rule to allow this like so:<\/p>\n This results in better performance and on older\/slower hardware will increase the number of packets-per-second that a device can handle.\u00a0 This is a very good thing!\u00a0 You can see its working if you see [OFFLOAD]<\/strong> in your conntrack table:<\/p>\n <\/p>\n Once I’d turned on Flowtable though, I started to have issues with Firebase Cloud Messaging on my Android phones.\u00a0 It’d keep timing out and I wouldn’t get push notifications until I woke up my phone.\u00a0 I spent ages debugging, Wiresharking, testing with Flowtable on and off.\u00a0 It always would work with Flowtable off, but would fail\/disconnect with Flowtable enabled. In the end, convinced I had found a bunch in nftables (quite the accusation to make!) I logged a bug<\/a> in the Netfilter BugTracker. Turns out I was actually correct, there was<\/em> an issue with PPPoE encapsulation and Flowtable.\u00a0 I’d actually switched ISPs to one that does DHCP (not because of the bug!) and I hadn’t noticed the problem with DHCP, it was good validation to see it was a PPPoE + Flowtable bug. So now my router was working perfectly, but for some reason at some stage I decided to look at the conntrack table statistics.\u00a0 I just like to see how things work “under the hood” I guess.<\/p>\n Wait, what’s this? What the hell is clash_resolve<\/strong> in my conntrack statistics and why is it going up by ~300 a minute? That can’t be a good thing, can it?<\/p>\n I spent a lot of time googling, but trying to find any real information about what it does was hard.\u00a0 There were the main<\/a> links<\/a> I found that offered some insight.<\/p>\n It turns out what a clash_resolve is, at least to my understanding<\/em>, is that when conntrack tries to create an entry, if there’s already an entry for that tuple [source IP, source port, dest port]:[destination ip, source port, dest port] that it will instead shift the source port of the incoming packet so that it’s unique, and create a conntrack entry based on that.\u00a0 I haven’t explained that very well because I never could quite find exactly what was going on myself to a level I felt I understood.\u00a0 Probably I’m too stupid really, so if you have a better plain english explanation I’d welcome it!<\/p>\n
\nI really feel the pfSense project lost its way when they fucked up Wireguard and then lashed out at everyone who was just trying to help get it into FreeBSD<\/a>.\u00a0 But I digress.<\/p>\nFlow Offload (Flowtable) Bug?<\/h2>\n
firewall {\r\n flowtable FastVyos {\r\n description \"Vyos Fast Software Offload Table\"\r\n interface eth1\r\n interface eth0\r\n offload software\r\n }\r\n ipv4 {\r\n forward {\r\n filter {\r\n default-action accept\r\n description \"Filter for packets being forwarded through the router\"\r\n rule 10 {\r\n action offload\r\n description \"Offload Established TCP and UDP Traffic\"\r\n offload-target FastVyos\r\n protocol tcp_udp\r\n state established\r\n state related<\/strong>\r\n }<\/pre>\nconntrack -L -u offload\r\n<snip>\r\ntcp 6 src=x.x.x.x dst=x.x.x.x sport=43392 dport=x packets=2317 bytes=192092 src=192.168.0.5 dst=x.x.x.x sport=x dport=43392 packets=2644 bytes=149748 [OFFLOAD<\/strong>] mark=0 use=2\r\ntcp 6 src=x.x.x.x dst=x.x.x.x sport=55400 dport=x packets=1336 bytes=111233 src=192.168.0.5 dst=x.x.x.x sport=x dport=55400 packets=1535 bytes=88580 [OFFLOAD<\/strong>] mark=0 use=2\r\ntcp 6 src=x.x.x.x dst=x.x.x.x sport=49836 dport=x packets=850 bytes=70559 src=192.168.0.5 dst=x.x.x.x sport=x dport=49836 packets=1060 bytes=58247 [OFFLOAD<\/strong>] mark=0 use=2\r\ntcp 6 src=x.x.x.x dst=x.x.x.x sport=60797 dport=x packets=10014 bytes=827040 src=192.168.0.5 dst=x.x.x.x sport=x dport=60797 packets=10125 bytes=570347 [OFFLOAD<\/strong>] mark=0 use=2\r\nconntrack v1.4.7 (conntrack-tools): 693 flow entries have been shown.\r\n<\/pre>\n
\nI should point out for any PPPoE users out there, the bug is fixed in Linux 6.6.30<\/span> onwards, which the latest Vyos 1.5 rolling images are using.<\/p>\nConntrack Clashes?<\/h2>\n
tim@ferrari:~$ conntrack -S\r\ncpu=0 found=13872 invalid=64978 insert=0 insert_failed=2130 drop=2130 early_drop=0 error=1966 search_restart=0 clash_resolve=1091611 chaintoolong=0 \r\ncpu=1 found=13353 invalid=64876 insert=0 insert_failed=2164 drop=2164 early_drop=0 error=1760 search_restart=0 clash_resolve=1098408 chaintoolong=0<\/pre>\n