Tag Archives: linux

Vyos and the mystery conntrack counter

1 Reply

VyOS

Router Go Fast!

For those that know me, it’s no secret I’m a huge Vyos fan.  I moved away from pfSense after they released a version that had issues with more than 1 vCPU, instead trying Vyos.  Once I’d seen how much better it performed under Proxmox I stayed on it and have never looked back.
I really feel the pfSense project lost its way when they fucked up Wireguard and then lashed out at everyone who was just trying to help get it into FreeBSD.  But I digress.

Flow Offload (Flowtable) Bug?

I recently upgraded from Vyos 1.3 to Vyos 1.4. 1.4 is a huge step forward for the project as it moves from iptables to nftables.  It also brings some great new features, like Flowtable (software/hardware flow offload).  This means that once a flow has created a conntrack entry, all future packets that match this flow are fastpath’d through the conntrack service, assuming you have a rule to allow this like so:

firewall {
    flowtable FastVyos {
        description "Vyos Fast Software Offload Table"
        interface eth1
        interface eth0
        offload software
    }
    ipv4 {
        forward {
            filter {
                default-action accept
                description "Filter for packets being forwarded through the router"
                rule 10 {
                    action offload
                    description "Offload Established TCP and UDP Traffic"
                    offload-target FastVyos
                    protocol tcp_udp
                    state established
                    state related
                }

This results in better performance and on older/slower hardware will increase the number of packets-per-second that a device can handle.  This is a very good thing!  You can see its working if you see [OFFLOAD] in your conntrack table:

conntrack -L -u offload
<snip>
tcp      6 src=x.x.x.x dst=x.x.x.x sport=43392 dport=x packets=2317 bytes=192092 src=192.168.0.5 dst=x.x.x.x sport=x dport=43392 packets=2644 bytes=149748 [OFFLOAD] mark=0 use=2
tcp      6 src=x.x.x.x dst=x.x.x.x sport=55400 dport=x packets=1336 bytes=111233 src=192.168.0.5 dst=x.x.x.x sport=x dport=55400 packets=1535 bytes=88580 [OFFLOAD] mark=0 use=2
tcp      6 src=x.x.x.x dst=x.x.x.x sport=49836 dport=x packets=850 bytes=70559 src=192.168.0.5 dst=x.x.x.x sport=x dport=49836 packets=1060 bytes=58247 [OFFLOAD] mark=0 use=2
tcp      6 src=x.x.x.x dst=x.x.x.x sport=60797 dport=x packets=10014 bytes=827040 src=192.168.0.5 dst=x.x.x.x sport=x dport=60797 packets=10125 bytes=570347 [OFFLOAD] mark=0 use=2
conntrack v1.4.7 (conntrack-tools): 693 flow entries have been shown.

 

Once I’d turned on Flowtable though, I started to have issues with Firebase Cloud Messaging on my Android phones.  It’d keep timing out and I wouldn’t get push notifications until I woke up my phone.  I spent ages debugging, Wiresharking, testing with Flowtable on and off.  It always would work with Flowtable off, but would fail/disconnect with Flowtable enabled. In the end, convinced I had found a bunch in nftables (quite the accusation to make!) I logged a bug in the Netfilter BugTracker. Turns out I was actually correct, there was an issue with PPPoE encapsulation and Flowtable.  I’d actually switched ISPs to one that does DHCP (not because of the bug!) and I hadn’t noticed the problem with DHCP, it was good validation to see it was a PPPoE + Flowtable bug.
I should point out for any PPPoE users out there, the bug is fixed in Linux 6.6.30 onwards, which the latest Vyos 1.5 rolling images are using.

Conntrack Clashes?

So now my router was working perfectly, but for some reason at some stage I decided to look at the conntrack table statistics.  I just like to see how things work “under the hood” I guess.

Wait, what’s this? What the hell is clash_resolve in my conntrack statistics and why is it going up by ~300 a minute? That can’t be a good thing, can it?

tim@ferrari:~$ conntrack -S
cpu=0 found=13872 invalid=64978 insert=0 insert_failed=2130 drop=2130 early_drop=0 error=1966 search_restart=0 clash_resolve=1091611 chaintoolong=0 
cpu=1 found=13353 invalid=64876 insert=0 insert_failed=2164 drop=2164 early_drop=0 error=1760 search_restart=0 clash_resolve=1098408 chaintoolong=0

I spent a lot of time googling, but trying to find any real information about what it does was hard.  There were the main links I found that offered some insight.

It turns out what a clash_resolve is, at least to my understanding, is that when conntrack tries to create an entry, if there’s already an entry for that tuple [source IP, source port, dest port]:[destination ip, source port, dest port] that it will instead shift the source port of the incoming packet so that it’s unique, and create a conntrack entry based on that.  I haven’t explained that very well because I never could quite find exactly what was going on myself to a level I felt I understood.  Probably I’m too stupid really, so if you have a better plain english explanation I’d welcome it!

But I did find the cause of the problem.  My Vyos router runs a caching nameserver, it’s my home router so it makes sense for it to cache most DNS lookups.  I have a Zabbix Server at home too and it generates A LOT of DNS requests.  I found as soon as I turned off my Zabbix server that the clash_resolve stopped incrementing.  After looking at the conntrack table I realised there were hundreds of conntrack entries between the DNS Server on the router and my Zabbix server:

<snip snip>
udp      17 19 src=192.168.0.253 dst=192.168.0.1 sport=48288 dport=53 packets=2 bytes=124 src=192.168.0.1 dst=192.168.0.253 sport=53 dport=48288 packets=2 bytes=189 mark=0 use=1
udp      17 12 src=192.168.0.253 dst=192.168.0.1 sport=56102 dport=53 packets=1 bytes=68 src=192.168.0.1 dst=192.168.0.253 sport=53 dport=56102 packets=1 bytes=84 mark=0 use=1
udp      17 12 src=192.168.0.253 dst=192.168.0.1 sport=56240 dport=53 packets=2 bytes=136 src=192.168.0.1 dst=192.168.0.253 sport=53 dport=56240 packets=2 bytes=201 mark=0 use=1
udp      17 28 src=192.168.0.253 dst=192.168.0.1 sport=38695 dport=53 packets=2 bytes=124 src=192.168.0.1 dst=192.168.0.253 sport=53 dport=38695 packets=2 bytes=189 mark=0 use=1
udp      17 10 src=192.168.0.253 dst=192.168.0.1 sport=33689 dport=53 packets=2 bytes=128 src=192.168.0.1 dst=192.168.0.253 sport=53 dport=33689 packets=2 bytes=193 mark=0 use=1
udp      17 13 src=192.168.0.253 dst=192.168.0.1 sport=36236 dport=53 packets=2 bytes=128 src=192.168.0.1 dst=192.168.0.253 sport=53 dport=36236 packets=2 bytes=193 mark=0 use=1
udp      17 10 src=192.168.0.253 dst=192.168.0.1 sport=49932 dport=53 packets=2 bytes=116 src=192.168.0.1 dst=192.168.0.253 sport=53 dport=49932 packets=2 bytes=181 mark=0 use=1
udp      17 3 src=192.168.0.253 dst=192.168.0.1 sport=54581 dport=53 packets=2 bytes=126 src=192.168.0.1 dst=192.168.0.253 sport=53 dport=54581 packets=2 bytes=191 mark=0 use=1
udp      17 1 src=192.168.0.253 dst=192.168.0.1 sport=40209 dport=53 packets=2 bytes=126 src=192.168.0.1 dst=192.168.0.253 sport=53 dport=40209 packets=2 bytes=191 mark=0 use=2
udp      17 12 src=192.168.0.253 dst=192.168.0.1 sport=48388 dport=53 packets=2 bytes=136 src=192.168.0.1 dst=192.168.0.253 sport=53 dport=48388 packets=2 bytes=201 mark=0 use=1
udp      17 4 src=192.168.0.253 dst=192.168.0.1 sport=51367 dport=53 packets=2 bytes=128 src=192.168.0.1 dst=192.168.0.253 sport=53 dport=51367 packets=2 bytes=193 mark=0 use=1
conntrack v1.4.7 (conntrack-tools): 167 flow entries have been shown.

Fixing Conntrack

And here’s the fix: Those connections don’t need to be in conntrack! There’s no NAT going on and I’m not doing any firewalling.  It’s local LAN traffic.  So the fix is to put in an exception rule, so that all traffic from my LAN, talking to my DNS Server on the router, bypasses conntrack.  Meaning there’s no state at all, it’s just a normal routed packet.
Note that you have to create a rule in both directions, otherwise the router sending back replies generates a conntrack entry.

The configuration looks like this, placed under the “system conntrack” stanza:

show system conntrack
 ignore {
     ipv4 {
         rule 10 {
             description "Ignore Conntrack for LAN DNS Requests to Router"
             destination {
                 address 192.168.0.1
                 port 53
             }
             inbound-interface eth1
             protocol udp
             source {
                 address 192.168.0.0/24
             }
         }
         rule 200 {
             description "Ignore Conntrack for LAN DNS Replies from Router"
             destination {
                 address 192.168.0.0/24
             }
             protocol udp
             source {
                 address 192.168.0.1
                 port 53
             }
         }
     }

With this in place, the conntrack table statistics for clash_resolve have stopped going up rapidly.  I still see some increasing, but that’s expected.  In fact clash_resolve isn’t even a problem as such, it’s just saying a clash was noticed and resolved.

I’ve also saved myself 650+ entries in the conntrack table that I didn’t need:

[Before the change above was made]
tim@ferrari# conntrack -L -s 192.168.0.0/24 -d 192.168.0.0/24
<snip>
conntrack v1.4.7 (conntrack-tools): 665 flow entries have been shown.

And my Vyos router is as performant as ever!

Tim

grsecurity logo

grsecurity on a Xubuntu laptop

Leave a Reply

I installed xubuntu on my now very old Dell XPS m1330 the other day.  Windows 10 just wasn’t cutting it for me, though that’s probably because I had an excess of crap installed.  It was taking upwards of 5 minutes to reboot though, so I thought I’d try something else.

Xubuntu was the obvious choice, Gnome can die in a fire and KDE while nice is too UI heavy for what I want.  I love XFCE, it’s small, clean and does a great job, so xubuntu got the nod.

Once installed and working, I then downloaded the 4.1.3 kernel source and the latest grsecurity patch for it.  Patched the source and fired up make menuconfig.  Ubuntu being ubuntu it comes with pretty much every freaking option, module and setting defaulted to yes.  Rather than piss about making a nice small custom kernel, I just went with all the defaults, then turned on pretty much every grsec feature.  The few items I kept disabled are:

They’re only minor things (well, the RBAC system isn’t really “minor”), all of the main memory protection features (thanks, PaX) and the other grsecurity hardening features are enabled.

Then it’s just a matter of making sure all the right packages are installed to be able to do a “make-kpkg –initrd kernel_image” and waiting for a very long time.  Oh and it helps to set the CONCURRENCY_LEVEL to 2, which is how many Core’s the CPU has.  Then you wait about 3 hours…

Finally you end up with a .deb that you install and off you go.  Install it and reboot and….

It worked first go. Not that I really expected otherwise.  The only problems encountered are the expected ones, some binaries don’t like the hardened memory protections, so those protections have to turned off on a per-binary basis.  So it’s apt-get install pax-utils and apt-get install paxctl.

The binaries I adjusted flags for are:

  • chrome
  • thunderbird
  • python3
paxctl -cpm /path/to/binary

c: creates a pax header, m: disables mProtect, p: disables pageexec

The only problems I’ve faced apart from this are issues with the sound module.  Under a default ubuntu kernel the sound just works.  Under my compiled kernel, the module needs to be removed and re-added for sound to work, and then it fails if you suspend the laptop.  I’m 99% sure this isn’t anything to do with grsecurity, but rather the fact it’s a vanilla kernel source, not a heavily-patched ubuntu kernel source with fixes for all those sorts of things.  I’ll get to the bottom of it at some stage.

But the laptop works and works well.  I’m not using the proprietary Nvidia drivers, just the nouveau ones.  Suspend works.  It’s still fast and browsing is quick, despite all of the PaX and grsecurity options turned on, some of which have a known performance impact (I’m looking at you Userland Dereference and Memory Sanitize)

The whole process has been easy, quick and painless.  The hardest part has been waiting for the kernel to compile.  When I have a bit more free time I’ll go through and build an image for just this laptop, disabling all the drivers and options that are totally unnecessary.  I’ll end up with a much leaner kernel that’s quick to compile.  But this image I have now could be given to anyone with a x64 system and it’d boot and work perfectly.

Thanks spender, pipacs and emese for their work on PaX/grsecurity.

Tim

UPDATE: A couple of updates to this post.  Firstly, the issue with sound was caused by CONFIG_GRKERNSEC_SYSFS_RESTRICT being set.  Disabling this, building again works.  I also found that I was getting slower performance, disabling uderef on the command line as well as slab sanitization has fixed this, giving me excellent performance again.  Beause they’re command line, I can re-enable them easily withouth having to recompile.  My full linux boot command is:

BOOT_IMAGE=/boot/vmlinuz-4.1.4-grsec root=/dev/sda1 ro reboot=w nouveau.runpm=1 nouveau.pstate=1 pax_nouderef pax_sanitize_slab=off pax_extra_latent_entropy

Finally, I sat down over the weekend and stripped out all the unneeded modules and settings.  Ubuntu by default sets a lot of debugging features, so all those are turned off now in the aims of squeezing a bit more performance out of the laptop.  Plus now my kernel image is ~25Mb, instead of ~250mb.

Raspberry Pi

Leave a Reply

My Pi arrived yesterday.  I plugged it in for 30 minutes while I baked a cake for Sarah’s birthday. Seems pretty neat, I’m impressed that the distribution I chose to test with (OpenELEC) detected the Wifi Dongle straight away and just worked with it.

The major issue at the moment is I don’t have a decently sized MicroSD card.   I have a 2G one, which was fine for some basic testing.  And it’s actually fine for OpenELEC as well, but it’s a GUI oriented thing and I don’t want that, I want something that’s basically a Linux Distro (i.e. Rasbian)

Will have to get it a decent SD Card today.  And the case that my workmate 3D Printed for it should be arriving in the next day or two.

I’ll be crabbed off if I can’t get grsecurity running on it…

grsecurity RBAC system

Leave a Reply

A few notes I wrote down about enabling the (very intense) grsecuriy RBAC system on micro.

  1. If you have role_allow_ip defined and you try and “do something” from an IP not in that allowed list – you’ll end up in the default role.  That can be hard to figure out at first, you know you have a role setup for the user, but you’re not ending up in it.  That’s why.
  2. A subject in a less specific policy with an “i” will override a more specific subject.i.e.This “/” subject:subject / o {    /bin        rxi}will win over this:subject /bin/bash o {}/bin/bash will use the polices from the first subject, not the second.
  3. To be able to unlink (delete!) a file, you need both d and the w subject flags.  d alone isn’t enough.
  4. Be VERY careful with the C subject flag.  It will kill everything with the same IP address, silently.  Your MYSQL server might be dead and you won’t know it.
  5. Shared Libraries (.so) need the rx flag, x alone isn’t enough.
    subject /bin/bash {    

    /     
/lib            rx
}
  6. The -V flag to gradm -E will give some useful information!

Building MPD

Leave a Reply
micro:~/mpd-0.18.7> make distclean ; ./configure --disable-dependency-tracking --enable-libmpdclient --enable-alsa --enable-bzip2 --enable-curl --enable-ao --enable-ffmpeg --enable-flac --enable-httpd-output --enable-id3 --disable-ipv6 --enable-lame-encoder --enable-lsr --enable-mad --enable-mikmod --enable-mms --enable-modplug --enable-mpg123 --enable-pipe-output --enable-recorder-output --enable-shout --enable-vorbis --enable-vorbis-encoder --enable-zzip --prefix=/usr/local/mpd ; make

modsecurity

Leave a Reply

I re-enabled modsecurity, this time with the PCRE JIT.

To do this I had to do the following:

Build the latest version of pcre.
Install it in /usr/local/pcre

Modify the apache startup to use LD_PRELOAD to load the new libpcre.so instead of the standard system one.
I could have overwritten this using ld.so.preload but that’s a system-wide changes and I don’t know what else I might break, seeing as most other things will be compiled against the old version.

Finally I had to modify the apache2 binary so that it’s allowed to create code at runtime (we are doing JIT here, afterall)

paxctl -cm /usr/lib/apache2/mpm-prefork/apache2

Now it seems to be working fine.

PCRE was compiled like so:

./configure –prefix=/usr/local/pcre –enable-jit –enable-pcre16 –enable-pcre32 –enable-utf –enable-unicode-properties –disable-static

Full RELRO for Bitlbee

Leave a Reply

Took me ages of fucking around to get bitlbee to compile with full RELRO.

In the end I had to hack the makefile.
At line 182 (the line where it gets linked) I had to add the following:

180 $(OUTFILE): $(objects) $(subdirs)
181 @echo '*' Linking $(OUTFILE)
182 @$(CC) $(objects) $(subdirobjs) *-march=native -O2 -fstack-protector-all -fpic -pipe -Wl,-z,relro,-z,now* -o $(OUTFILE) $(LDFLAGS_BITLBEE) $(LFLAGS) $(EFLAGS)