Debug systemd and define a loglevel

In the past days I had to debug quite old (up2date “enterprise”) software. It ships with multiple bash and sh scripts that together start a java service. All nicely wrapped into a sysv init script. Wrapped in a systemd unit. sucks…

A systemctl start $unit / systemctl stop $unit works most of the time, a systemctl restart $unit never worked. The process forks multiple bash/sh scripts which fork the java process at the end. systemd isn’t able to keep track of the correct pid for the main process, it thinks the main process is one of the sh scripts. The whole startup only works because of RemainAfterExit=yes. A nasty workaround would be to set PIDFile=. The software even creates one, but that isn’t owned by root. systemd declines reading pidfiles that aren’t owned by root.

All in all this sucks hard. The scripts come from the vendor so I cannot easily change them. One way to improve debugging and the lost pid is by increasing the systemd-internal log level:

systemd-analyze set-log-level debug

Afterwards the journal of the restarted unit contains more information. systemd in my case complained that it received a SIGCHLD from a forked process which sometimes(?) triggers systemd to immediately stop the unit again. Now it’s time for futher debugging!

Posted in General, Linux, Short Tips | Leave a comment

systemd unit hardening followup followup

I did some more research on systemd hardening and found another blogpost series that I can highly recommend: https://www.ctrl.blog/entry/systemd-service-hardening.html . The first article is quite similar to mine, but the followup articles go a bit more into details. Check them out!

Posted in General, IT-Security, Linux, Short Tips | Leave a comment

Puppet PQL Queries

PQL syntax can be a bit tricky/ugly. It took me some time to figure this out so I thought sharing it isn’t a bad idea.

Get all nodes with a specific class in their last catalog

 puppet query 'nodes[certname] {resources {type = "class" and title = "CapitalizedClassname"}}' 

This gives us a list of all nodes that have have a class, for example Apache, in their catalog. Maybe want to have a list of all nodes that use a specific module (that could be Apache::Vhosts but not Apache):

 puppet query 'nodes[certname] {resources {type = "class" and title ~ "CapitalizedClassname"}}' 

The query can be combined with more matchers, for example on the certificate name:

 puppet query 'nodes[certname] {certname ~ "^bla-" and resources {type = "class" and title = "CapitalizedClassname"}}' 
Posted in General, Linux, Puppet, Short Tips | Leave a comment

PostgreSQL: Do a VACUUM FULL without exclusive locks!

So, a strange title today. What’s an exclusive lock, what’s a vacuum, why can it be full and what has all this to do with PostgreSQL you might ask yourself.

How PostgreSQL deletes data

In very short: If you delete a row or dataset (also called tuple in the PostgreSQL world) in a table, this is marked as ‘please delete me from disk later whenever you have time but please before the disk is actually full’. PostgreSQL runs a vacuum in the background from time to time. This deletes old tuples from disk and frees up diskspace. This is called autovacuum.

Performance problems with autovacuum

Scanning the data on disk for tuples that are marked to-be-deleted costs IO. People using the database and doing lots of updates/inserts also require IO. If autovacuum is too agressive, the other database operations will be slow. If autovacuum is disabled/ or too slow your disk will fill up. That’s the moment you have to run a VACUUM FULL. This is a manual command that locks the tables with an exclusive lock (nobody else can write) and cleans up the tuples. Also the manual vacuum can often delete a few tuples that the autovacuum cannot.

Using pg_repack

Now comes pg_repack into play! This is a PostgreSQL extension that can cleanup dead tuples like VACUUM FULL, but without an exclusive lock! pg_repack is currently not distributed as a package from the PostgreSQL people, so you’ve to compile it yourself. Afterwards load it into the database and it’s ready to be used! Their documentation is quite good so I won’t copy and paste their installation docs: reorg.github.io/pg_repack/

Posted in General, Linux | Leave a comment

systemd unit hardening followup

at https://blog.bastelfreak.de/2022/01/systemd-unit-hardening/ I blogged about systemd hardening. While doing some research for a followup post I discovered https://docs.arbitrary.ch/security/systemd.html. This covers *a lot* about systemd hardening and general linux optimization. I can highly recommend reading the whole documentation (and it kinda makes my planned blogpost obsolete, so I will postpone it).

Posted in General, IT-Security, Linux, Short Tips | 1 Comment

Migrate CentOS 8 to AlmaLinux

CentOS 8 is dead since the end of 2021 (while CentOS 7 still has support but is really really old). There are a few alternatives. You can upgrade to CentOS Stream, to AlmaLinux or Rocky Linux. CentOS Stream is an odd rolling release distribution that I’m not interested in (I already maintain Gentoo and Arch Linux boxes). I wanted to do a huge article about migrating CentOS to AlmaLinux, sadly that’s really simple:

 dnf --assumeyes update sync reboot curl --silent --remote-name https://raw.githubusercontent.com/AlmaLinux/almalinux-deploy/master/almalinux-deploy.sh bash almalinux-deploy.sh reboot 

It’s a really good idea to review the script before you execute it. it basically updates the repositories and reinstalls all packages.

Posted in Linux, Short Tips | Leave a comment

DNS Setup for own domains

There are many different options to operate your own domain. From a registrar you buy a domain name. The registrar publishes NS records to the registry. Those NS records point to nameservers (or DNS servers or authoritative DNS servers). Registrars usually offer you to host the DNS zone on their nameservers. Think about a DNS Zone as a txt file. Here is a short zone file for bastelfreak.de:

 $ORIGIN . _dmarc.bastelfreak.de 86400 IN TXT "v=DMARC1; p=none; rua=mailto:dmarc-reports@bastelfreak.de; ruf=mailto:dmarc-reports@bastelfreak.de" bastelfreak.de 86400 IN A 144.76.249.219 bastelfreak.de 86400 IN AAAA 2a01:4f8:171:1152::8 bastelfreak.de 86400 IN CAA 0 issue "letsencrypt.org" bastelfreak.de 86400 IN CAA 0 iodef "mailto:tim@bastelfreak.de" bastelfreak.de 86400 IN MX 1 mail.bastelfreak.de. bastelfreak.de 86400 IN NS robotns3.second-ns.com. bastelfreak.de 86400 IN NS robotns2.second-ns.de. bastelfreak.de 86400 IN NS ns3.inwx.eu. bastelfreak.de 86400 IN NS ns1.first-ns.de. bastelfreak.de 86400 IN NS ns2.inwx.de. bastelfreak.de 86400 IN NS ns.inwx.de. bastelfreak.de 86400 IN SOA ns.inwx.de. postmaster.bastelfreak.de. 2021122302 1200 1800 1209600 3600 bastelfreak.de 86400 IN SSHFP 3 2 2006a5beab176e9061e0f8c4ab49097ebc2c4566093822a5e98b09a66a7627e3 bastelfreak.de 86400 IN SSHFP 2 2 eec6ad6e8b3b46ee2d27e24c5c299732bba2ca04f93435573f8391ad1193e116 bastelfreak.de 86400 IN SSHFP 1 2 d36e80e044c0ed9db13c623b4b480ef3b8c5c3d3a96dbb13a5434dc6ff152079 bastelfreak.de 86400 IN SSHFP 4 2 479269b5636c85b0b071cf084e6235168bd14309471da1cbd620a7cef9cab05e bastelfreak.de 86400 IN TXT "v=spf1 ip4:144.76.249.221 ip6:2a01:4f8:171:1152::7 mx -all" blog.bastelfreak.de 86400 IN A 144.76.249.219 blog.bastelfreak.de 86400 IN AAAA 2a01:4f8:171:1152::a default._domainkey.bastelfreak.de 86400 IN TXT "v=DKIM1; k=rsa; s=email; " "p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCkElHw8QtjcK39chikZjBgbN2pc6kI4z4Xa3TsfbtizWEbjnjPuO7WX0mvo+ARJBNeOuBN+Ez6fPo/UOBCjx/mIuHJFY68Vea81qeM5NSYvo16fUxEONojYTPAK7tn+Zf80n+e17MJGADNFTF7YcbhRJtxtK9jeRK0kNOm5qGMxwIDAQAB" mail.bastelfreak.de 86400 IN A 144.76.249.221 mail.bastelfreak.de 86400 IN AAAA 2a01:4f8:171:1152::7 www.bastelfreak.de 86400 IN CNAME bastelfreak.de. 

Often you can edit your zonefile via a webinterface from the registrar. Which alternatives exist? You can host your own nameserver(s)! Most Registries (for example the DENIC, that operates .de tld) require you to provide multiple nameservers, and if you want to have good connectivity around the globe and low latency, you need many many nameservers that are anycasted. So Instead of operating all nameservers for your domains, you can also just run one, called hidden primary, which sends updates via AXFR to the public nameservers, that will be queried by the rest of the internet / are written down via NS resource records at the registry. That’s the approach that I’m doing since some time.

I’ve all my domains registered via INWX. They are cheap and they support a hidden primary server. For each change on a domain, my DNS server needs to send a notify to their special nameserver. That will notify their public servers and they will do an IXFR from my DNS server. To make this more redundant, I also do AXFR to the Hetzner nameservers. This means all domains are served by three INWX DNS servers and three Hetzner DNS servers. I do all this with PowerDNS on Arch Linux.

How does that work you might ask yourself. Let me explain! First, I configured the desired nameserver in the registrar, INWX, to publish them into the .de zone:

 # whois bastelfreak.de | grep Nserver Nserver: ns1.first-ns.de Nserver: ns2.inwx.de Nserver: ns3.inwx.eu Nserver: ns.inwx.de Nserver: robotns2.second-ns.de Nserver: robotns3.second-ns.com 

Afterwards I configured the zone within PowerDNS, as shown in the zone above. In addition, we need to allow AXFR from the Hetzner nameserver and notify the special INWX nameserver when we update the zone:

 pdnsutil get-meta bastelfreak.de Metadata for 'bastelfreak.de' ALLOW-AXFR-FROM = 185.181.104.96, 2a01:4f8:0:a101::a:1, 213.133.105.6, 2a01:4f8:d0a:2004::2, 193.47.99.3, 213.239.242.238, 2001:67c:192c::add:a3 ALSO-NOTIFY = 185.181.104.96 
Posted in Linux | Leave a comment

Setup Gentoo on a Hetzner server

I really like Gentoo for their awesome package manager, Portage. Gentoo is a really flexible distribution that you can customize (and break) in many ways. It’s a good opportunity to learn a lot about linux. I documented the installation process. Given is an EX server from Hetzner, booted into the Debian rescue system.

As a first step, we setup the partitioning + mdadm Raid + LVM + filesystems:

 parted /dev/nvme0n1 --script mklabel gpt parted /dev/nvme1n1 --script mklabel gpt parted /dev/nvme0n1 --script mkpart primary ext3 2048s 4095s parted /dev/nvme1n1 --script mkpart primary ext3 2048s 4095s parted /dev/nvme0n1 --script mkpart primary ext3 4096s 1953791s parted /dev/nvme1n1 --script mkpart primary ext3 4096s 1953791s parted /dev/nvme0n1 --script mkpart primary ext3 1953792s 100% parted /dev/nvme1n1 --script mkpart primary ext3 1953792s 100% parted /dev/nvme0n1 --script set 1 bios_grub on parted /dev/nvme1n1 --script set 1 bios_grub on mdadm --verbose --create /dev/md/0 --level=1 --raid-devices=2 --metadata=1.2 /dev/nvme0n1p2 /dev/nvme1n1p2 mdadm --verbose --create /dev/md/1 --level=1 --raid-devices=2 --metadata=1.2 /dev/nvme0n1p3 /dev/nvme1n1p3 echo 999999999 > /proc/sys/dev/raid/speed_limit_min echo 999999999 > /proc/sys/dev/raid/speed_limit_max until ! grep -q resync /proc/mdstat; do echo "sleeping for 2s"; sleep 2; done mkfs.ext4 -v /dev/md/1 pvcreate --verbose /dev/md/2 vgcreate --verbose vg0 /dev/md/2 lvcreate --verbose --name root --size 50G vg0 mkfs.ext4 -v /dev/mapper/vg0-root mount /dev/mapper/vg0-root /mnt/ 

Next, we download a stage 3 tarball (minimal precompiled Gentoo basically) and verify it:

 url='https://mirror.netcologne.de/gentoo/releases/amd64/autobuilds/current-stage3-amd64-systemd' latest='stage3-amd64-systemd-20220102T170545Z.tar.xz' for file in '' {.CONTENTS.gz,.DIGESTS.asc}; do wget "${url}/${latest}${file}"; done gpg --keyserver hkps://keyserver.ubuntu.com --recv-keys 0xBB572E0E2D182910 gpg --verify "${latest}.DIGESTS.asc" grep " ${latest}.CONTENTS.gz" *.DIGESTS.asc openssl dgst -r -sha512 "${latest}.CONTENTS.gz" 

This will verify the gpg signature in the .asc File. Afterwards we grep the SHA512 checksum for CONTENTS.gz from the .asc file. Then we run openssl to compute the SHA512 checksum on the actual stage 3. Compare the two SHA512 checksums, they have to be identical. If they are, continue with extracting the tarball:

 mv stage3-amd64-hardened-selinux-*.tar.xz /mnt/ cd /mnt/ tar xpvf stage3-*.tar.xz --xattrs-include='*.*' --numeric-owner mount /dev/md/1 /mnt/boot/ 

Now we can prepare the chroot:

 cp --dereference /etc/resolv.conf /mnt/etc/ mount --types proc /proc /mnt/proc mount --rbind /sys /mnt/sys mount --make-rslave /mnt/sys mount --rbind /dev /mnt/dev mount --make-rslave /mnt/dev mkdir /mnt/hostlvm mount --bind /run/lvm /mnt/hostlvm chroot /mnt/gentoo /bin/bash ln -s /hostlvm /run/lvm source /etc/profile export PS1="(chroot) ${PS1}" 

Now we can configure portage:

 mkdir /etc/portage/repos.conf cp /usr/share/portage/config/repos.conf /etc/portage/repos.conf/gentoo.conf echo 'MAKEOPTS="-j6"' >> /etc/portage/make.conf echo 'USE="systemd ipv6"' >> /etc/portage/make.conf sed -i 's/^COMMON_FLAGS.*/COMMON_FLAGS="-march=native -O2 -pipe"/g' /etc/portage/make.conf emerge --sync emerge --ask --verbose --update --deep --newuse @world emerge --depclean echo '=sec-policy/selinux-base-policy-9999 **' >> /etc/portage/package.accept_keywords # set profile to hardened..., see eselect profile list emerge --ask --verbose --unmerge sysvinit eudev echo 'sys-fs/mdadm static' >> /etc/portage/package.use/mdadm echo 'sys-fs/lvm2 lvm2create_initr' >> /mnt/etc/portage/package.use/lvm2 echo 'app-admin/puppet augeas diff doc rrdtool' > /etc/portage/package.use/puppet emerge --ask --autounmask-write --verbose sys-kernel/gentoo-sources sys-apps/systemd vim htop nload iftop iptraf-ng strace lsof gentoolkit intel-microcode pciutils genkernel dstat grub mdadm lvm2 smartmontools dropbear ccze fail2ban tcpdump dev-vcs/git puppet dfc gptfdisk ethtool net-misc/ipcalc ndisc6 echo 'LANG="en_US.utf8"' >> /etc/locale.conf # setup /etc/systemd/network/50-dhcp.network mdadm --detail --scan >> /etc/mdadm.conf sed -i 's/.*LVM=.*/LVM="yes"/' /etc/genkernel.conf sed -i 's/.*MICROCODE=.*/MICROCODE="yes"/' /etc/genkernel.conf sed -i 's/.*SSH=.*/SSH="yes"/' /etc/genkernel.conf sed -i 's/.*BUSYBOX=.*/BUSYBOX="yes"/' /etc/genkernel.conf sed -i 's/.*MDADM=.*/MDADM="yes"/' /etc/genkernel.conf sed -i 's/.*E2FSPROGS=.*/E2FSPROGS="yes"/' /etc/genkernel.conf genkernel all sed -i 's/.*GRUB_DISABLE_SUBMENU.*/GRUB_DISABLE_SUBMENU=y/g' /etc/default/grub sed -i 's/.*GRUB_TIMEOUT=.*/GRUB_TIMEOUT=15/g' /etc/default/grub sed -i 's|.*#GRUB_CMDLINE_LINUX=.*|GRUB_CMDLINE_LINUX="init=/usr/lib/systemd/systemd dolvm domdadm dossh rootfstype=ext4"|' /etc/default/grub for dev in /dev/nvme?n1; do grub-install "${dev}"; done grub-mkconfig -o /boot/grub/grub.cfg # get UUID with `blkid /dev/mapper/vg0-root /dev/md1` and update /etc/fstab passwd mkdir ~/.ssh echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKC4uaKuYzMGK4jlTvPlbnMP9n+gdac65480/eDTMWRw bastelfreak" > ~/.ssh/authorized_keys sed -i 's/.*PermitRootLogin.*/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config systemctl enable systemd-networkd sshd 

Setup language/locales

 echo 'en_US.UTF-8 UTF-8' >> /etc/locale.gen # verify that our locale is present: locale -a | grep en_US.utf8 eselect locale set en_US.utf8 echo 'dns_domain_lo="bastelfreak.org"' >> /etc/conf.d/net echo 'hostname="hypervisor01"' >> /etc/conf.d/hostname . /etc/profile env-update && source /etc/profile 

Setup timesyncd and DNS

 sed -i 's/#NTP=/NTP=ntp1.hetzner.de ntp2.hetzner.com ntp3.hetzner.net/' /etc/systemd/timesyncd.conf rm /etc/localtime ln -s /usr/share/zoneinfo/Europe/Berlin /etc//localtime sed -i 's/#DNS=/DNS=2a01:4f8:0:1::add:1010 2a01:4f8:0:1::add:9999 2a01:4f8:0:1::add:9898/' /etc/systemd/resolved.conf sed -i 's/#Domains=/Domains=bastelfreak.org/' /etc/systemd/resolved.conf systemctl enable systemd-timesyncd systemd-resolved 

Now we can reboot! I suggest to configure a password first and/or create a user. After the first boot we can continue with some fancy pancy stuff:

Setup LLDP:

 echo 'net-misc/lldpd jansson' > /etc/portage/package.use/lldpd emerge --ask lldpd systemctl enable lldpd systemctl start lldpd 

Setup all the fancy dotfiles:

 cd ~ git clone https://github.com/bastelfreak/scripts.git ln -s ~/scripts/vimrc ~/.vimrc ln -s ~/scripts/bashrc ~/.bashrc ln -s ~/scripts/bash_profile ~/.bash_profile mkdir -p ~/.vim/backupdir/ mkdir ~/.vim/ftplugin echo "set colorcolumn=80" >> ~/.vim/ftplugin/tex.vim git clone https://github.com/gmarik/Vundle.vim.git ~/.vim/bundle/Vundle.vim vim +PluginInstall +qall 

And last but not least, you might want to run some virtual machines, so install Qemu and Libvirt:

 echo 'app-emulation/libvirt zeroconf virt-network pcap parted lvm' > /etc/portage/package.use/libvirt echo 'app-emulation/qemu spice virtfs usb usbredir' > /etc/portage/package.use/qemu emerge --ask qemu libvirt ebtables dmidecode openvswitch bridge-utils dispatch-conf emerge --ask qemu libvirt ebtables dmidecode openvswitch bridge-utils systemctl enable ovsdb-server systemctl start ovsdb-server systemctl start ovs-vswitchd systemctl enable ovs-vswitchd ovs-vsctl add-br br0 

You now have a proper Gentoo box. I suggest to configure at least backups and a firewall. Please keep in mind that Gentoo is a rolling release distribution, so some of the commands might be obsolete after some time or you need to configure something differently. Still, this walkthrough should give you a good first impression.

Posted in General, Linux, Virtualization | Leave a comment

systemd-networkd + wireguard configuration

As mentioned in the previous post, networkd is quite nice for network configurations. It can also configure network devices, such as wireguard tunnels. The following config can go into a .netdev file (like /etc/systemd/network/as3668-1.netdev):

 [NetDev] Name=as3668-1 Kind=wireguard MTUBytes=1412 [WireGuard] PrivateKeyFile=/etc/wireguard/as3668-1 ListenPort=1337 [WireGuardPeer] PublicKey=WiN46vGCfAGuH7p6mc+9zLvtmuACdyMtXULETbGP2SM= Endpoint=router01.tld:1337 PersistentKeepalive=5 AllowedIPs=fe80::/64 AllowedIPs=fd00::/8 AllowedIPs=0.0.0.0/0 

The configuration reads the private key from /etc/wireguard/as3668-1, it connects to router01.tld on port 1337, sends a keepalive packet every 5 seconds and allows all traffic to flow through the tunnel. The is configured to 1412 because this goes through a VDSL line. Of course you can configure all of this via Puppet! I published a module at https://forge.puppet.com/modules/puppet/wireguard. The following snippet creates the above configuration:

 wireguard::interface {'as3668-1': source_addresses => ['144.76.249.216', '2a01:4f8:171:1152::11'], public_key => 'WiN46vGCfAGuH7p6mc+9zLvtmuACdyMtXULETbGP2SM=', endpoint => 'router01.tld1337', dport => 1337, input_interface => $facts['networking']['primary'], addresses => [{'Address' => '169.254.0.8/32', 'Peer' =>'169.254.0.7/32'},{'Address' => 'fe80::beef:e/64'},], destination_addresses => [], persistent_keepalive => 5, } 

It will also create a .network file:

 [Match] Name=as3668-1 [Network] DHCP=no IPv6AcceptRA=false IPForward=yes # for networkd >= 244 KeepConfiguration stops networkd from # removing routes on this interface when restarting KeepConfiguration=yes [Address] Address=169.254.0.8/32 Peer=169.254.0.7/32 [Address] Address=fe80::beef:e/64 

Posted in General, Linux, Puppet | Leave a comment

systemd-networkd configuration

Systemd is used in all major Linux distributions. One of the components, systemd-networkd, provides a unified way to manage network interfaces and related settings (like routes, MTU) in a inifile-like way. This is quite awesome because it enables system administrators to use the same configuration style on all their operating systems.

Simple layer 2 IP assignment

Here is a very simple configuration to assign an IPv4 and IPv6 address to one interface

 [Match] Name=uplink [Address] Address=192.168.1.2/32 Peer=192.168.1.1/32 [Address] Address=2a01:4f8:181:1157::3/128 [Route] Gateway=192.168.1.1 GatewayOnLink=yes PreferredSource=192.168.1.2 InitialCongestionWindow=10 InitialAdvertisedReceiveWindow=10 [Route] Gateway=fe80::506e:45ff:fef1:ba4d GatewayOnLink=yes PreferredSource=2a01:4f8:181:1157::3 InitialCongestionWindow=10 InitialAdvertisedReceiveWindow=10 

This can be saved in /etc/systemd/network/ with a .network suffix. I recommend to use the interface name in the file: /etc/systemd/network/uplink.network. This example assumes that the system is in 192.168.1.2/24. But configured is the address as /32. That means it has no layer 2 peers. It has an onlink connection to the router, 192.168.1.1. This is an important detail, especially for larger environments. The configuration does no ARP traffic. And there won’t be much ARP on the switches as well. The same happens with IPv6 and NDP.

Rename an interface

You might ask yourself, why is the interface named uplink? You can rename interfaces with systemd-network too! And I like to name them by their purpose. Throw the following into a .link file (the MAC address is used as an identifier, many other options are available as well):

 # /etc/systemd/network/10-uplink.link [Match] Driver=virtio_net MACAddress=52:54:00:78:63:9c [Link] Description=Uplink interface Name=uplink 
Posted in General, Linux, Virtualization | Leave a comment