Testing glibc for Garden Linux 934.11
Why?
Garden Linux has daily versions. Each Garden Linux version has a set of packages installed. This includes packages like glibc, kernel, systemd, containerd and more.
Packages in Garden Linux are either mirrored from debian testing, or are rebuild based on the debian testing packages. A rebuild can have two reasons:
- We need to use a specific build environment/build dependency (e.g. rebuild with a specific GO Version)
- We apply Garden Linux patches on top of the debian sources.
Now why do we need to rebuild a glibc for Garden Linux?
Debian testing is a rolling release. When Garden Linux decides to do a LTS release (e.g. 934.0), debian testing may continue to bump the glibc version. In the case of Garden Linux 934, the glibc version 2.35 was used. Debian testing on the date of 934.0 release (23rd of Oktober 2022) had glibc 2.35. In the meantime glibc was upgraded to 2.36 in debian testing, then testing transitioned to a stable release (bookworm) with a glibc 2.36.
Debian backports security patches only for versions used in any debian distro, thus it is the task of Garden Linux to provide backports for glibc.
How?
Generally, a rebuild is easy in theory. We take the source from debian snapshot, apply our patches, add our gardenlinux<N> version suffix to that package, publish it, and we are good to go - in theory.
However, glibc is a tricky beast. Why?
- glibc needs to be bootstraped
- The compiler used matters
- Runtime permissions for the integrated testsuite matters
- system and kernel configruation also matters for the testsuite (protection mechanisms can prevent some tests to run)
For patching debian’s glibc/2.35 we can use glibc/2.35 to build our own version with our security patch applied. So we created a glibc package with our Garden Linux package pipeline and published it to the 934.11 apt repo. During the build in our build pipeline we noticed that three tests from the testsuite have failed:
FAIL: misc/tst-personality
FAIL: misc/tst-pkey
FAIL: time/tst-clock2
A brief investigation shows, that these tests most likely fail because of issues with the runner executing the tests. So we disable the testsuite that runs automatically after the build, and now run it in a separate environment.
How do we test glibc anyway?
Testing glibc for 934.11
After publishing the glibc package and the matching dependency upgrades to the 934.11 repo, we can now build a 934.11 image with the updated glibc.
Build 934.11
# I always recommend to clone a fresh repository when building 934.11 locally.
git checkout rel-934
# I edit the VERSION file in the repo. Change 934.10 -> 934.11
# With this change, the build now uses the 934.11 apt repo we created.
make metal-dev
Increase 934.11 .raw image size
Everything went well, and we have a .raw image for Garden Linux 934.11 in metal_dev flavour.
Since we need to download the glibc sources and install some build dependencies, we need to increase the raw image.
# Increase the raw file (filled with zeros)
truncate -s 30G .build/metal_dev-amd64-934.11-local.raw
# Start the VM (I needed to remove the --watchdog parameter code in start-vm.sh, since my version of qemu does not support that flag)
./start-vm .build/metal_dev-amd64-934.11-local.raw
# Now we need to extend the partion via.
fdisk /dev/sda
# now we are in fdisk...
# Check free unpartitioned space
F
# Show the partitions
p
# Delete the rootfs partition
d
# Create a new rootfs partition (using the defaults, which use the entire available space)
n
# Change type to Linux root (x86-64) as before
t
# Write the changes
w
# now we exited fdisk..
resize2fs /dev/sda2
# aaaaand we hav an additional 20G of free space
df -H
Get glibc sources
Ok cool. Lets build glibc from our Garden Linux sources, so we can finally run the testsuite for it again. For this, we take the same sources - with our applied CVE patch of course.
cd /usr/src/glibc/
wget https://gitlab.com/gardenlinux/gardenlinux-package-glibc/-/jobs/5279072855/artifacts/download
# Unpack the gitlab artifacts zip
apt install unzip
unzip download
# We can see our patch backport-fix-2023-4911.patch in glibc-2.35/debian/patches
Of course, we should verify the sha256 sums after downloading some artifacts.
80eeec1b7fb6e1d257e580778c69ca8df285f1035ccd9203908b32070e0e1a50 download
e502e2b849dea3124ed5252a467f83792ea9466fa08b3f88f550abc3fda02828 glibc_2.35-4gardenlinux1.debian.tar.xz
da98bf95cb8c0db1837f5483f462ca4b33e955d07bc07d521e8f4cb2789fb3e4 glibc_2.35-4gardenlinux1.dsc
5974ad20d579ffcf50cc9f74cdd0cb74ffdcd7b2712d68d15a19525f03247f0a glibc_2.35-4gardenlinux1_source.changes
bacb01a162ccf9a683c26684dbf8f077fe9b445db5be439db4e69bbcbc3adb49 glibc_2.35.orig.tar.xz
Install build dependencies
Now we need the build dependencies. For that, we change the /ets/apt/preferences.d/gardenlinux for gardenlinux from 900 to 500. This allows us to install build dependencies from debian repositories. This is dangerous, and can lead to a non-functional Frankenstein OS. Spoiler: it does not work with bookworm. I completly ignored my own warning - again - and upgraded libc6 to 2.36 by accident. Did not notice until a late build step of debian’s glibc:
/usr/bin/ld: /lib/x86_64-linux-gnu/libstdc++.so.6: undefined reference to `arc4random@GLIBC_2.36'
So, we need to use debian snapshots for build dependencies.
# put these in /etc/apt/sources.list
deb [check-valid-until=no] https://snapshot.debian.org/archive/debian/20221024T000000Z/ bookworm main
deb [check-valid-until=no] http://snapshot.debian.org/archive/debian-security/20221024T000000Z bookworm-security main
deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20221024T000000Z bookworm-updates main
apt-mark hold libc6
apt update
# Downgrade libcap2, due to a security fix we upgraded libcap2 in 934.10.. but this makes 934.10 incompatible to build glibc..
apt install libcap2=1:2.44-1
apt install libc6-dev
# build dependencies for glibc package
# This may take a while, since some of the build dependencies are pulled from debian snapshot, because we did not mirror all build dependencies for glibc for 934.0
apt-get build-dep -qy --indep-only -o DPkg::Options::=--force-unsafe-io ./_output/*.dsc
# general debian build dependencies
# We need to downgrade perl, so we can install the dpkg-dev package.
# 934.11 has an upgraded perl version, but we in order to install dpkg-dev from debian snapshot, we need to:
apt install perl-base=5.34.0-5
apt install perl=5.34.0-5
# Install the build tool dependencies for the debian package build
apt install --no-install-recommends devscripts build-essential fakeroot dpkg-dev
First, let us quickly verify that libc is still in version 2.35
ldd --version
> ldd (Debian GLIBC 2.35-4gardenlinux1) 2.35
good.
Build glibc (and start tests via debian build environment)
Running the tests is part of the debian/rules.d/build.mk and can be disabled with the nocheck DEB_BUILD_OPTIONS set.
Now we do want to run the tests after we have build it, so we do not set it this time.
cd _output
dpkg-source -x *.dsc src
chown nobody -R .
cd src
su -s /bin/sh -c "set -euE; dpkg-buildpackage -B -a amd64" nobody
… Build runs and my fingers are crossed that the testsuite can run in this environment …
Re-Run tests
export glibc_install="$(pwd)/install"
cd build-tree
../configure --prefix "$glibc_install"
make -j `nproc` check
https://sourceware.org/glibc/wiki/Testing/Testsuite