Monthly Archives: February 2023

Using Fedora 37+ at CERN

This post is an update to my previous post for Fedora 24.

Much has changed in the past years for Linux at CERN, most notably the surprise end of support for Centos8. In the wake of this news, and unhappy with the new re-shuffling of CentOS in the RHEL development pipeline, several open-source “clones” were created to take the place of the now-EOL CentOS8. CERN, together with FermiLab, chose Alma linux, which is now available and officially supported.

Now that the dust has settled we have much more modern options available (although many experiments are continuing with Centos7 for some time). The current survey of officially supported releases include:

  • Alma/RHEL 9 (5.15 LTS, Python 3.9 LTS)
  • Centos Stream 9 (5.15 LTS, Python3.9 incremental)

For those people who need even newer, the process for using Fedora has gotten even simpler. The default behavior of CERN workstations (as of rhel/cs/alma 9) is to no longer to set the user $HOME directory to their AFS path, but rather a local /home/$USER path. This avoids some of the headaches caused by sssd-krb5, and means that all users need to do to have a “CERN workstation” experience with Fedora (or Ubuntu, Arch, etc. ) is setup krb5 and AFS.

Setting up Fedora 37+ for use at CERN

Starting from a fresh install, we need the dependencies for AFS and kerberos:

$ dnf install -y krb5-workstation sssd-krb5
$ curl -LO /etc/krb5.conf https://linux.web.cern.ch/docs/krb5.conf

You should now be able to get kerberos tokens with $ kinit username@CERN.CH. Don’t forget to enable using your tokens with SSH:

$ cat <<EOF >> ~/.ssh/config
GSSAPITrustDns yes
GSSAPIAuthentication yes
GSSAPIDelegateCredentials yes
EOF 

Now for AFS. There’s been various efforts to make Fedora rpms available via copr or even yum repos directly from openAFS team over the years, but it seems these are inevitably unmaintained. AFS requires kernel modules, so the only sure-fire way is to pull the latest source rpm from OpenAFS and build it yourself:

$ dnf install -y rpm-build krb5-devel ncurses-devel pam-devel swig perl-ExtUtils-Embed
$ curl -LO https://openafs.org/dl/openafs/1.8.9/openafs-1.8.9-1.src.rpm

$ rpmbuild --rebuild --define "build_userspace 1" --define "build_modules 0" openafs-1.8.9-1.src.rpm

$ cd ~/rpmbuild/RPMS/x86_64
$ dnf install -y openafs-client-1.8.9-1.fc37.x86_64.rpm openafs-krb5-1.8.9-1.fc37.x86_64.rpm dkms-openafs-1.8.9-1.fc37.x86_64.rpm openafs-1.8.9-1.fc37.x86_64.rpm
$ echo "cern.ch" > /usr/vice/etc/ThisCell
$ systemctl enable --now openafs-client

Here, we install the minimal AFS client: openafs, openafs-client, openafs-krb5, and dkms-openafs. If you really want the static kmod-openafs, you can set define "build_modules 1" but this is often failing for bleeding-edge kernels. Finally, try it out:

$ kinit userid@CERN.CH
$ aklog
$ klist
Valid starting       Expires              Service principal
02/10/2023 11:25:29  02/11/2023 10:45:25  krbtgt/CERN.CH@CERN.CH
02/10/2023 16:17:35  02/11/2023 10:45:25  afs/cern.ch@CERN.CH

See https://linux.web.cern.ch/docs/kerberos-access for full notes

Addendum: Using CERN repo packages to automate the above

Many of the CERN-maintained packages meant for supporting RHEL/Alma can be used for Fedora as well.

$ cat <<EOF > /etc/yum.repos.d/CERN.repo
[CERN]
name=CentOS-9 Stream - CERN [HEAD]
baseurl=http://linuxsoft.cern.ch/cern/centos/s9/CERN/x86_64
enabled=0
gpgcheck=0
#gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-kojiv2

[CERN-testing]
name=CentOS-9 Stream - CERN - testing [HEAD]
baseurl=http://linuxsoft.cern.ch/cern/centos/s9-testing/CERN/x86_64
enabled=1
gpgcheck=0
#gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-kojiv2

[CERN-source]
name=CentOS-9 Stream - CERN - source [HEAD]
baseurl=http://linuxsoft.cern.ch/cern/centos/s9-testing/CERN/Source
enabled=1
gpgcheck=0
#gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-kojiv2
EOF

As an example, a commonly used helper package is locmap. It’s not directly install-able, due to the rpm being hard-linked to Alma9’s system python 3.9. You’ve already seen how to rebuild a source rpm against your newer kernel, so how about newer python?

$ repoquery --requires locmap
/usr/bin/python3
bind-utils
iproute
krb5-workstation
locmap-plugin-puppet-facts
locmap-plugin-xldap
locmap-release
puppet-afs
puppet-agent >= 1.9
puppet-alternatives
puppet-augeasproviders_core
puppet-augeasproviders_pam
puppet-cernbox
puppet-cernphone
puppet-chrony
puppet-concat
puppet-cvmfs
puppet-eosclient
puppet-firewalld
puppet-inifile
puppet-kerberos
puppet-keytab
puppet-lpadmin
puppet-mailalias_core
puppet-nscd
puppet-postfix
puppet-ssh
puppet-stdlib
puppet-sudo
puppet-systemd
puppet-zoom
python(abi) = 3.9
python3-PyYAML
python3-ldap
python3-netaddr
python3-setuptools
python3.9dist(setuptools)
useraddcern

There’s a couple of options here:

  1. We could install this rpm while ignoring all of it’s dependencies, then manually install them ourselves as needed (tedious)
  2. Update the python3 version requested by the rpm (slightly less tedious)

The relevant fix looks like this:

$ dnf download locmap
$ rpmrebuild -enp locmap-*.rpm

This will open your editor to the rpm .spec file. The relevant lines causing our particular headache are:

Requires:      python(abi) = 3.9
Requires:      python3.9dist(setuptools)

Comment these lines out (or possibly update them and the install paths to your system python version (3.11 for fc37.) The resulting rpm is ready for install in ~/rpmbuild/RPMS/noarch/locmap-*noarch.rpm. If you looked further in the spec file, you’ll see that locmap is essentially a python module like you might install from Pypi, which installs python3.9 site-package files. So don’t forget to install python3.9 for use, along with the modules requested by the rpm:

$ dnf install -y python3.9
$ alternatives --install /usr/bin/python3 python3 /usr/bin/python3.9 1
$ python3 -m ensurepip
$ python3 -m pip install pyyaml netaddr pyyaml setuptools

Rinse and repeat for any dependencies. As you can see, it can turn into a headache, so best avoided if possible.