It is common in IT projects to encounter difficulties due to the lack of updating projects to current technological solutions, which significantly hinders or completely prevents the introduction of new functionalities in them. This was also the case with the project we were implementing in November 2021.

Our DevOps Engineer/SysAdmin Bartlomiej faced a major challenge of creating a script to fully remotely and automatically upgrade a 12-year-old Linux distribution – Gentoo to the latest Debian 11.

How did he accomplish this? You can find out by reading the article below!

Problem

The client had devices, responsible for playing audio tracks. The Fingoweb team developed a modern Node.js-based application that connects to an API written in PHP running in the AWS cloud. It downloads the music files and playback schedule and starts playing the campaign on its own. Unfortunately, all the devices had a Gentoo distribution installed from around 2010/2011. After analysis, it turned out that it was impossible to run the application on such an old system, due to the outdated kernel and standard C library. Moreover, many of the needed libraries to run the application were missing and there was no compiled ALSA support in the kernel. We didn’t want to incur the cost of replacing all the devices, so we decided to upgrade the operating system.

Solution

The first step was to choose the right operating system. After weighing all the pros and cons, it fell on one of the most popular Linux distributions – Debian 11.

Debian 11 is one of the few distributions that still supports the i386 architecture, which Gentoo no longer does since 2018. This was important because of the processors used in the customer’s devices, and the fact that the new system had to be configured partly from within the existing system – which from a 32-bit operating system would not be able to run tools compiled for a 64-bit system. Due to the deployment of devices in multiple locations, the entire system installation process had to be done remotely. Access to the devices was possible via SSH.

I divided the installation process into 3 main phases:

  1. Download the prepared packages with the system – without interrupting the work of the existing application;
  2. Proper preparation of partitions;
  3. Target system installation.

I took advantage of the fact that the disk that was in the player was divided into several partitions – including one large partition where all the music files are located. Without this, installing Debian 11 from Gentoo would have been impossible.

Another challenge was to maintain the connection to the VPN and the Internet after the system installation. The device could be hooked up to the Internet via a wired Ethernet network, a WiFi network or an LTE modem. I needed to create a suitable mechanism that could read the configuration files, then parse them and create the appropriate configuration for NetworkManager on the target system. The process had to be repeated for WiFi, LTE and VPN.

Unfortunately, after preparing all the configuration, it turned out that it was not possible to use the new debootstrap, whose task was to install the base Debian system on a new partition from within the already installed system.

It was not possible to prepare the new filesystem on a clean partition, because, as before, the Node.js binaries from the new system required a newer glibc ABI than the one available on the running Gentoo. Consequently, I performed an upgrade cycle: Debian 5->6->7->8->9->10->11 fully automatically and without internet access.

I created a tar archive that debootstrap used to initially install the basic system on the new partition. After debootstrap installed the basic system, I had to install all the required packages using apt and tasksel.

Tasksel and APT:

DEBIAN_FRONTEND=noninteractive apt-get install -y 
--force-yes $(aptitude search ~pstandard ~prequired ~pimportant -F%p)

Debian did not download packages from remote repositories, but from the repositories I had previously prepared, which were downloaded in step one to the device’s disk. A separate repository was prepared for each version of the system. Then, after installing the packages, I configured the bootloader, whose task was to boot the corresponding operating system.

In order to automate subsequent updates, I wrote scripts that performed the update automatically after a reboot. For this purpose, cron in Debian was used, which can execute commands after a reboot. Using the appropriate command, lines were added to the crontab. After the completion of each stage, the crontab was cleaned up, and then a line was added that allowed the next stage to start after a reboot. In addition, logs were recorded from the operation of each stage, which helped verify that everything performed successfully after the update was completed.

cron:

crontab -r
(crontab -l ; echo "@reboot /root/upgrade_67 >> /root/upgrade_67.log 2>&1") | crontab -

After all the upgrades were performed, only in Debian 11 did the device access the network. In addition, a full-upgrade was run to ensure that the latest packages would be installed on the system. After the upgrade, old Gentoo partitions are automatically removed and the current partition is expanded to 100% of the disk using parted, growpart and resize2fs programs.

parted /dev/sda --script rm 2
parted /dev/sda --script rm 3
parted /dev/sda --script rm 5
parted /dev/sda --script rm 5
parted /dev/sda --script rm 5

growpart /dev/sda 4
growpart /dev/sda 5
partx -u /dev/sda

update-grub
grub-install /dev/sda

cat << EOF | debconf-set-selections
grub-pc grub-pc/install_devices multiselect     /dev/sda
grub-pc grub-pc/install_devices_empty   boolean false
grub-pc grub-pc/install_devices_disks_changed   multiselect
EOF

Effect

In December 2021, we gave the customer a finished package with instructions for installing the new system. According to the guidelines, the customer was to download the package to the device, unpack it, and then execute the 4 necessary commands:

01_download – this script downloads the package with the packages needed to install and update the system – the device then consumes about 1 GiB of Internet transfer.

02_stop_app – before starting to prepare the disk, close everything that uses the partition on which Debian will be prepared

03_prep_disk – this is to prepare the disk for installation, the script prepares the partition and tools and unzips the ready package of repositories

04_start_upgrade – a script that upgrades the system – the process takes about 1h. Here the transfer consumption is at the level of 50 MiB, only when updating some packages in the final stage of the installation.

The entire installation process, starting from the start of the upgrade until you can log in over SSH, takes a maximum of 2h on the target device. After passing all the stages, the upgraded Debian 11 system is installed on the device. In addition, after the installation and upgrade of the system, an application based on Node.js is installed right away, which allows the customer to take full advantage of our solution.