ArticlesLinuxOpen Source Software

EOL of Ubuntu Xenial: Beware of these Ubuntu LTS upgrade gotchas

In these days the widely used Linux distribution Ubuntu 16.04, codenamed Xenial Xerus, will be end of life (EOL) and won't receive any updates anymore. Looking at the EOL date of the previous LTS versions 14.04 (April 25th 2019) and 12.04 (April 28th 2017), the EOL date of 16.04 will probably be set to April 28th or 29th.

Canonical, the company behind Ubuntu, continue to support 16.04 LTS for another 3 years for paying customers through the Extended Security Maintenance plan.

Although an upgrade of Ubuntu 16.04 to a newer LTS version, such as 18.04 or 20.04, usually works very well, there are always some gotchas. The base system around the Linux Kernel, such as login, shell and filesystems usually don't change too much – making an upgrade painless. However packages from the repositories contain newer application versions. And this is where the gotchas are.

We've put together a list of applications to watch out for when upgrading from Ubuntu 16.04 to the currently newest Ubuntu 20.04 LTS.

RIP Ubuntu 16.04 Xenial Xerus

MySQL 8

Ubuntu 16.04 (Xenial) used the MySQL 5.7 version which has been around for a while and is still supported by Oracle. However with the upgrade to Ubuntu 20.04 (Focal), MySQL received a drastic change: MySQL is now installed as version 8.0.

For new applications and empty databases this might be alright, but for existing applications there are some breaking changes. Especially the way the (default) user authentication works on MySQL 8.x has changed a lot. This could potentially break applications using a MySQL database – at least it would mean that the applications all need to support the newer (and more secure) authentication method.

A downgrade to MySQL 5.7 is possible, however not by using the Ubuntu repositories. Follow the article "Install MySQL 5.7 on Ubuntu 20.04 Focal (and properly avoid MySQL 8.0 packages)" for a how to guide.

Nginx

Nginx also has made a big version jump. In Xenial, Nginx was installed with version 1.10.0. In the new Ubuntu Focal, Nginx 1.18.0 is installed from the Ubuntu repositories.

Luckily almost all Nginx configuration should still work, however there are some major changes under the hood, mainly how modules are packaged and loaded into Nginx. Also see article "Important Nginx configuration changes after upgrading Ubuntu from 16.04 to 20.04" for a more detailled description.

Dynamic module loading

Since Ubuntu 18.04 (Bionic), the modules are loaded dynamically instead of hard-compiled into the Nginx binary. You can spot the difference by using dpkg and checking for the installed Nginx packages.

ck@ubuntu1604:~$ dpkg -l|grep nginx
ii  nginx-common                          1.10.3-0ubuntu0.16.04.4                    all          small, powerful, scalable web/proxy server - common files
ii  nginx-extras                          1.10.3-0ubuntu0.16.04.4                    amd64        nginx web/proxy server (extended version)

ck@ubuntu1804:~$ dpkg -l|grep nginx
ii  libnginx-mod-http-auth-pam             1.14.0-0ubuntu1.7                               amd64        PAM authentication module for Nginx
ii  libnginx-mod-http-cache-purge          1.14.0-0ubuntu1.7                               amd64        Purge content from Nginx caches
ii  libnginx-mod-http-dav-ext              1.14.0-0ubuntu1.7                               amd64        WebDAV missing commands support for Nginx
ii  libnginx-mod-http-echo                 1.14.0-0ubuntu1.7                               amd64        Bring echo and more shell style goodies to Nginx
ii  libnginx-mod-http-fancyindex           1.14.0-0ubuntu1.7                               amd64        Fancy indexes module for the Nginx
ii  libnginx-mod-http-geoip                1.14.0-0ubuntu1.7                               amd64        GeoIP HTTP module for Nginx
ii  libnginx-mod-http-headers-more-filter  1.14.0-0ubuntu1.7                               amd64        Set and clear input and output headers for Nginx
ii  libnginx-mod-http-image-filter         1.14.0-0ubuntu1.7                               amd64        HTTP image filter module for Nginx
ii  libnginx-mod-http-lua                  1.14.0-0ubuntu1.7                               amd64        Lua module for Nginx
ii  libnginx-mod-http-ndk                  1.14.0-0ubuntu1.7                               amd64        Nginx Development Kit module
ii  libnginx-mod-http-perl                 1.14.0-0ubuntu1.7                               amd64        Perl module for Nginx
ii  libnginx-mod-http-subs-filter          1.14.0-0ubuntu1.7                               amd64        Substitution filter module for Nginx
ii  libnginx-mod-http-uploadprogress       1.14.0-0ubuntu1.7                               amd64        Upload progress system for Nginx
ii  libnginx-mod-http-upstream-fair        1.14.0-0ubuntu1.7                               amd64        Nginx Upstream Fair Proxy Load Balancer
ii  libnginx-mod-http-xslt-filter          1.14.0-0ubuntu1.7                               amd64        XSLT Transformation module for Nginx
ii  libnginx-mod-mail                      1.14.0-0ubuntu1.7                               amd64        Mail module for Nginx
ii  libnginx-mod-nchan                     1.14.0-0ubuntu1.7                               amd64        Fast, flexible pub/sub server for Nginx
ii  libnginx-mod-stream                    1.14.0-0ubuntu1.7                               amd64        Stream module for Nginx
ii  nginx-common                           1.14.0-0ubuntu1.7                               all          small, powerful, scalable web/proxy server - common files
ii  nginx-extras                           1.14.0-0ubuntu1.7                               amd64        nginx web/proxy server (extended version)

This also means that the Nginx main configuration file (/etc/nginx/nginx.conf) needs to be adjusted. The upgrade process does install a new nginx.conf – unless you chose to keep the existing configuration file (which makes sense in a lot of servers). The following include line must exist now at the top of the config file:

ck@ubuntu:~$ head -n 4 /etc/nginx/nginx.conf
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

Without this include line, Nginx will fail to (re-) start when additional modules, such as GeoIP, are required.

Enabling SSL

The way SSL is enabled in a server context (virtual host) has also changed. Although the old style (using ssl on;) still works, it is marked as deprecated. That's the perfect (and last) moment to change the configuration.

In the old configuration style, SSL was enabled in the following way:

root@ubuntu:~# cat /etc/nginx/sites-enabled/api.example.com.conf
server {
  listen 443;
  server_name api.example.com;
  access_log /var/log/nginx/api.example.com.access.log include_ssl_params;
  error_log /var/log/nginx/api.example.com.error.log;

  ssl on;
  ssl_certificate /etc/nginx/ssl.crt/wildcard.example.com.crt;
  ssl_certificate_key /etc/nginx/ssl.key/wildcard.example.com.key;
[...] 

The new way is to add an ssl keyword in the listen configuration:

root@ubuntu:~# cat /etc/nginx/sites-enabled/api.example.com.conf
server {
  listen 443 ssl;
  server_name api.example.com;
  access_log /var/log/nginx/api.example.com.access.log include_ssl_params;
  error_log /var/log/nginx/api.example.com.error.log;

  ssl_certificate /etc/nginx/ssl.crt/wildcard.example.com.crt;
  ssl_certificate_key /etc/nginx/ssl.key/wildcard.example.com.key;
[...] 

New TLS 1.3

Nginx 1.18.0 on Focal also adds (by default) a new TLS version: TLSv1.3. This means that the ssl_protocols configuration parameter now allows and should be extended with TLSv1.3:

ck@ubuntu2004:~$ cat /etc/nginx/nginx.conf | grep ssl_protocols
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE

Note that the Ubuntu-provided nginx.conf still contains the deprecated TLS protocols 1.0 and 1.1. For security reasons, only TLSv1.2 and TLSv1.3 should be enabled.

Crash on logrotate

In Xenial, the default logrotate command to reload Nginx is set to invoke-rc.d nginx rotate. However under certain conditions, Nginx's default logrotate configuration on 16.04 did not work. When this command was replaced by service nginx reload, the log rotation worked correctly.

However after an upgrade to Ubuntu 20.04 and with the adjusted logrotate config (using service nginx reload) still in place, this will now crash Nginx.

Mar  7 00:00:11 focal systemd[1]: logrotate.service: Succeeded.
Mar  7 00:00:11 focal systemd[1]: Finished Rotate log files.
Mar  7 00:00:13 focal systemd[1]: nginx.service: Main process exited, code=dumped, status=11/SEGV
Mar  7 00:00:13 focal systemd[1]: nginx.service: Killing process 627492 (nginx) with signal SIGKILL.
Mar  7 00:00:13 focal systemd[1]: nginx.service: Killing process 627493 (nginx) with signal SIGKILL.
Mar  7 00:00:13 focal systemd[1]: nginx.service: Killing process 627492 (nginx) with signal SIGKILL.
Mar  7 00:00:13 focal systemd[1]: nginx.service: Killing process 627493 (nginx) with signal SIGKILL.
Mar  7 00:00:13 focal systemd[1]: nginx.service: Failed with result 'core-dump'.

In this case revert the reload command in Nginx's logrotate configuration (/etc/logrotated.d/nginx) to the default:

postrotate
     invoke-rc.d nginx rotate >/dev/null 2>&1
endscript

Keepalived

Keepalived is a software to enable virtual IP addresses using the VRRP protocol (often used in routers and firewalls). An upgrade should not cause any problems, however after an upgrade from Xenial to Focal, certain health checks stopped working.

If you are using the pidof command inside a health check (vrrp_script) and pidof returns multiple PIDs in the response (which is the case for Nginx for example), Keepalived will not like this and the check will fail with an error. This can be seen in syslog where log events such as these show up:

Apr 16 09:27:27 ubuntu killall5[645003]: only one argument, a signal number, allowed

Although no difference can be spotted between the pidof output on a Xenial and on a Focal machine, the updated Keepalived application seems to handle the output differently.

The quickest way to solve this is to replace pidof with pgrep.

Netplan

Ubuntu introduced Netplan as new network interface configuration, replacing the old but very stable ifupdown, in 17.10. The LTS version after this, 18.04 (Bionic), therefore also contains Netplan as default network configuration tool.

If you are upgrading from Xenial, the existing configuration in /etc/network/interfaces is still kept and still works. The ifupdown package is upgraded to 0.8.35. The directory /etc/netplan exists after the upgrade but should be empty.

But if you are installing a new Ubuntu 18.04 or 20.04, you will find that ifupdown is not installed and instead the network interface(s) are defined and configured in /etc/netplan/01-netcfg.yaml.

To see a comparative basic interface configuration:

root@upgraded:~# cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto ens160
iface ens160 inet static
	address 192.168.1.13
	netmask 255.255.255.0
	network 192.168.1.0
	broadcast 192.168.1.255
	gateway 192.168.1.1
	# dns-* options are implemented by the resolvconf package, if installed
	dns-nameservers 1.1.1.1 8.8.8.8
	dns-search example.local


root@newinstall:~# cat /etc/netplan/01-netcfg.yaml 
# This file describes the network interfaces available on your system
# For more information, see netplan(5).
network:
  version: 2
  renderer: networkd
  ethernets:
    ens160:
      addresses: [ 192.168.1.13/24 ]
      gateway4: 192.168.1.1
      nameservers:
          search: [ example.local ]
          addresses:
              - "1.1.1.1"
              - "8.8.8.8"

Note that Netplan can cause serious problems, especially when used in LXC containers. See LXC container not getting configured static IP addresse because of netplan. If Netplan cannot be used (complex interface configurations using virtual bridges are known to cause problems), or you simply prefer the old configuration way, remove the netplan.io package and install ifupdown package instead.

Python and its modules

When installing a new Ubuntu 20.04, Python will (by default) only be installed using Python 3. Python 2 can be installed through the python2 package.

The biggest challenge is that on a new Focal installation /usr/bin/python does not exist or does not point to either python3 or python2. In order to do this, update-alternatives (which is basically a sophisticated symbolic link) can be used to point to a specific version:

root@focal:~# update-alternatives --install /usr/bin/python python /usr/bin/python3 1

When upgrading from Xenial (and you've had python installed), you will find that both python2 (2.7.18) and python3 (3.8.5) packages are installed.

However in Focal a lot of python modules have disappeared as they are not built as deb packages anymore and are therefore not found in the Ubuntu repositories. python-pywbem is a great example. The package has been around for years, but since Ubuntu 20.04, this package has disappeared along with many other Python modules.

The preferred way to install Python modules has now shifted to pip3 where modules can be installed from Python's own module repository.

root@focal:~# apt-get install python3-pip python3-setuptools
root@focal:~# pip3 install pywbem

Time synchronisation

For time synchronisation you most likely used NTP for many years, but you might be surprised that starting with Ubuntu 20.04, NTP has been replaced by yet another Systemd service called systemd-timesyncd. Although ntp package can still be installed, by default it is systemd-timesyncd which will be reponsible for keeping the system clock in sync.

This also means that a NTP configuration file /etc/ntp.conf is ignored – instead /etc/systemd/timesyncd.conf is used. A very basic timesyncd.conf file pointing to a NTP server:

root@upgraded:~# cat /etc/systemd/timesyncd.conf
[Time]
NTP=ntp1.example.com

See timesyncd.conf(5) for details.

Experienced more gotchas?

Did you experience more gotchas on other applications/packages? Let us know in the comments below.

Claudio Kuenzler
Claudio has been writing way over 1000 articles on his own blog since 2008 already. He is fascinated by technology, especially Open Source Software. As a Senior Systems Engineer he has seen and solved a lot of problems - and writes about them.

Leave a reply

Your email address will not be published. Required fields are marked *

More in:Articles