Manage multiple service instances with systemctl |

Manage multiple service instances with systemctl

Running several instances of a service on a single host gets easier with a systemd hack.

People work on a computer server with devices
Image by :

Subscribe now

Get the highlights in your inbox every week.

Services, services, services. A service is a huge part of computing. You're reading this article on a service. Your computer is running services. The internet is filled with them.

About systemctl

On Linux, the standard way of running and managing services is through the systemd utility and the command systemctl. Its usage is fairly simple: you just need to know the name of the service you want to manage, then you can use this command to start or stop the service, check its status, or do other functions:

# systemctl start httpd

# systemctl stop httpd

# systemctl status httpd

Unit files

A service is defined in a file called a unit file. They are generally located in the directory /usr/lib/systemd/system. Here's a sample list:

# ls /usr/lib/systemd/system
 auditd.service                                        sys-kernel-debug.mount
 autovt@.service                                 sys-kernel-tracing.mount                                NetworkManager-dispatcher.service    syslog.socket                          NetworkManager.service     
 blk-availability.service                    NetworkManager-wait-online.service  

Sometimes you need to run more than one instance of a service on a single host. For example, if you are maintaining websites for several clients, security regulations demand that each client has its own individual instance. This means you need to run a new Apache HTTP service for each, but having a lot of services can be tough to manage. If you have 10 different customers, you would have to create 10 separate unit files to manage them. Fortunately, there is a better way to handle multiple instances of a service.

The @ symbol

Notice that some of the unit files have an @ symbol in their name. This symbol has an interesting significance. As a matter of fact, after installing Apache HTTP Server, you may notice several service unit files, including one that has the @ symbol in its name—httpd@.service.

I will show you how this can make a system administrator's life a little easier, using the Apache HTTP Server for this example:

# dnf -y install httpd

# cd /usr/lib/systemd/system
# ls -d http*
httpd.service  httpd@.service  httpd.service.d  httpd.socket  httpd.socket.d

The @ symbol indicates that a substitution will be made. Systemd will take whatever you type after it and replace the variable %i inside the service unit file. The variable can be seen in this excerpt of the httpd@.service file:

Description=The Apache HTTP Server

ExecStartPre=/bin/mkdir -m 710 -p /run/httpd/instance-%i
ExecStartPre=/bin/chown root.apache /run/httpd/instance-%i
ExecStartPre=/bin/mkdir -m 700 -p /var/lib/httpd/instance-%i
ExecStartPre=/bin/chown apache.apache /var/lib/httpd/instance-%i
ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND -f conf/%i.conf
ExecReload=/usr/sbin/httpd $OPTIONS -k graceful -f conf/%i.conf

Note that the default httpd unit file, httpd.service, does not have the @ symbol in its name and looks different. It is used for a single basic default instance of the Apache HTTP server.

Run multiple instances

To run multiple instances using this special unit file, each one needs a unique name. Suppose you have two clients, The Little Bank and The Big Bank. Name the instances littlebank and bigbank. To manage one, just append its name after the @ symbol. Start each instance with the appropriate command:

  • littlebank: systemctl start httpd@littlebank
  • bigbank: systemctl start httpd@bigbank

Keep in mind that most services need certain configurations to avoid conflicts between different instances. I will not cover all of them here, but Apache needs several things, including unique listen port numbers, PID files, and document roots.

I'll demonstrate by trying to start the littlebank instance. I expect it to fail, but it will provide a useful status message:

# systemctl start httpd@littlebank

# systemctl status httpd@littlebank
● httpd@littlebank.service - The Apache HTTP Server
     Loaded: loaded (/usr/lib/systemd/system/httpd@.service; disabled; vendor preset: disabled)
     Active: failed (Result: exit-code) since Tue 2020-11-10 11:41:20 EST; 1min 58s ago
       Docs: man:httpd@.service(8)
    Process: 2205 ExecStartPre=/bin/mkdir -m 710 -p /run/httpd/instance-littlebank (code=exited, status=0/SUCCESS)
    Process: 2207 ExecStartPre=/bin/chown root.apache /run/httpd/instance-littlebank (code=exited, status=0/SUCCESS)
    Process: 2208 ExecStartPre=/bin/mkdir -m 700 -p /var/lib/httpd/instance-littlebank (code=exited, status=0/SUCCESS)
    Process: 2209 ExecStartPre=/bin/chown apache.apache /var/lib/httpd/instance-littlebank (code=exited, status=0/SUCCESS)
    Process: 2210 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND -f conf/littlebank.conf (code=exited, status=1/FAILURE)
   Main PID: 2210 (code=exited, status=1/FAILURE)
        CPU: 26ms

Nov 10 11:41:20 localhost.localdomain systemd[1]: Starting The Apache HTTP Server...
Nov 10 11:41:20 localhost.localdomain httpd[2210]: httpd: Could not open configuration file /etc/httpd/conf/littlebank.conf>
Nov 10 11:41:20 localhost.localdomain systemd[1]: httpd@littlebank.service: Main process exited, code=exited, status=1/FAIL>
Nov 10 11:41:20 localhost.localdomain systemd[1]: httpd@littlebank.service: Failed with result 'exit-code'.
Nov 10 11:41:20 localhost.localdomain systemd[1]: Failed to start The Apache HTTP Server.

You can see that systemd correctly substitutes the instance name in place of the %i variable in the unit file. However, Apache fails to start because it could not open a configuration file. So, now I'll set up the instances to include creating the necessary configuration files (littlebank.conf and bigbank.conf) in /etc/httpd/conf/:

# ls /etc/httpd/conf
bigbank.conf  httpd.conf  littlebank.conf

Now you can manage each instance with systemctl. Issue start and status commands to each. I've shortened the output to the most relevant details:

# systemctl start httpd@bigbank
# systemctl start httpd@littlebank

# systemctl status httpd@bigbank
● httpd@bigbank.service - The Apache HTTP Server
     Loaded: loaded (/usr/lib/systemd/system/httpd@.service; disabled; vendor preset: disabled)
     Active: active (running) since Tue 2020-11-10 12:26:06 EST; 56min ago
# systemctl status httpd@littlebank
● httpd@littlebank.service - The Apache HTTP Server
     Loaded: loaded (/usr/lib/systemd/system/httpd@.service; disabled; vendor preset: disabled)
     Active: active (running) since Tue 2020-11-10 12:25:58 EST; 55min ago

Here is the process view of these instances. There is more than one because Apache is forking server processes, which it normally does:

# pgrep -a httpd
2834 /usr/sbin/httpd -DFOREGROUND -f conf/littlebank.conf
2835 /usr/sbin/httpd -DFOREGROUND -f conf/littlebank.conf
2836 /usr/sbin/httpd -DFOREGROUND -f conf/littlebank.conf
3061 /usr/sbin/httpd -DFOREGROUND -f conf/bigbank.conf
3062 /usr/sbin/httpd -DFOREGROUND -f conf/bigbank.conf
3064 /usr/sbin/httpd -DFOREGROUND -f conf/bigbank.conf

Put it to use

This small systemd feature is very useful when you need many instances of a service. I hope you will be able to put it to good use in your work.

Penguin driving a car with a yellow background

systemd is the mother of all processes, responsible for bringing the Linux host up to a state where productive work can be done.
Captain Marvel toy fighting

We've rounded up a complete list of our favorite go-to sysadmin tips and tricks.


About the author

Alan Formy-Duval - Alan has 20 years of IT experience, mostly in the Government and Financial sectors. He started as a Value Added Reseller before moving into Systems Engineering. Alan's background is in high-availability clustered apps. He wrote the 'Users and Groups' and 'Apache and the Web Stack' chapters in the Oracle Press/McGraw Hill 'Oracle Solaris 11 System Administration' book. He earned his Master of Science in Information Systems from George Mason University. Alan is a long-time proponent of Open...