others-how to run python program as system service by using Systemd ?

Mar 11, 2021

1. Purpose

In this post, I would demo how to use Systemd to run a python program as a linux system service, the python program is just a web server, which is built by flask framework.

2. Environment

Linux system: Ubuntu or CentOS

3. The code of the python program

3.1 app.py

This python program is based on flask-restful framework, The code just start a hello world restful service and bind the port to 8080.

app.py:

from flask import Flask
from flask_restful import Resource, Api

app = Flask(__name__)
api = Api(app)

class HelloWorld(Resource):
    def get(self):
        return {'hello': 'world'}

api.add_resource(HelloWorld, '/')

if __name__ == '__main__':
    app.run(debug=True, port=8080)

3.2 Run the app

When I run the above python code as follows:

python app.py

I got this result:

 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://127.0.0.1:8080/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 717-204-539

Now we have a working python web server, how to run it as a system service, i.e. it can be started automatically when the system starts, if it’s killed, it can be restarted automatically by the system.

4. The systemd service

Systemd is a Linux system tool used to start the daemon and has become a standard configuration for most distributions.

4.1 The history of systemd in short

Before systemd, the linux system use the init command to start services, just as this:

$ sudo /etc/init.d/apache2 start
# Or
$ service apache2 start

This method has two disadvantages:

One is the long start-up time. The init process is started serially, and the next process will be started only after the previous process is started.
The second is that the startup script is complicated. The init process just executes the startup script, regardless of other things. The script needs to deal with various situations by itself, which often makes the script very long.

image-20210311211125121 systemd's author

Above picture show the author of systemd Lennart Poettering.

Systemd was born to solve these problems. Its design goal is to provide a complete solution for system startup and management.

According to Linux conventions, the letter d is short for daemon. The meaning of the name Systemd is that it guards the entire system.

If you use Systemd, you don’t need to use init anymore. Systemd replaces initd and becomes the first process of the system (PID equals 1), and all other processes are its children.

4.2 Systemd’s architecture

You can use the below command to check your systemd’s version:

 systemctl --version

The advantage of Systemd is that it is powerful and easy to use, but the disadvantage is that the system is huge and very complicated. In fact, many people still oppose the use of Systemd because it is too complicated and strongly coupled with other parts of the operating system, which violates the “keep simple, keep stupid” Unix philosophy.

The architecture of systemd:

4.3 Systemd’s concepts for services management

You can use systemd to manage system resources, these resources are called ‘unit’ in systemd.

Here are the list of commands that do the job of service unit management:

sysystemctl status bluetooth.service # view the status of a service
sudo systemctl start/stop/restart/kill/reload apache.service # start / stop / restart / kill / reload a service
systemctl daemon-reload # reload system service configurations

4.4 Systemd’s configurations and useful commands

Each Unit has a configuration file that tells Systemd how to start the Unit.

Systemd reads configuration files from the directory /etc/systemd/system/ by default. However, most of the files stored in it are symbolic links, pointing to the directory /lib/systemd/system/, where the real configuration files are stored.

The systemctl enable command is used to establish a symbolic link relationship between the above two directories.

You can use the below command to list all your service’s configurations:

$ sudo systemctl list-unit-files --type=service

You can check the service’s status by :

$ sudo systemctl status bluetooth.service

If you change your service configurations, you can reload and restart the service as follows:

$ sudo systemctl daemon-reload
$ sudo systemctl restart httpd.service

4.5 The format of Systemd’s configuration file

Now we know that systemd would load service’s configuration file to create and monitor services. How to write a systemd’s configuration file?

Here is a template of systemd’s configuration file:

$ systemctl cat app.service

[Unit]
Description=app daemon

[Service]
Type=forking
ExecStart=/usr/bin/atd

[Install]
WantedBy=multi-user.target

Details of the configurations:

It’s case sensitive in the configuration file
As you can see from the above output, the configuration file is divided into several blocks. The first line of each block is the distinguished name expressed in square brackets, such as [Unit].
Inside each block are some key-value pairs connected by equal signs, such as:
```
[Section]
Directive1=value
Directive2=value
```

4.5.1 The ‘[Unit]’ block

The [Unit] block is usually the first block of the configuration file, which is used to define the metadata of the unit and the relationship between the configuration and other units. Its main fields are as follows

Description: short description
Documentation: document address
Requires: Other Units that the current Unit depends on. If they are not running, the current Unit will fail to start
Wants: other Units that cooperate with the current Unit, if they are not running, the current Unit will not fail to start
BindsTo: Similar to Requires, if the specified Unit exits, it will cause the current Unit to stop running
Before: If the Unit specified in this field is also to be started, it must be started after the current Unit
After: If the Unit specified in this field is also to be started, it must be started before the current Unit
Conflicts: The Unit specified here cannot run at the same time as the current Unit
Condition…: The conditions that the current Unit must meet to run, otherwise it will not run Assert…: The conditions that the current Unit must meet to run, otherwise it will report startup failure

4.5.2 The [Service] block

This block is used to configure the Service. Only Service type Units have this block. Its main fields are as follows:

Type: Define the process behavior at startup. It has the following values.
- Type=simple: the default value, execute the command specified by ExecStart, start the main process
- Type=forking: create a child process from the parent process by fork, and the parent process will exit immediately after creation
- Type=oneshot: a one-time process, Systemd will wait for the current service to exit before continuing to execute
- Type=dbus: The current service is started via D-Bus
- Type=notify: After the current service is started, Systemd will be notified, and then continue to execute
- Type=idle: If other tasks are completed, the current service will run
ExecStart: The command to start the current service
ExecStartPre: The command executed before starting the current service
ExecStartPost: The command executed after starting the current service
ExecReload: The command executed when the current service is restarted
ExecStop: The command executed when the current service is stopped
ExecStopPost: Stop the command executed after its service
RestartSec: The number of seconds between automatic restart of the current service
Restart: Define under what circumstances Systemd will automatically restart the current service.

Possible values include always (always restart), on-success, on-failure, on-abnormal, on-abort, on-watchdog
TimeoutSec: Defines the number of seconds that Systemd waits before stopping the current service
Environment: Specify environment variables

You can get full documentation about the fields here.

4.5.3 The ‘[Install]’ block

[Install] is usually the last block of the configuration file, which is used to define how to start and whether to start when system boots. Its main fields are as follows.

WantedBy: Its value is one or more Targets. When the current Unit is activated (enable), the symbolic link will be placed in a subdirectory of the /etc/systemd/system directory with Target name + .wants suffix
RequiredBy: Its value is one or more Targets. When the current Unit is activated, the symbolic link will be placed in the subdirectory formed by the Target name + .required suffix under the /etc/systemd/system directory
Alias: the alias that the unit can be used to start
Also: When the current Unit is activated (enable), other Units will be activated at the same time

4.5.4 What’s a `Target` in systemd

When starting the computer, a large number of Units need to be started. If every time you start, you have to specify which Units you need for this start, which is obviously very inconvenient. Systemd’s solution is Target.

Simply put, Target is a Unit group that contains many related Units. When a Target is started, Systemd will start all Units in it. In this sense, the concept of Target is similar to a “state point”. Starting a Target is like starting to a certain state.

In the traditional init startup mode, there is the concept of RunLevel, which is similar to the role of Target. The difference is that RunLevels are mutually exclusive. It is impossible for multiple RunLevels to start at the same time, but multiple Targets can start at the same time.

The corresponding relationship between Target and traditional RunLevel is as follows:

Traditional runlevel      New target name     Symbolically linked to...

Runlevel 0           |    runlevel0.target -> poweroff.target
Runlevel 1           |    runlevel1.target -> rescue.target
Runlevel 2           |    runlevel2.target -> multi-user.target
Runlevel 3           |    runlevel3.target -> multi-user.target
Runlevel 4           |    runlevel4.target -> multi-user.target
Runlevel 5           |    runlevel5.target -> graphical.target
Runlevel 6           |    runlevel6.target -> reboot.target

4.6 How to check a system service’s logs?

Now we know how to define a systemd service , but when the service starts, how to find the logs of it?

You can use the below commands to debug the logs of the service:

sudo journalctl -u nginx.service  # show all logs of the service

sudo journalctl -u nginx.service --since today # show today's logs of the service

sudo journalctl -u nginx.service -f # tail the logs of the service at real time

5. Configure our app as systemd service

I have introduced the key points of Systemd , and we have a working app that can run in console, Now we can start to configure our app as the systemd service.

5.1 The service configuration file

We can define the service as follows (Suppose our working directory is /root/app-flask-restful ):

Create a file named app.service in current working directory:

[Unit]
Description=My App Service
After=multi-user.target

[Service]
Type=simple
User=root
WorkingDirectory=/root/app-flask-restful
ExecStart=/usr/bin/python /root/app-flask-restful/app.py
Restart=always

[Install]
WantedBy=multi-user.target

Explanation of the above configuration file:

Our service description is ‘My App Service’
The service is a simple service that start by a command ‘/usr/bin/python /root/app-flask-restful/app.py’
The ‘WantedBy=multi-user.target’ indicates that this service should start in runlevels 3, 4 and 5” in SysVinit systems: it tells systemd that this service should be started as part of normal system start-up, whether or not a local GUI is active.
The service runs as root, and its working directory is /root/app-flask-restful

5.2 Tell Systemd we want to create a service

First, we should copy the above file to the specifc location that systemd would load files:

$ sudo cp app.service /lib/systemd/system

Now we should reload the services:

$ sudo systemctl daemon-reload

5.3 Start the service

Now starts the service as follows:

$ sudo  systemctl enable app.service
$ sudo  systemctl start app.service

5.4 Verify the service

$ sudo  systemctl status app.service
● app.service - My App Service
   Loaded: loaded (/usr/lib/systemd/system/app.service; enabled; vendor preset: disabled)
   Active: active (running) since 2021-03-10 17:39:04 CST; 4s ago
 Main PID: 22165 (python)
    Tasks: 3
   Memory: 32.6M
   CGroup: /system.slice/app.service
           ├─22165 /usr/bin/python /root/app-flask-restful/app.py
           └─22166 /usr/bin/python /root/app-flask-restful/app.py

You can see that the service is running!

Now it works.

6. If you encounter problems when starting systemd services

6.1 Error #1

For example, if you start the service with such errors:

Failed to start app.service: Unit app.service is not loaded properly: Exec format error.
See system logs and 'systemctl status app.service' for details.

Debug the error as follows:

root@launch-advisor-20191120:~/app-flask-restful# systemctl status app.service
● app.service - My App Service
   Loaded: error (Reason: Exec format error)
   Active: inactive (dead)

Mar 10 16:34:11 launch-advisor-20191120 systemd[1]: /lib/systemd/system/app.service:8: Executable path
root@launch-advisor-20191120:~/app-flask-restful#
root@launch-advisor-20191120:~/app-flask-restful# systemctl status app.service
● app.service - My App Service
   Loaded: error (Reason: Exec format error)
   Active: inactive (dead)

Mar 10 16:34:11 launch-advisor-20191120 systemd[1]: /lib/systemd/system/app.service:8: Executable path is not absolute: python /root/app-flask-restful/app.py
root@launch-advisor-20191120:~/app-flask-restful# which python
/usr/bin/python

If you encounter the above error, you should check your ‘[service]’ block , where you define how to start your service. For example ,you should add the following fields to it:

[Service]
WorkingDirectory=/root/app-flask-restful
User=root
....

6.2 Error #2

If you encounter this error:

root@launch-advisor-20191120:~/app-flask-restful# systemctl restart app.service
root@launch-advisor-20191120:~/app-flask-restful# systemctl status app.service
● app.service - My App Service
   Loaded: loaded (/lib/systemd/system/app.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2021-03-10 16:46:52 CST; 1s ago
  Process: 21156 ExecStart=/usr/bin/python /root/app-flask-restful/app.py (code=exited, status=1/FAILURE)
 Main PID: 21156 (code=exited, status=1/FAILURE)

Mar 10 16:46:52 launch-advisor-20191120 python[21156]:     from flask import Flask
Mar 10 16:46:52 launch-advisor-20191120 python[21156]: ImportError: No module named flask
Mar 10 16:46:52 launch-advisor-20191120 systemd[1]: app.service: Main process exited, code=exited, status=1/FAILURE
Mar 10 16:46:52 launch-advisor-20191120 systemd[1]: app.service: Failed with result 'exit-code'.
Mar 10 16:46:52 launch-advisor-20191120 systemd[1]: app.service: Service hold-off time over, scheduling restart.
Mar 10 16:46:52 launch-advisor-20191120 systemd[1]: app.service: Scheduled restart job, restart counter is at 5.
Mar 10 16:46:52 launch-advisor-20191120 systemd[1]: Stopped My App Service.
Mar 10 16:46:52 launch-advisor-20191120 systemd[1]: app.service: Start request repeated too quickly.
Mar 10 16:46:52 launch-advisor-20191120 systemd[1]: app.service: Failed with result 'exit-code'.
Mar 10 16:46:52 launch-advisor-20191120 systemd[1]: Failed to start My App Service.

You should make sure that this program can run at console， please run as follows:

$  python app.py

7. Summary

In this post, I introduced the Systemd’s basic configuration, commands and debug methods , I created a python web server and make it run as a system service by creating a Systemd service. Actually you can use this method to run any application that can run in background. Thanks for your reading.