python

3 posts with the tag “python”

How to solve 'BPF' object has no attribute 'get_syscall_fnname'?

Dec 31, 2024

This post will show you how to solve ‘BPF’ object has no attribute ‘get_syscall_fnname’ when run bpf program in linux ?

Problem

When you run a python bpf_program in linux, you run this command:

For example, if our bpf program’s name is example.py:

python example.py

The example.py content is:

from bcc import BPF

bpf_source = """
#include <uapi/linux/ptrace.h>

int do_sys_execve(struct pt_regs *ctx) {
  char comm[16];
  bpf_get_current_comm(&comm, sizeof(comm));
  bpf_trace_printk("executing program: %s\\n", comm);
  return 0;
}
"""

bpf = BPF(text=bpf_source)
execve_function = bpf.get_syscall_fnname("execve")
bpf.attach_kprobe(event=execve_function, fn_name="do_sys_execve")
bpf.trace_print()

You get this error:

root@launch:~/linux-observability-with-bpf/code/chapter-4/kprobes# python example.py
Traceback (most recent call last):
  File "example.py", line 15, in <module>
    execve_function = bpf.get_syscall_fnname("execve")
AttributeError: 'BPF' object has no attribute 'get_syscall_fnname'

The error AttributeError: 'BPF' object has no attribute 'get_syscall_fnname' indicates that the BPF class from the bcc module does not have a method named get_syscall_fnname.

Environment

You check your os version by this command:

root@launch:~# cat /etc/os-release

NAME="Ubuntu"
VERSION="16.04.6 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.6 LTS"

Python version:

Python 2.7.17 (default, Apr 15 2020, 17:20:14)
[GCC 7.5.0] on linux2

Solution: Install the bcc dependencies

According to python bcc documents, you should install the libbcc and python bcc into system.

# add key server
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys D4284CDD
# add iovisor to repo
echo "deb https://repo.iovisor.org/apt/bionic bionic main" | sudo tee /etc/apt/sources.list.d/iovisor.list
# update the repo
sudo apt-get update
# install libbcc
sudo apt-get install libbcc
# install python-bcc
sudo apt-get install python-bcc

what is libbcc?

and what is python-bcc:

After all done, you can run the python bpf script again:

root@launch:~/linux-observability-with-bpf/code/chapter-4/kprobes# python example.py
            bash-12522 [001] .... 330817.825407: 0x00000001: executing program: bash

It works!

Final Words + More Resources

My intention with this article was to help others who might be considering solving such problem. So I hope that’s been the case here. If you still have any questions, don’t hesitate to ask me by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

👨‍💻 Python bpf library: bcc
👨‍💻 bcc Python Developer Tutorial

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!

How to check website or url health periodically and automatically by using python script?

Dec 30, 2024

Cowrie

Dev @ Bswen

1. Purpose

In this post, I will demo how to create a python script to check a url(website/webservice)‘s health periodically and automatically .

2. Environment

Python 2.x or 3.x
Linux system

3. The solution

3.1 The environment

Suppose we hava a web service which provides some sort of services to users, now we want it to run always and if it’s not healthy, we should restart the service automatically.

The service URL prefix is:

http://localhost:8080/myservice

And we also have a restart script for this service located at:

/opt/myservice/restart.sh

3.2 Add a health check endpoint for our service

For health check purpose, we should add a dedicated endpoint to be accessed by the health check script, it should be fast , here is our implementation in java language:

  @RequestMapping(method = RequestMethod.GET,value = "hello",headers = "Accept=application/json")
  public @ResponseBody String hello() {
    return "world";
  }

It means that if we caccess the below url:

http://localhost:8080/myservice/hello

We should get this result(http response status code equals 200):

world

Otherwise , the status code should not be 200 , just as the picture shows:

It health check fails, the restart script should be run immediately.

3.3 Setup the logger and imports for python script

Before coding the business, we should setup the environment for our script, just as follows:

import time
import os
import logging
from logging.handlers import TimedRotatingFileHandler

#setup the loggers for daily logging file rolling
logger = logging.getLogger()
logger.setLevel(logging.INFO)
file_handler = TimedRotatingFileHandler("healthcheck_myservice.log", when="midnight", interval=1)
file_handler.suffix = "%Y%m%d"
ch = logging.StreamHandler()
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
file_handler.setFormatter(formatter)
ch.setFormatter(formatter)
logger.addHandler(file_handler)

# global variables.
THE_URL = "http://localhost:8080/myservice/hello"
RESTART_SH = '/opt/myservice/restart.sh'
SLEEP_INTERVAL = 60

In the above code, we use the logging module in python to setup a daily rolling file handler for our program, it would create a log file everyday with the suffix year-month-day. You can view more information about the TimedRotatingFileHandler from this website.

The parameter when="midnight" in TimedRotatingFileHandler means that it would Roll over at midnight.

3.4 Implement the health check function with python

First, we need a function to check the status of the service, just as follows:

def health_check(url):
    try:
        logger.info("start access url %s" % (url))

        import urllib
        a=urllib.urlopen(url) #this is python2 version, the python3 compatible version is listed below

        if a:
            if a.getcode()==200:
                lines = a.readlines()
                if lines is not None and len(lines)==1:
                    if lines[0]=='world':
                        logger.info("health ok")
                        return True
            else:
                logger.info("warning code:"+str(a.getcode()))
    except Exception as e:
        logger.error('health check failed %s ' % (str(e)))
    logger.info("health not ok")
    return False

You can see that , the above function would return True only when the service response code is 200 and the responsecontent is world, or else, it would return False to the caller.

For python 2.x , we can use:

import urllib
urllib.urlopen(url)

For python 3.x , you can replace the urllib.urlopen with follows:

import urllib.request
with urllib.request.urlopen(url) as response:
   html = response.read()

3.5 Implement the periodically checking logic

Now we should create a function to run the health check periodically and restart the service automatically if the check fails.

def check_periodically(url,failed_action_func):
    while True:
        try:
            time.sleep(SLEEP_INTERVAL)
            logger.info("check awake, start check")
            if not health_check(url):
                failed_action_func()
        except Exception as e:
            logger.error('check periodically failed %s ' % (str(e)))
        pass

def restart_server():
    do_script(RESTART_SH)
    pass

def do_script(the_scripts):
    logger.info("=========start script %s======"%the_scripts)
    result = os.popen(the_scripts).read()
    logger.info(result)
    logger.info("=========end   script %s======" % the_scripts)
    pass

3.6 The main function

At last, we should provide the main function to execute the whole script:

if __name__ == '__main__':
    check_periodically(THE_URL,restart_server)
    pass

3.7 Save the script and execute it in background

Then we save the above script as healthcheck_myservice.py, and execute in background as follows:

$ nohup python healthcheck_myservice.py > healthcheck_myservice.out &

Then we can check the log as follows:

$ tail -f healthcheck_myservice.log

We can get these logs:

2021-06-16 15:54:23,775 - root - INFO - health ok
2021-06-16 15:55:23,834 - root - INFO - check awake, start check
2021-06-16 15:55:23,834 - root - INFO - start access url http://localhost:8080/myservice/hello
2021-06-16 15:55:23,839 - root - INFO - health ok
2021-06-16 15:56:23,899 - root - INFO - check awake, start check
2021-06-16 15:56:23,899 - root - INFO - start access url http://localhost:8080/myservice/hello
2021-06-16 15:56:23,906 - root - INFO - health ok

It works!

4. The whole script

#check if my service is running fine, or else, it would restart it

import urllib
import time
import os
import logging
from logging.handlers import TimedRotatingFileHandler

logger = logging.getLogger()
logger.setLevel(logging.INFO)
file_handler = TimedRotatingFileHandler("healthcheck_myservice.log", when="midnight", interval=1)
file_handler.suffix = "%Y%m%d"
ch = logging.StreamHandler()
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
file_handler.setFormatter(formatter)
ch.setFormatter(formatter)
logger.addHandler(file_handler)
#logger.addHandler(ch)

THE_URL = "http://localhost:8080/myservice/hello"
RESTART_SH = '/opt/myservice/restart.sh'
SLEEP_INTERVAL = 60


#perform the health check, return True if the url is accessible.
def health_check(url):
    try:
        logger.info("start access url %s" % (url))
        a=urllib.urlopen(url)
        if a:
            if a.getcode()==200:
                lines = a.readlines()
                if lines is not None and len(lines)==1:
                    if lines[0]=='world':
                        logger.info("health ok")
                        return True
            else:
                logger.info("warning code:"+str(a.getcode()))
    except Exception as e:
        logger.error('health check failed %s ' % (str(e)))
    logger.info("health not ok")
    return False

#check the health for specified period
def check_periodically(url,failed_action_func):
    while True:
        try:
            time.sleep(SLEEP_INTERVAL)
            logger.info("check awake, start check")
            if not health_check(url):
                failed_action_func()
        except Exception as e:
            logger.error('check periodically failed %s ' % (str(e)))
        pass

def restart_server():
    do_script(RESTART_SH)
    pass

def do_script(the_scripts):
    logger.info("=========start script %s======"%the_scripts)
    result = os.popen(the_scripts).read()
    logger.info(result)
    logger.info("=========end   script %s======" % the_scripts)
    pass

if __name__ == '__main__':
    check_periodically(THE_URL,restart_server)
    pass

5. Summary

In this post, I demonstrated how to use python to do health check job in linux.

How to copy file without overwriting destination file using python shutils?

Dec 30, 2024

Cowrie

Dev @ Bswen

1. Purpose

In this post, I will demonstrate how to copy file using python shutils , and will also show you how to copy without overwriting the destination file.

2. The Environment

Python 3
Shutils

3. The code

3.1 How to copy file?

This is the demo that shows how to copy file from src to dest:

import os,sys,shutil

def copy_file(src_path, src_file_name, dest_path, dest_file_name):
    # construct the src path and file name
    src_path_file_name = os.path.join(src_path, src_file_name)

    # construct the dest path and file name
    dest_path_file_name = os.path.join(dest_path, dest_file_name)

    # do the real job
    shutil.copyfile(src_path_file_name, dest_path_file_name)
    print("copy from %s to %s ok" % (src_path_file_name,dest_path_file_name))
    pass

if __name__ == '__main__':
    src_path = sys.argv[1]
    src_file_name = sys.argv[2]
    dest_path = sys.argv[3]
    dest_file_name = sys.argv[4]

    copy_file(src_path,src_file_name,dest_path,dest_file_name)

3.2 Test the copy function

Suppose our working directory structure as follows:

.
└── working_directory/
    ├── copy_file_demo.py
    ├── logo.png
    └── images/
        └── readme.txt

Now I want to copy logo.png to images directory, and name it logo_bak.png, I do this job as follows:

$ python copy_file_demo.py . logo.png images logo_bak.png
copy from ./logo.png to images/logo_bak.png ok

After run the above command, I get this directory structure:

.
└── working_directory/
    ├── copy_file_demo.py
    ├── logo.png
    └── images/
        ├── readme.txt
        └── logo_bak.png

3.3 How to copy without overwritten

The code:

def copy_file_without_overwrite(src_path, src_file_name, dest_path, dest_file_name):
    # construct the src path and file name
    src_path_file_name = os.path.join(src_path, src_file_name)

    # construct the dest path and file name
    dest_path_file_name = os.path.join(dest_path, dest_file_name)

    # test if the dest file exists, if false, do the copy, or else abort the copy operation.
    if not os.path.exists(dest_path_file_name):
        shutil.copyfile(src_path_file_name, dest_path_file_name)
        print("copy from %s to %s ok" % (src_path_file_name, dest_path_file_name))
    else:
        print("already exist %s, copy aborted"%dest_path_file_name)

    pass

3.4 Test the copy again

Our directory is :

.
└── working_directory/
    ├── copy_file_demo.py
    ├── logo.png
    └── images/
        ├── readme.txt
        └── logo_bak.png

Then I execute the below command in the working directory:

$ python copy_file_demo.py . logo.png images logo_bak.png
already exist images/logo.png, copy aborted

It works!

4. About the shutil

Shutil module in Python provides many functions of high-level operations on files and collections of files. … This module helps in automating process of copying and removal of files and directories. shutil. copy() method in Python is used to copy the content of source file to destination file or directory.

Shutil is the abbreviation of shell utility, which implements advanced functions such as file copying, moving, compression, and decompression in Python. It is a Python system module and does not require additional installation.

The commonly used functions in shutil are listed below:

shutil.copyfile(src, dst) Copy from source src to dst. Of course, the premise is that the target address has writable permissions. The exception information thrown is IOException. If the current dst already exists, it will be overwritten
shutil.move(src, dst) move file or rename
shutil.copymode(src, dst) just copy its permissions and other things will not be copied
shutil.copystat(src, dst) copy permission, last access time, last modification time
shutil.copy(src, dst) copy a file to a file or a directory
shutil.copy2(src, dst) is copied on the basis of copy and the last access time and modification time of the file are also copied, similar to cp -p
shutil.copy2(src, dst) If the file systems in the two locations are the same, it is equivalent to a rename operation, just rename; if it is not in the same file system, it is a move operation
shutil.copytree(olddir, newdir, True/Flase) copies olddir to newdir. If the third parameter is True, the symbolic link under the folder will be kept when copying the directory. If the third parameter is False, it will Generate a physical copy in the copied directory to replace the symbolic link
shutil.rmtree(src) recursively delete a directory and all contents in the directory

5. Summary

In this post, I demonstrated how to use shutil in python to copy files from here to there. And I also demonstrated how to avoid overwriting the dest file when copying. Thanks for your reading. Regards.