Advanced usage

pix4dmapper outside of the local environment

Introduction

It is possible to run pix4dmapper outside of the local environment by giving appropriate arguments to the EngineWrapper object. For instance, one can specify the command to be used for running pix4dmapper (base_command), or the Pix4D credentials (credentials) if one needs to log in everytime a processing is performed. It is also possible to provide callbacks (callbacks) to run specific code just before and after creating and processing projects.

As an example, we show below how to run pix4dmapper using the Docker technology.

Processing within a Docker image

There are two ways to run the pix4dmapper stored within a Docker image:

  1. Same user in the Docker image and in the host running the SDK (run_as_root=False)

    This requires having a user with the same user ID (eg. 1000) in the Docker image. It does not matter if the user name is different.

  2. As root (run_as_root=True)

    This requires the SDK user to be able to change the ownership as the files created by pix4dmapper are owned by root. Typically on Ubuntu, one needs to add a file in /etc/sudoers.d/ containing USER ALL=(ALL) NOPASSWD: /bin/chown where USER is the username of the user running the SDK. This allows USER to execute chown with root priviledge without using any authentification. Please refer to your OS documentation or system administrator for further questions.

The following code samples show how to run pix4dmapper in a stateless Docker mode. First, docker_cmd_line creates a Docker command line to run pix4dmapper. It bind-mounts some directories in the Docker container, for example, the directory containing the images and the working directory:

def docker_cmd_line(docker_config, mounts=None):
    """Creates a Docker command line."""

    # run Docker in stateless mode, delete the container once the command ends
    command = ["docker", "run", "--rm"]

    if docker_config.run_as_root is False:
        command.extend(("--user", str(os.getuid())))

    # bind mount directory inside the Docker container
    if mounts is not None:
        for mount in mounts:
            mount = os.path.abspath(mount)
            command.extend(("-v", ":".join((mount, mount))))

    # add the Docker image:tag and the path to Pix4dMapper
    command.extend((":".join((docker_config.image, docker_config.tag)),
                    docker_config.path_to_pix4dmapper))
    return command

The sudo_chown function changes the ownership of the work_dir back to the SDK user in the case Docker runs as root:

import os
import subprocess


def sudo_chown(directory):
    """Change the ownership of directory to the current user."""
    user = str(os.getuid())
    group = str(os.getgid())

    command = ("sudo", "chown", "-R", ":".join((user, group)), directory)
    subprocess.run(command)

A DockerConf data structure is created, containing all the information on the Docker and pix4dmapper installation. It is used to obtain a Docker command line using the docker_cmt_line defined above. Callbacks to the sudo_chown function are configured, to run the docker image as root:

from collections import namedtuple

import pix4dengine as p4d
from pix4dengine.utils.enginewrapper import Callbacks
from pix4dengine import create_project
from pix4dengine.enginewrapper import EngineWrapper
from pix4dengine.constants.processing import ProcessingStep

DockerConf = namedtuple("DockerConf", ("image",
                                       "tag",
                                       "path_to_pix4dmapper",
                                       "engine_email",
                                       "engine_password",
                                       "run_as_root"))

# named tuple containing all the configuration we need
docker_conf = DockerConf(image=docker_image,
                         tag=docker_tag,
                         path_to_pix4dmapper="/opt/pix4dmapper/bin/pix4dmapper",
                         engine_email=engine_email,
                         engine_password=engine_password,
                         run_as_root=True)


base_command = docker_cmd_line(docker_conf, mounts=[image_dir, work_dir])


# we need to define callbacks to change the working directory ownership back to the user
def on_success():
    sudo_chown(work_dir)


# create an EngineWrapper with the specific Docker configuration
engine_wrapper = EngineWrapper(base_command=base_command,
                               credentials=(docker_conf.engine_email,
                                            docker_conf.engine_password))

# create a project
project = create_project("test", image_dir,
                         engine_wrapper=engine_wrapper,
                         work_dir=work_dir,
                         callbacks=Callbacks(on_success=on_success))

# run the initial processing step
p4d.process_project(project, engine_wrapper,
                    steps=ProcessingStep.CALIB,
                    callbacks=Callbacks(on_success=on_success))

Using a similar logic, one can run pix4dmapper over, for instance, SSH on a remote cluster or inside different virtual environments.

Project validation during processing

The Pix4Dengine enables performing checks on the quality report after each processing step is completed. This is done by passing a callable to process_project(), via the argument stop_condition. The callable should accept one argument — an instance of Report. Here we show how this can be used for checking calibration quality and stopping processing if the calibration is judged insufficient based on some user-defined criterion.

First, we define a callable that returns False if the calibration quality in the project is too poor, and True otherwise. The callable also defines a message, providing a description of why the validation does not succeed.

class CalibrationChecker:
    def __init__(self):
        self.message = "calibration quality is insufficient"

    @staticmethod
    def __call__(report):
        from pix4dengine.utils.report import Quality
        return all(q == Quality.SUCCESS for q in report.calibration_quality_status())

An instance of the callable is passed to process_project(), which runs the check and raises a specific exception if the validator returns False. The exception contains the message attribute that was defined in the callable:

from pix4dengine.utils.errors import FailedValidation

try:
    p4d.process_project(project, engine_wrapper,
                        steps=(ProcessingStep.CALIB,
                               ProcessingStep.DENSE,
                               ProcessingStep.ORTHO),
                        validator=CalibrationChecker())

except FailedValidation as stop:
    print("Stopping processing, %s." % str(stop))
    engine_wrapper.logout()