Photogrammetric Processing Algorithms

Core Processing

Camera Calibration

Guidelines and Best Practices

It is strongly recommended to use basic setting templates, and only adjust particular settings if required. See TEMPLATES. For end-to-end settings by use case, see Starter settings.

Recommendations for the pipeline parameter

The calibration uses a selection of algorithms represented by the pipeline parameter. It is the main factor influencing the calibration processing outcome. Each algorithm has a specific purpose, advantages, and limitations. No pipeline is one-size-fits-all.

Possible pipeline values:

  • Scalable All-purpose calibration pipeline including support for large-scale, corridor and uncorrected GPS accuracy-referenced captures. It does not support multispectral rigs.

  • TrustedLocationOrientation This calibration pipeline is designed for projects with highly accurate relative image positions and orientations, for example, from RTK/PPK.

  • LowTexturePlanar Legacy calibration pipeline (‘Alternate’ in Mapper) designed for aerial nadir images with accurate geolocations and homogeneous or repetitive content of flat-like scenes and rig cameras. It ignores the image orientations.

Recommendations for ImagePairSettings

These are the settings used for image pair matching.

Image pair matching defines the strategy for selecting the set of pairs used by calibration in the feature matching algorithm step. Feature matching is the process of finding the same feature in both images of an image pair.

It is possible to try matching features on all possible image pairs, matching each image with every other image in the dataset by setting match_all=True. However, the processing time of this approach does not scale well with input data size, resulting in long processing times for even moderately sized datasets.

Reoptimization

Reoptimization is a calibration feature that updates the internal and external camera parameters when new inputs (such as marks on images or GCPs and MTPs) have been added after calibration has been run.

It is a faster method for updating calibration and improving accuracy than re-running calibration.

Currently, the only known usage of reoptimization is after AutoGCP automatic GCP mark detection.

See reoptimize().

Point Cloud Densification

Usage

A densified point cloud is obtained by calling densification().

The area of the sparse point cloud subject to densification can be limited by using a region of interest (ROI) constructed with the help of the geometry utilities.

Certain segments on images can be masked, thus excluding the 3D points projecting onto these areas from being considered for densification. Masks excluding sky and water segments can be obtained using the sky and water segmentation algorithm.

The densification can also produce sparse synthetic depth maps if this is enabled by the compute_depth_maps configuration parameter. These can be further densified and used in mesh geometry generation.

In order to visualize a point cloud in third-party software, it must be exported as a file in one of the following formats: LAS, Cesium, or SLPK, using the respective functions write_pcl(), write_pcl(), and write_pcl().

Guidelines and Best Practices

To configure the densification algorithm, it is strongly recommended to use the configuration templates available for common use cases. See TEMPLATES. Individual configuration parameters can be adjusted if necessary. More details on the parameters are given in the API reference for Settings.

Gaussian Splatting

Usage

The Gaussian splatting model is generated by calling gen_gs(). The model is generated from input cameras, calibrated cameras, and a partitioned, densified point cloud.

The resulting model is stored on disk using the glTF format with the PIX4D_gaussian_splatting extension.

The algorithm can optionally produce two additional outputs during generation:

  • Synthetic depth maps: depth information derived directly from the model, for use in the 3D mesh processing step.

  • Point cloud: a point cloud generated from the Gaussian model, for use in 3D mesh and DSM processing steps.

Important

These outputs (depth maps and point clouds) can only be produced during the model generation phase by setting compute_point_cloud or compute_depth_maps to True. They cannot be generated as a post-processing step.

Guidelines and Best Practices

Configuration Templates

It is mandatory to use the pre-defined settings templates available in TEMPLATES. Choose the template that matches your capture method:

  • TERRESTRIAL Optimized for handheld/terrestrial captures, such as those from PIX4Dcatch.

  • DRONE Tailored for aerial nadir or oblique flights.

  • MIXED_DRONE_TERRESTRIAL Designed for projects that combine both aerial and terrestrial imagery.

Sky Masking

To reduce artifacts from the sky around objects, it is highly recommended to provide sky masks via the mask_map parameter.

  • Format: unsigned 8-bit rasters the same size as input images.

  • Values: use 0 for unmasked (content) and 255 for masked (sky).

Geometric Constraints

  • Depth maps: if input cameras contain depth information, geometry accuracy may improve.

  • Confidence: for every depth map provided, a pairing confidence map must be included.

  • ROI: a 3D region of interest (ROI) can be defined as a 2D polygon with elevation constraints.

Visualization and Export

The GaussianSplatting class provides a bounding_box() method to help with spatial indexing and a scene_ref_frame() method to ensure the model aligns correctly with other project layers, such as orthomosaics or meshes, in third-party GIS or CAD software.

Gaussian splatting models can be stored in and loaded from an OPF project with the experimental ProjectWriter / ProjectReader helpers, the same way as point clouds:

project = ProjectWriter(...)
project.add_item_composite(obj=gs_model)

project = ProjectReader(...)
gs_model = project.load_obj(gsplat.GaussianSplatting)

DSM/DTM

Two types of DSM generation are available: Triangulation and IDW. At the API level, this choice is made by selecting one of the two available settings classes:

  • Triangulation
    Recommended for rural areas, agriculture, or low-texture captures.
    Provides a less accurate DSM. Typically produces fewer empty cells; rare isolated gaps may appear. Tile-border gaps are mitigated by increased border overlap (hardcoded).
  • IDW
    Recommended for urban areas, construction sites, and buildings.
    Provides good accuracy, but can produce empty cells and outliers. Gap cover is configurable via gap interpolation using interpolation_nn_count.

Recommendations for IDW

It is recommended to use preset and default settings for the parameters:

  • gsd
    The GSD value as generated by calibration.
    It is recommended to use the value computed by calib.analytics.compute_gsd(scene).
  • interpolation_nn_count
    When set, gaps that might appear in the DSM are completed by interpolation. The value sets the number of nearest neighbors to be used for interpolation. Since the surface is approximated with neighboring points, some inaccuracies can be expected. A larger value allows larger gaps to be filled, increasing the likelihood of a gap-free surface, but also allowing larger inaccuracies in the output. A lower value ensures higher accuracy in the output, but increases the risk of generating surfaces with gaps.
    Recommended value: 10 [default value]
    Typical range: 10-30
  • smoothing_median_radius
    When set, the algorithm performs post-processing surface smoothing of the DSM. With low values, the final DSM contains sharper features and edges, and vice versa. For urban areas (constructions, parks, roads), where surface heights change often and rapidly, it is recommended to use a value in the lower range. For surveying hilly areas, slopes, or scenes with large objects, higher values can be used.
    Recommended value: 12 [default value]
    The values should be in the range 0-20:

    • 0 No smoothing is applied.

    • 1 Tries to keep sharp features such as corners and edges of buildings. Value 1 is considered a very low value.

    • 20 Tries to smooth areas considering a large surrounding area. Sharp edges will be smoothed out. Areas that are not too planar are smoothed and may become planar. Value 20 is considered a high value.

Orthomosaic

Guidelines and Best Practices

  • The NADIR capture pattern is best suited for nadir aerial photography with the cameras pointing approximately 90 degrees down.

  • The OBLIQUE capture pattern is best suited for oblique aerial and terrestrial photography, as well as for nadir aerial photography when the cameras are tilted, typically in the direction of the flight.

  • For the same dataset, changing the capture pattern from NADIR to OBLIQUE increases processing time.

  • The FAST algorithmic pipeline gives good quality results for NADIR photography.

  • The DEGHOST algorithmic pipeline can be used when moving objects (cars, people, and so on) are present in the scene and should be removed from the orthomosaic.

  • The FULL and DEGHOST algorithmic pipelines require more processing time and more temporary disk space than the FAST pipeline.

  • Combinations of algorithmic pipeline and capture pattern for typical use cases:

    • Inspection of runways, surveying, mapping, large cadasters: FAST + NADIR.

    • Inspection of motorways, inspection of busy streets for road work quality: DEGHOST + NADIR.

    • Inspection of tower structures: FULL + OBLIQUE.

    • 3D modeling of objects in busy traffic locations: DEGHOST + OBLIQUE.

    • A car crash scene: FULL + OBLIQUE.

    • Busy construction sites (using a crane camera): DEGHOST + NADIR.

    • Terrestrial photography: FULL + OBLIQUE.

    • Agricultural or Thermal surveying: FAST + NADIR or FULL + NADIR.

Recommended settings for these combinations are also available as TEMPLATES (NADIR, DEGHOST, and OBLIQUE).

When generating an orthomosaic after calibrate(), use NADIR. The fast calibration is a 2D / nadir algorithm developed specifically for flat terrains.

Optical radiometry

Optical radiometric correction is applied typically to the multispectral images, and is enabled by passing Radiometry to make_input_cameras(). The weather condition at time of acquisition WeatherCondition is a user input. The correction type applied to each sensor is selected automatically based on image and sky sensor metadata.

For more general information on optical radiometry, refer to the Pix4D support articles.

Which sensor is corrected with which correction type can be inspected in the quality report.

Python members:

  • NONE - No optical radiometric correction is applied (radiometry data is not available or a correction is not applicable).

  • CAMERA - Corrects camera settings and properties. Without a reflectance target, estimates at-camera radiance, when a reflectance target is used, estimates reflectance.

  • SKY_SENSOR_WITHOUT_ANGLE - In addition to CAMERA corrections, uses the irradiance measured by a downwelling light sensor (DLS) to estimate reflectance, without correcting for the sensor orientation, weather condition, or sun angle.

  • SKY_SENSOR_WITH_ANGLE_POSE_FROM_CAMERA - In addition to CAMERA corrections, uses the irradiance measured by a downwelling light sensor (DLS), the orientation of the DLS, the sun angle, and the weather condition to derive reflectance. The orientation of the DLS is derived from the camera pose and the known relative orientation between the camera and the DLS. This requires a rig between the DLS and the camera.

  • SKY_SENSOR_WITH_ANGLE_POSE_FROM_IMU - In addition to CAMERA corrections, uses the irradiance measured by a downwelling light sensor (DLS), the orientation of the DLS, the sun angle, and the weather condition to derive reflectance. The orientation of the DLS is derived from an IMU inside the DLS.

  • SKY_SENSOR_WITH_PROVIDED_ILLUMINATION_MODEL - Uses output provided by the MicaSense DLS2 model.

Note

Reflectance targets (radiometric calibration panels) are being searched for, evaluated, and considered for sensors that do not have the correction type NONE.

Multi-sensor captures

Camera creation, calibration, densification, and the DSM need to run once. If the project has more than one sensor layout (for example RGB+Thermal or RGB+Multispectral), the orthomosaic generation and GeoTIFF exports must run once per sensor layout.

See examples/gen_thermal_ortho.py.

Workflow (dual sensor capture)

  1. make_input_cameras() recognizes FLIR or DJI radiometric JPEG.

  2. Optionally pass thermal_parameter.

  3. Calibrate, densify, and build the DSM as usual.

  4. Run get_sensor_layouts() to list sensor layouts. (groups of sensors that can go into a single orthomosaic processing step). Each item is a SensorLayout. Use band_types() to see what it is, f.i. Red, Green, Blue (RGB) vs. Thermal IR (Thermal) vs. Green, Red, RedEdge, NIR (Multispectral).

  5. For each SensorLayout, where the GeoTIFF export is required, run gen_orthomosaic() or gen_tiled_orthomosaic() (and fastalgo equivalents) with sensor_layout parameter, then write the raster.

The cameras.Settings.Radiometry is applicable to multispectral processing only (targets, weather), not to thermal processing.

Sensor layouts (SensorLayout)

A sensor layout is a sensor or a group of sensors requiring a separate orthomosaic processing run (to be managed by the processing python scripting). For example, in case of the RGB and Thermal IR, the RGB has one sensor layout, and the Thermal IR has another sensor layout. Each sensor layout run delivers one GeoTIFF at the end.

get_sensor_layouts() returns the sensor layouts. SensorLayout is the object you pass as sensor_layout so that ortho run uses only that sensor layout.

Sensor layout helper get_sensor_layouts() can return:

  • RGB only - one sensor layout. Standalone.

  • Thermal IR only - one sensor layout. Standalone.

  • RGB + Thermal IR - two sensor layouts.

  • RGB + multispectral - two sensor layouts.

A standalone set (one sensor layout) may omit sensor_layout. When there are two sensor layouts, pass one sensor layout per ortho run. One run = one sensor layout = one GeoTIFF (for example RGB and Thermal IR in two runs).

If you omit sensor_layout when there are several sensor layouts, all cameras are sent to that run and the sensor layout becomes ambiguous. The orthomosaic processing step will throw an error. There is no default sensor layout.

A small utility band_types() exists, to help evaluating the sensor layout contents. It returns a list of band types in the sensor layout, for example ['Red', 'Green', 'Blue'] or ['Thermal IR'].

Thermal IR processing

Radiometric Thermal IR images can be processed to a thermal orthomosaic (nadir / 2D only).

Supported formats

  • Radiometric JPEG (R-JPG) - JPEG with radiometric temperature information in the metadata (FLIR and DJI).

  • Non-radiometric JPEG (JPG) - colorized temperature representation or greyscale RGB without the original radiometric values. These are not a radiometric Thermal IR product; use the regular RGB workflow for processing.

Minimum acquisition requirements

  • Capture: strict nadir.

  • Overlap: at least 90% front and 90% side (use an automated grid mission planning software).

  • Resolution: at least 640x512 pixels. Lower resolutions are not supported.

Thermal parameters (thermal_parameter)

The ThermalParameter contains parameters that are used to transform raw Thermal IR data to temperature.

  • When calling make_input_cameras() without passing thermal_parameter, parameters are extracted from the radiometric JPEG and averaged. If reading from the radiometric JPEG fails in some images, this information is logged and the remaining images are used to derive the initial guess; the respective images will not be removed from the project.

  • If the thermal_parameter is passed to make_input_cameras(): those values will be used instead of reading from the radiometric JPEG.

Meaning of the four parameters (see more details in the API reference):

  • relative_humidity - Relative humidity at the time of acquisition, set as a number between 0 and 1 (0.5 means 50%).

  • air_temperature_K - Outside air temperature (OAT) at the time of acquisition, in kelvin [K].

  • apparent_reflected_temperature_K - Reflected apparent temperature, in kelvin [K] - an estimation of an apparent reflected temperature by measuring the apparent temperature of a reflective material with near-zero emissivity (for example aluminium foil). This is not the temperature of any of the surrounding objects, but an effective description of the interaction of all the different surrounding objects.

  • emissivity - Material property which states how much radiation received from an object is emitted from that object and how much is reflected, as a fraction between 0 and 1. Typical values for common materials are listed in publicly available material tables.

DJI radiometric JPEG also sets radiometry_data()._requires_dji_sdk. The DJI Thermal SDK is needed to extract the thermal parameters from the DJI radiometry JPEGs during camera creation, and to read the DJI radiometric JPEGs during the orthomosaic step.

3D Mesh

Usage

3D mesh model geometry, texture, and LOD are generated in a single step using gen_mesh().

The level-of-detail (LOD) 3D mesh is always generated and is the target output.

To visualize a 3D mesh in third-party software, it must be exported as a file in one of the following formats: Cesium, SLPK, or FBX, using write_mesh(), write_mesh(), and write_mesh(). These exports require an LOD mesh as input.

Guidelines and Best Practices

For configuring the scalable mesh generation algorithm, a number of configuration templates are provided as SCALABLE_MESH_TEMPLATES:

  • CATCH exclusively for PIX4Dcatch captures

  • OBLIQUE for all oblique captures

  • NADIR for nadir captures

The gen_mesh() method supports a range of constraints: mask maps, region of interest, and depth maps. Using constraints may improve geometry and help achieve better reconstruction quality for thin structures, such as power lines and transmission towers.

When masks were used in point cloud densification, it is recommended to provide the same masks to the silhouette_mask_map parameter.

More Processing

AutoGCP

Automatic GCP detection tools.

AutoGCPs consists of a set of tools for the automatic detection in images of control point targets with pixel level accuracy.

It supports three types of targets with black and white Haar-like features: square, diagonal and Aeropoint.

Despite its name, AutoGCPs imposes no restrictions on the use of targets. The functionality only concerns itself with the detection of targets in images and the obtainment of an accurate estimate of the position of their marker. The user can then use the information as ground control points (GCPs), check-points (CPs) or anything else.

For more information on control and check points, see Control Points.

Usage

The basic usage of the AutoGCPs functionality is straightforward: the user calls a function to obtain marks. Usage within the context of a processing pipeline may be more involved, since it requires specific processing to obtain the inputs to AutoGCPs, and typically the results are used in further processing, either to improve or measure camera calibration accuracy.

Guidelines and Best Practices

  • Although only nadir flights are supported, detection might work well (with no guarantees) with oblique captures without excessive image angles and very precise geolocation and orientation.

  • Detection is supported only on RGB images.

  • Images should not be excessively blurry.

  • Targets should be clean, unobstructed by objects above or nearby from the camera point of view, and should not be crossed by shadow lines.

  • If an obstruction cannot be avoided, the CP targets should be moved to another location so that a sufficient minimum number of them is always available for re-calibration or re-optimization.

  • The CP target must have a size in pixels within an optimal range in order to be detected reliably. This depends on the image resolution but, as a rule of thumb, should be roughly within the range of 20 to 100 pixels.

  • Configuring the correct horizontal and vertical uncertainty values can speed up target detection if georeferencing of images is accurate. The default values cover most cases but are in meters and must be converted if a different unit of measurement is being used (refer to Settings for more information). Using a lower uncertainty than the default values is generally not recommended.

  • Targets on the ground should be well placed at a sufficient distance from each other. The spacing must be larger than the camera georeferencing horizontal position uncertainty; otherwise, image marks may get an incorrect physical target-to-CP assignment.

  • Good image overlap ensures that AutoGCPs results are more reliable, with each target visible on multiple images. A minimum of three reliable and confirmed marks (by the algorithm) are needed to generate the AutoGCPs output for a CP.

Depth Processing

Depth processing uses depth maps from PIX4Dcatch to create depth point clouds. It can also merge depth point clouds with dense point clouds obtained through photogrammetry, thus producing point clouds of superior quality. Densification of sparse synthetic depth maps is another functionality that can be useful in specific 3D mesh use cases.

Usage

A depth point cloud is generated by calling gen_pcl() with input cameras containing LiDAR depth maps and a scene reconstructed by calibration as inputs. A region of interest defined with the help of the geometry utilities can limit the area of the point cloud if necessary.

Merging of depth and dense point clouds is performed by calling pcl_merge().

Sparse synthetic depth maps optionally obtained by densification() can be densified (completed) using densify(). The resulting depth maps can be used as a constraint in mesh geometry generation.

Sky and Water Segmentation

Usage

Sky and water segmentation identifies sky segments, water segments, or both on images (in this context, an image segment is an image region or area).

The output may be used to provide additional input to point cloud densification. Computing depth from sky or water segments in images is unreliable. Allowing densification to exclude such segments greatly reduces or eliminates noise in the point cloud originating from these segments.

Guidelines and Best Practices

  • For images where water is present and should be removed, the full segmentation mode must be used (that is, when the masking_type argument is WATER or SKY_WATER, the mode argument must be FULL).

  • For images where only sky is present and should be removed, the fast segmentation mode is recommended (that is, when the masking_type argument is SKY, the mode argument is recommended to be FAST).