Image2GaussianSplats: From Images to 3D Gaussian Splatting

This tutorial guides you through the process of using the TUM2Twin dataset’s UAS photographs for 3D Gaussian Splatting (3DGS) and Neural Radiance Fields (NeRF) reconstruction. It covers the essential steps from data preparation to generating Gaussian splats and dense point clouds for high-quality 3D scene reconstruction.

This tutorial assumes you have basic knowledge of 3D reconstruction, Gaussian splatting, and NeRF.

1. Download the Dataset

Before starting 3DGS or NeRF reconstruction, we need to download and prepare the dataset. The dataset is available here.
We need the following two files from this dataset:

images.zip – The set of aerial images captured by the Uncrewed Aerial System (UAS).
opf.zip – The Open Photogrammetry Format (OPF) files, which include camera calibration and sparse reconstruction results.

The OPF means the Open Photogrammetry Format (OPF) by Pix4D. It contains essential metadata, including camera parameters, image geolocation, and sparse 3D reconstructions.

Alternatively, you can use the command line:

# Download images.zip
wget https://zenodo.org/records/14548134/files/images.zip
 
# Download opf.zip
wget https://zenodo.org/records/14548134/files/opf.zip

2. Data Format Conversion

Since both 3DGS** and NeRF require sparse reconstruction results as input, and each method requires different data formats, we need to convert the OPF files into the required formats.

To achieve this, we use the pyopf Python library, which provides tools for processing OPF files.

2.1 Installing `pyopf`

First, install the necessary dependencies using pip:

pip install pyopf
pip install pyopf[tools]

This will install the core pyopf library along with additional tools required for OPF data conversion.

More details on the OPF Tools can be found in the pyopf GitHub repository under the “OPF Tools” section.

2.2 Converting OPF to COLMAP Format

After the library installation, we need to convert the OPF dataset into the COLMAP format. This step extracts essential information, including camera intrinsics, extrinsic parameters, and a sparse 3D point cloud.

Required Output Files

After conversion, three important files will be generated:

cameras.txt – Contains camera intrinsic parameters.
images.txt – Stores camera poses (extrinsic parameters).
points3D.txt – Represents the sparse 3D point cloud.

Running the Conversion

We use the opf2colmap tool from pyopf to perform the conversion. Run the following command:

opf2colmap path/to/opf/project.opf
# Ensure that the OPF dataset is properly extracted and that the pyopf library is correctly installed before running this command.

3. Preprocessing Data with COLMAP

COLMAP is a powerful Structure-from-Motion (SfM) and Multi-View Stereo (MVS) pipeline used for 3D reconstruction from images. It is also a common tool for preparing custom datasets for reconstruction with 3DGS and NeRF. In the previous steps, we converted the dataset into COLMAP-compatible format. However, to successfully run 3DGS, we need to further process the data using COLMAP.

3.1 Install COLMAP

COLMAP supports Windows, Linux, and macOS, and installation methods vary depending on the operating system.

For detailed installation instructions, refer to the COLMAP GitHub repository. The repository provides:

Precompiled binaries for Windows and macOS.
Build instructions for compiling COLMAP from source on Linux.
Dependency requirements for different systems.

Verifying the Installation

After installing COLMAP, you can check if it was installed correctly by running:

colmap -h

3.2 Using COLMAP for Data Preprocessing

To prepare the dataset for 3DGS, we need to further process the COLMAP outputs. This includes:

Converting TXT files to BIN format, as COLMAP stores models in binary format by default.
Undistorting images, since 3DGS has limitations on camera models and requires undistorted images as input.

Run the following command to convert TXT files to BIN:

colmap model_converter \
    --input_path /path/to/txt/files \
    --output_path /path/to/sparse/0 \
    --output_type BIN

Run the following command to perform image undistortion:

colmap image_undistorter \
    --image_path /path/to/downloaded/images \
    --input_path /path/to/sparse/0 \
    --output_path /path/to/undistorted/images

This command performs the following tasks:

Removes lens distortions from the original images.
Exports new camera parameters that match the undistorted images.
Prepares images that are compatible with 3DGS.

After this step, a new folder /path/to/undistorted/images/ will be created, containing the processed images and updated camera parameters.

4. Performing 3D Reconstruction with 3DGS

After completing the preprocessing steps, we now have the required input data for 3DGS. The dataset has been converted and structured in a format that 3DGS expects.

To ensure a successful reconstruction, your dataset should be organized as follows:

/data/
├── images/          # Undistorted images
│   ├── 00001.jpg
│   ├── 00002.jpg
│   ├── ...
│
├── sparse/0/        # Undistorted sparse reconstruction in BIN format
│   ├── cameras.bin
│   ├── images.bin
│   ├── points3D.bin

These two components form the essential input for running 3DGS.

The original implementation of 3D Gaussian Splatting (3DGS) can be found at:
GraphDeco INRIA 3DGS Repository.

However, since our dataset consists of large-scale scene, training with this version may lead to CUDA out of memory (OOM) errors, causing failures during reconstruction.

To mitigate this issue, we recommend using the optimized version:
3DGS-Lightning.

By following the Installation and Training steps provided in the repository, you can successfully obtain the 3DGS reconstruction results for our scene.

5. Performing 3D Reconstruction with NeRF

Nerfstudio provides a simple API that allows for a streamlined end-to-end process of creating, training, and testing NeRFs.

To take advantage of its simplicity and efficiency, we will use Nerfstudio for training our NeRF model.

To install Nerfstudio, follow the official installation guide:
Nerfstudio Installation

Based on the results obtained in Section 3, we now need to convert our dataset into a format that Nerfstudio can process.

Run the following command to generate the required input format:

ns-process-data images \
    --data /path/to/undistorted/images \
    --output-dir /path/to/save/input/nerfstudio \
    --colmap-model-path /path/to/undistorted/sparse/0 \
    --skip-colmap

After preparing the dataset, we can now train a NeRF model using Nerfstudio.

Follow the official training guide:
Train Your First NeRF

For example, to start training with the Nerfacto model, run the following command:

ns-train nerfacto --data /path/to/save/input/nerfstudio

CityGML to OBJ Facade Texturing Using Sketchup