The computer vision field has largely solved 2D annotation, with mature tooling, established workflows, and production-ready pipelines for bounding boxes, polygons, segmentation masks, and more. It has also made significant progress on 3D annotation for sensor-fused autonomous vehicle setups, where LiDAR provides metric depth and a controlled calibration procedure places the camera precisely in the world. What remains largely unsolved is the space in between: monocular cameras deployed in real environments, without depth sensors, and without any ability to control the scene, which is not a niche edge case, but the default condition for most infrastructure and industrial deployments.

A 3D object detection model learns to place objects in metric space, which means its training data must do the same.

Before writing a line of code, we evaluated six of the most widely used 3D annotation platforms available, CVAT, 3D-BAT, Label Studio, Segments.ai, Dataloop, and Diffgram, and every one of them failed, not because of a missing configuration option, but for reasons embedded in their architecture.

Recovering a camera’s position and orientation in 3D space from a single 2D image is a fundamentally ill-posed problem: a small object close to the camera and a large object far away produce identical projections onto the image plane, and without additional information there is no way to distinguish them from pixel data alone. The missing sensor information can, however, be overcome with geometric priors drawn from the environment itself, and this is where most structured environments turn out to be more information-rich than they might appear.

If a reference surface with a known geometric relationship to the scene can be identified, points on that surface are no longer free in 3D space, they are constrained to lie on a known plane, which is sufficient to make the camera’s pose relative to that plane recoverable. This approach requires the camera’s intrinsic parameters, focal length and principal point, to be known in advance, which is achievable once under controlled conditions using a standard checkerboard or similar target. What the project addresses is the harder part: extrinsic calibration, meaning the camera’s real-world position and orientation, derived entirely from scene geometry rather than a dedicated calibration rig present at capture time.

The technical foundation for this is classical projective geometry, a domain we apply across a wide range of computer vision consulting projects, spanning industrial inspection and beyond. The rest of this section is a bit technical and you can skip it without losing context. Three categories of geometric constraints are central to the proposed calibration system. Geometric constraints allow us to link the pixel coordinates from a camera C to the 3D world outside of that camera. With enough geometric constraints in place we can answer the following central question: where is the camera (rotation R and translation T) with respect to the 3D world?

The 3D point P_1, shown in the image on the left (a), is constrained to lie on the same camera ray defined by its 2D projection D_1 onto the image plane. This can be translated in C, D_1 and P_1 being linearly dependent, i.e. one can be obtained by a convex combination of the other two. More formally, we can choose R and T such that the following matrix has null determinant:

Null determinant, or rank 2 in this case, is verified if all 3×3 submatrices have null determinant: out of 4 linear equations arising, only two though will be non-redundant, in fact point P_1 is losing two degree of freedom, but can still slide on the C-D_1 ray. So for every 2D-3D point we get two constraints on (R, T).

We can reason about lines very similarly. The following 3 lines shown on the image on the left (b) are all coplanars:

• the camera ray defined by the 2D segment start D_1
• the camera ray defined by the 2D segment end D_2
• the 3D line L_1-L_2

Meaning that we can choose R and T so that point L_1 is a convex combination of C, D_1 and D_2, or that the following matrix has rank 3:

We get one useful linear constraint from L_1, in fact L_1 is only loosing one degree of freedom being able to “slide” on the C-D_1-D_2 plane. L_2 will of course produce a second constraint on (R, T).

A more esoteric, but very interesting, type of constraint comes from ellipses (or conics more in general, although we will focus on ellipses for the rest of this article). The key intuition is that an ellipse is still an ellipse after projection, although a different one. Given an image ellipse E_{img}, all camera rays through image points on that ellipse form a 3D surface X^TQX=0, where Q is a rank-3 quadric passing through the camera origin C.

That surface is an elliptic cone, with apex in C and every generator line is a viewing ray of the camera. It is easy to picture that the 3D ellipse E_w is the intersection of this cone with an unknown plane (the supporting plane of the ellipse itself). Given the two ellipses both lay on planes, they are linked through an homography H that encodes the (R, T) of the camera with respect to world coordinates:

up to an ambiguous rotation around the Z_w axis. To recover this ambiguity, and have a system that is well constrained, we need to add at least one 2D-3D point correspondence.

So any combination of N points, M lines and K ellipses, yielding 2N+2M+5K >= 6 constraints will suffice the requirements of recovering the 6 unknown of pose and translations. The most renowned configuration is using N=3 2D-3D point correspondences, a problem also known as P3P in literature. Interestingly, different types of correspondences can also be mixed across dimensions: a 3D line or a 3D ellipse can be matched to a 2D point, a 2D line to a 3D point and so on… this flexibility is the true power of this mathematical framework!

Basically, combining multiple correspondence types, each carrying different geometric information about the scene, is what makes calibration tractable from uncontrolled real-world images where no single correspondence type is sufficient on its own.

In practice, calibration in SIBORIA is performed by identifying correspondences between elements visible in the camera image and their known positions in a georeferenced map via Google Maps, with GPS coordinates, so no calibration targets need to be placed in the scene, the annotator works entirely with what is already there.

Three correspondence types are supported, each matched to scene elements that occur naturally in urban environments and many others.

1. Point correspondences are drawn from precisely locatable ground-level features such as corners of pedestrian crossings or road marking intersections. They are the most geometrically informative correspondence type and the preferred starting point for any calibration.
2. Line correspondences are drawn from linear structures such as poles and stop lines. A line in 3D projects to a line in the image, and knowing the 3D direction of that line contributes a rotational constraint independent of what point correspondences provide.
3. Ellipse correspondences are drawn from circular ground-level geometry, with roundabouts as the clearest urban example. They are particularly useful in scenes where point and line features are sparse or ambiguous.

More correspondences improve the robustness of the estimate, since the system applies a robust statistical fitting method that tolerates noisy or imprecise individual matches and produces a stable extrinsic result even when some correspondences are approximate or contain errors.

Calibration quality is directly verifiable within the tool without leaving the interface: SIBORIA computes per-correspondence metric errors alongside a mean error across all correspondences, and generates a bird’s-eye homographic projection of the image that can be compared against the map view as a geometric sanity check. If the calibration is correct, the projected scene aligns with the map; if it does not, the annotator can identify and correct the problematic correspondences before annotation begins.

Once calibration is established, it becomes the geometric foundation for the entire 3D annotation pipeline, with every subsequent operation, pre-annotation import, manual refinement, and quality verification, grounded in the camera model that calibration produced.

Pre-annotations are generated by the deep learning model developed by Università di Udine, which predicts 3D bounding boxes from camera images and imports them into SIBORIA via a shared JSON format agreed on by both teams. The model is effective at identifying objects and establishing their approximate 2D extent but introduces depth errors, as is expected for a monocular task, so the annotator’s job is to correct those errors in a metrically grounded workspace rather than draw annotations from scratch.

The annotation interface presents the camera image and a live bird’s-eye homographic projection side by side, with bounding boxes overlaid on both views simultaneously and changes in one view reflected immediately in the other. A snap-to-ground mechanism projects bounding boxes onto the calibrated reference plane automatically, ensuring every 3D annotation is metrically grounded in world coordinates from the first interaction, after which annotators refine position and orientation using on-screen manipulation controls.

SIBORIA was built for roadside cameras in urban environments, but the calibration logic places no requirement on that specific context. Wherever a monocular camera operates in an environment that contains a surface or structure with predictable, surveyable geometry, the same approach applies.

This holds across a wider range of domains than initially apparent: