The analysis performed by a Reality Analysis job usually requires one or more Machine Learning models (e.g. a deep learning neural network). We call them `Detectors`.

Detectors are specific to Reality Analysis. They are classified in different categories, each Reality Analysis job type requiring a given category. Here are the categories used by current Reality Analysis jobs:

• photo object detector: ML model detecting objects as boxes in a photo.
• photo segmentation detector: ML model classifying pixels in a photo (semantic segmentation).
• orthophoto segmentation detector: ML model classifying pixels in an orthophoto (semantic segmentation). The difference with a photo segmentation detector is that pixel resolution is taken into account to improve detection.
• point cloud segmentation detector: ML model classifying points in a point cloud (semantic segmentation)

Detectors are the basic blocks of Reality Analysis: limiting to these categories does not mean that analysis is limited to these basic tasks. For instance, 3D object detection is available but uses either a photo object detector or a point cloud detector underneath. Mesh analysis is also available, etc.

Detectors from Reality Management

Bentley provided detectors are available for download on the iTwin Capture detectors download page. To use one of them with Reality Analysis, just upload it to Reality Management Service in your Enterprise projects.

Reality Analysis Detectors API

A new detector library is also available in the Reality Analysis API where you can upload your own detectors or use Bentley provided public detectors. Please try out the get detectors API endpoint to get available detectors. The public detectors are denoted with @bentley/ prefix.

Follow the below steps to add a new detector.

1. Create a detector entry using create detector operation.
2. Create a detector version using create detector version operation.
3. The operation will return a uploadUrl and a completeUrl. Take a copy of these URLs.
4. Upload the detector zip file to Reality Analysis Storage using the uploadUrl.
5. Follow the complete detector process with the completeUrl to make the detector ready to use.

sequenceDiagram
    participant user as User
    participant ras as Reality Analysis
    participant storage as Reality Analysis Storage

    user->>+ras: Create detector
    ras->>-user: Return created detector
    user->>+ras: Create detector version
    ras->>-user: Return uploadUrl and completeUrl

    user->>storage: Upload detector ZIP file to cloud storage using uploadURL
    user->>ras: Make detector ready to use using completeURL

The detectors can be used for a Reality Analysis job by providing the detector name and version (optional) in the job's input id as detectorName/detectorVersion.