Get started with the Reality Analysis API

Introduction

This quick start tutorial is going to help you work with Reality Analysis.

In this tutorial, you will submit a simple job to detect objects in images.

It is assumed that you already have:

Knowledge of the Reality Data API.
Knowledge of the Reality Analysis concepts. Check Reality Analysis and read the Overview, Context Scenes, Context Detectors and Job Types sections.

Prerequisites

A small set of photos to analyse.
A photo object detector. You may download the Coco 2D object detector of the library page
An iTwin and its ID. For instance you can create a project of ID “ProjectID”. Check the create and query projects guide.

If you want to test the REST API calls directly, you can use Postman or any other solution capable of sending HTTP requests. If you do it this way, you will require an authorization token for the requests to work. This is covered in the “Get a token” section below.

Postman

1. Register an Application

You will need to register an application to use the iTwin Platform APIs. You can use the Register button to automatically create your first single page application (SPA). This will allow you to configure Authorization Code Flow for your SPA application and get the correct access token.

Once generated, you will be shown a few lines of code under the button.

IMJS_AUTH_CLIENT_CLIENT_ID - this is the unique identifier for your application. Displayed on application details page as Client ID.
IMJS_AUTH_CLIENT_REDIRECT_URI - specifies where users are redirected after they have chosen whether or not to authenticate your app. Displayed on application details page as one of Redirect URIs.
IMJS_AUTH_CLIENT_LOGOUT_URI - specifies where users can be returned to after logging out. Displayed on application details page as one of Post logout redirect URIs.
IMJS_AUTH_CLIENT_SCOPES - list of accesses granted to the application. Displayed on application details page as Scopes.

Or optionally: Register and configure your application manually following instructions in Register and modify an Application tutorial. Make sure that your application is associated with Reality Modeling API and has realitydataanalysis:read, realitydataanalysis:modify, realitydata:modify, realitydata:read scopes enabled.

Single page application

Requires you to sign in. Will automatically generate a Single page application (SPA) that is required to complete this tutorial. You will be able to manage your SPA from your My apps page.

2. Get a token

To make a request to the API, you need a user token. There are several ways to get it. You can consult the Authorization API documentation or you can use a sample powershell script to use in your app.

In the following, JWT_TOKEN will stand for this token.

3. Upload data

Upload your photos to RDS as CCImageCollection attached to your iTwin project using the Reality Data API.
Upload your detector to RDS as ContextDetector attached to your iTwin project using the Reality Data API.

In the following, ProjectID stands for the iTwin project id, ImageCollectionID for the CCImageCollection entry id and DetectorID for the ContextDetector entry id.

Note that your iTwin project will usually store some detectors and all your raw reality data (photos, point clouds, etc.). They are not intended to be uploaded for each Reality Analysis jobs. In many cases, they will have been uploaded previously and probably by someone else.

4. Prepare job input

You will now chose on which reality data the job will be applied, creating the ContextScene pointing to them. It will be very simple in our case, a few photos in a CCImageCollection but remember it could be more complex: pointing to several RDS entries, adding metadata like orientation, etc.

Create ContextScene file pointing to the photos your want to analyze. It should look like the file besides.
Upload it to RDS as ContextScene entry. Remember to name the file ContextScene.xml as should be any ContextScene uploaded on RDS. In the following, PhotosID stands for its id.

XML

<?xml version="1.0" encoding="utf-8"?>
<ContextScene version="3.0">
    <PhotoCollection>    
        <Photos>
           <Photo id="0">
                <ImagePath>0:IMAGE_1059.JPG</ImagePath>
            </Photo>
            <Photo id="1">
                <ImagePath>0:IMAGE_1060.JPG</ImagePath>
            </Photo>
            <Photo id="2">
                <ImagePath>0:IMAGE_1061.JPG</ImagePath>
            </Photo>
        </Photos>
    </PhotoCollection>    
    <References>
        <Reference id="0">
            <Path>rds:ImageCollectionID</Path>
        </Reference>
    </References>
</ContextScene>

5. Create job

Your next step will be to create the job.

Request

The job is created sending an HTTP POST message to https://api.bentley.com/realitydataanalysis/jobs endpoint with the payload describing the job. Be sure to replace JWT_TOKEN by its value.

HTTP

POST https://api.bentley.com/realitydataanalysis/jobs HTTP/1.1
Authorization: Bearer JWT_TOKEN
Content-Type: application/json

Request Body

The request body define the job settings. Be sure to replace ProjectID, PhotosID and DetectorID by their values.

JSON

{
  "type": "objects2D",
  "name": "O2D job sample",
  "iTwinId": "ProjectID",
  "settings": {
    "inputs": [
      {
        "name": "photos",
        "realityDataId": "PhotosID"
      },
      {
        "name": "photoObjectDetector",
        "realityDataId": "DetectorID"
      }
    ],
    "outputs": [
      "objects2D"
    ]
  }
}

Response body

The API will respond with a 201 Created status code if the call is successful. The most important entry of this response is the “id” which you will use to refer to the job in following steps.

JSON

{
  "settings": {
    "outputs": [
      {
        "name": "objects2D"
      }
    ],
    "inputs": [
      {
        "name": "photos",
        "realityDataId": "PhotosID"
      },
      {
        "name": "photoObjectDetector",
        "realityDataId": "DetectorID"
      }
    ]
  },
  "state": "unsubmitted",
  "dataCenter": "EastUs",
  "createdDateTime": "2022-03-08T10: 55: 24Z",
  "lastModifiedDateTime": "2022-03-08T10: 55: 24Z",
  "id": "JobID",
  "email": "name@bentley.com",
  "costEstimation": {
    "estimatedCost": -1.0
  },
  "type": "objects2D",
  "name": "O2D job sample",
  "iTwinId": "ProjectID"
}

6. Estimate job cost

You can optionally provide relevant quantities about the input to get an estimation of the job cost before deciding to submit it.

Request

Send an HTTP PATCH message to https://api.bentley.com/realitydataanalysis/jobs/JobID endpoint with the payload describing the quantities. Be sure to replace JWT_TOKEN and JobID by their values.

HTTP

PATCH https://api.bentley.com/realitydataanalysis/jobs/JobID HTTP/1.1
Authorization: Bearer JWT_TOKEN
Content-Type: application/json

Request Body

For an 2D object detection job type, these quantities are the number of photos and the total number of gigapixels that will be analyzed.

JSON

{
  "costEstimation": {
    "numberOfPhotos": 3,
    "gigaPixels": 12
  }
}

Response body

The API will respond with a 201 Created status code if the call is successful. The estimatedCost is now set.

JSON

{
  "settings": {
    "outputs": [
      {
        "name": "objects2D"
      }
    ],
    "inputs": [
      {
        "name": "photos",
        "realityDataId": "PhotosID"
      },
      {
        "name": "photoObjectDetector",
        "realityDataId": "DetectorID"
      }
    ]
  },
  "state": "unsubmitted",
  "dataCenter": "EastUs",
  "createdDateTime": "2022-03-08T10: 55: 24Z",
  "lastModifiedDateTime": "2022-03-08T13: 55: 24Z",
  "id": "JobID",
  "email": "name@bentley.com",
  "costEstimation": {
    "estimatedCost": 2.4
  },
  "type": "objects2D",
  "name": "O2D job sample",
  "iTwinId": "ProjectID"
}

7. Submit job

To submit your job, simply change its state to active.

Request

Send an HTTP PATCH message to https://api.bentley.com/realitydataanalysis/jobs/JobID endpoint with the payload to change its state to active. Be sure to replace JWT_TOKEN and JobID by their values.

HTTP

PATCH https://api.bentley.com/realitydataanalysis/jobs/JobID HTTP/1.1
Authorization: Bearer JWT_TOKEN
Content-Type: application/json

Request Body

The only entry to patch is the state of the job.

JSON

{
  "state": "active"
}

Response body

The API will respond with a 200 OK status code if the call is successful. The response indicates that the state has changed to active and adds an executionInformation section, filled so far with a submission time.

JSON

{
  "settings": {
    "outputs": [
      {
        "name": "objects2D"
      }
    ],
    "inputs": [
      {
        "name": "photos",
        "realityDataId": "PhotosID"
      },
      {
        "name": "photoObjectDetector",
        "realityDataId": "DetectorID"
      }
    ]
  },
  "state": "active",
  "executionInformation": {
    "submissionDateTime": "2022-03-08T13:23:52Z"
  },
  "dataCenter": "EastUs",
  "createdDateTime": "2022-03-08T10: 55: 24Z",
  "lastModifiedDateTime": "2022-03-08T10: 55: 24Z",
  "id": "JobID",
  "email": "name@bentley.com",
  "costEstimation": {
    "estimatedCost": 2.4
  },
  "type": "objects2D",
  "name": "O2D job sample",
  "iTwinId": "ProjectID"
}

8. Monitor job

You will probably want to monitor your job to check its state and advancement.

Request

To monitor the job, send an HTTP GET message to https://api.bentley.com/realitydataanalysis/jobs/JobID/progress endpoint. Be sure to replace JWT_TOKEN and JobID by their values.

HTTP

PATCH https://api.bentley.com/realitydataanalysis/jobs/JobID/progress HTTP/1.1
Authorization: Bearer JWT_TOKEN
Content-Type: application/json

Response body

The response give you the job advancement, state and a step name.

JSON

{
  "progress": {
    "percentage": 25,
    "state": "active",
    "step": "Run_Production"
  }
}

9. Get results

Request

At any time, you can get all job information sending an HTTP GET message to https://api.bentley.com/realitydataanalysis/jobs/JobID endpoint.

HTTP

PATCH https://api.bentley.com/realitydataanalysis/jobs/JobID HTTP/1.1
Authorization: Bearer JWT_TOKEN
Content-Type: application/json

Response body

Once the job is over, your can get its final state and check success. The executionInformation section is also filled with more useful data. Your will also get an id in RDS for every output entry. This will allow you to download the results using the Reality Data API.

JSON

{
  "settings": {
    "outputs": [
      {
        "name": "objects2D",
        "realityDataId": "SomeIDProvidedByTheService"
      }
    ],
    "inputs": [
      {
        "name": "photos",
        "realityDataId": "PhotosID"
      },
      {
        "name": "photoObjectDetector",
        "realityDataId": "DetectorID"
      }
    ]
  },
  "state": "success",
  "executionInformation": {
    "exitCode": 0,
    "estimatedUnits": 3.0,
    "startedDateTime": "2022-03-08T14:20:44Z",
    "endedDateTime": "2022-03-08T14:27:31Z",
    "submissionDateTime": "2022-03-08T13:23:52Z"
  },
  "dataCenter": "EastUs",
  "createdDateTime": "2022-03-08T10: 55: 24Z",
  "lastModifiedDateTime": "2022-03-08T10: 55: 24Z",
  "id": "JobID",
  "email": "name@bentley.com",
  "costEstimation": {
    "estimatedCost": 2.4
  },
  "type": "objects2D",
  "name": "O2D job sample",
  "iTwinId": "ProjectID"
}

Output file

You should obtain a ContextScene file with 2D objects.

XML

<?xml version="1.0" encoding="utf-8"?>
<ContextScene version="3.0">
    <PhotoCollection>
        <Photos>
            <Photo id="0">
                <ImagePath>0:IMG_1059.JPG</ImagePath>
            </Photo>
            <Photo id="1">
                <ImagePath>0:IMG_1060.JPG</ImagePath>
            </Photo>
            <Photo id="2">
                <ImagePath>0:IMG_1061.JPG</ImagePath>
            </Photo>
        </Photos>
    </PhotoCollection>
    <Annotations>
        <Labels>
            <Label id="3">
                <Name>car</Name>
            </Label>
            <Label id="4">
                <Name>motorcycle</Name>
            </Label>
        </Labels>
        <Objects2D>
            <ObjectsInPhoto>
                <PhotoId>0</PhotoId>
                <Objects>
                    <Object2D id="0">
                        <LabelInfo>
                            <Confidence>0.998535</Confidence>
                            <LabelId>3</LabelId>
                        </LabelInfo>
                        <Box2D>
                            <xmin>0.0319100581109524</xmin>
                            <ymin>0.537032723426819</ymin>
                            <xmax>0.374318599700928</xmax>
                            <ymax>0.66499537229538</ymax>
                        </Box2D>
                    </Object2D>
                    ...

10. Other requests

In this tutorial, you have seen almost all the API requests except two:

deleting a job: this is only possible if the job has not been submitted yet. The job will be deleted from the service database.
canceling a job: it can be done while active. The job will stay in the service database but will not be processed anymore. Please refer to the API Reference.

As mentioned in the documentation, Reality Analysis API is batch oriented to allow concurrent analysis on large datasets. This is why the input and output ContextScenes go through an upload/download to/from RDS. To simplify use cases where these files are light enough to go be transferred via the API requests, future versions will allow direct json transfers as a possible choice.

Continue learning

Congratulations for completing the Get started with the Reality Analysis API tutorial! You should now be able to detect 2D objects in a set of photos. Stay tuned for more tutorials. Meanwhile, you might want to try photo segmentation jobs.

More resources that you may like

Reality Data API

Reality Data API is necessary for uploading inputs for Reality Modeling, and downloading outputs.

Detector library

LIbrary of available Context Detectors.