Get started with the Reality Analysis API
Important notice
The following tutorial is a step by step detailed description of the low level calls needed for an analysis. It involves using other services, understanding authorizations and creating different json files. It might not be the best way to discover Reality Analysis. We encourage using our Python or TypeScript iTwin Capture SDKs and the included examples if you plan to develop a Python or TypeScript application or for a first discovery even if you plan to use a different language.
Introduction
This quick start tutorial is going to help you work with Reality Analysis.
In this tutorial, you will submit a simple job to detect objects in images.
It is assumed that you already have:
- Knowledge of the Reality Management API.
- Knowledge of the Reality Analysis concepts. Check Reality Analysis and read the Overview, Context Scenes, Context Detectors and Job Types sections.
Prerequisites
- A small set of photos to analyse.
- A photo object detector. You may download the Coco 2D object detector of the library page
- An iTwin and its ID. For instance you can create a project of ID “ProjectID”. Check the create and query projects guide.
1. Register an Application
You will need to register an application to use the iTwin Platform APIs. You can use the Register button to automatically create your first single page application (SPA). This will allow you to configure Authorization Code Flow for your SPA application and get the correct access token.
Once generated, you will be shown a few lines of code under the button.
- IMJS_AUTH_CLIENT_CLIENT_ID - this is the unique identifier for your application. Displayed on application details page as Client ID.
- IMJS_AUTH_CLIENT_REDIRECT_URI - specifies where users are redirected after they have chosen whether or not to authenticate your app. Displayed on application details page as one of Redirect URIs.
- IMJS_AUTH_CLIENT_LOGOUT_URI - specifies where users can be returned to after logging out. Displayed on application details page as one of Post logout redirect URIs.
- IMJS_AUTH_CLIENT_SCOPES - list of accesses granted to the application. Displayed on application details page as Scopes.
Or optionally: Register and configure your application manually following instructions in Register and modify an Application tutorial. Make sure that your application is associated with Reality Modeling API and has
realitydataanalysis:read, realitydataanalysis:modify, realitydata:modify, realitydata:readscopes enabled.
Single page application
Requires you to sign in. Will automatically generate a Single page application (SPA) that is required to complete this tutorial. You will be able to manage your SPA from your My apps page.
2. Get a token
To make a request to the API, you need a user token. There are several ways to get it. You can consult the Authorization API documentation or you can use a sample powershell script to use in your app.
In the following, JWT_TOKEN will stand for this token.
3. Upload data
- Upload your photos to Reality Management API as CCImageCollection attached to your iTwin project using the Reality Management API.
- Upload your detector to Reality Management API as ContextDetector attached to your iTwin project using the Reality Management API.
In the following, ProjectID stands for the iTwin project id, ImageCollectionID for the CCImageCollection entry id and DetectorID for the ContextDetector entry id.
Note that your iTwin project will usually store some detectors and all your raw reality data (photos, point clouds, etc.). They are not intended to be uploaded for each Reality Analysis jobs. In many cases, they will have been uploaded previously and probably by someone else.
4. Prepare job input
You will now chose on which reality data the job will be applied, creating the ContextScene pointing to them. It will be very simple in our case, a few photos in a CCImageCollection but remember it could be more complex: pointing to several Reality Management API entries, adding metadata like orientation, etc.
- Create ContextScene file pointing to the photos your want to analyze. It should look like the file besides.
- Upload it to Reality Management API as ContextScene entry. Remember to name the file
ContextScene.jsonas should be any ContextScene uploaded on Reality Management API. In the following, PhotosID stands for its id.
{ "version" : "5.0", "PhotoCollection" : { "Photos" : { "0" : { "ImagePath" : "0:IMAGE_1059.JPG", }, "1" : { "ImagePath" : "0:IMAGE_1060.JPG", }, "2" : { "ImagePath" : "0:IMAGE_1061.JPG", } } }, "References" : { "0" : { "Path" : "rds:ImageCollectionID" } } }
5. Create job
Your next step will be to create the job.
The job is created sending an HTTP POST message to https://api.bentley.com/realitydataanalysis/jobs endpoint with the payload describing the job. Be sure to replace JWT_TOKEN by its value.
POST https://api.bentley.com/realitydataanalysis/jobs HTTP/1.1 Authorization: Bearer JWT_TOKEN Content-Type: application/json
The request body define the job settings. Be sure to replace ProjectID, PhotosID and DetectorID by their values.
{ "type": "objects2D", "name": "O2D job sample", "iTwinId": "ProjectID", "settings": { "inputs": [ { "name": "photos", "realityDataId": "PhotosID" }, { "name": "photoObjectDetector", "realityDataId": "DetectorID" } ], "outputs": [ "objects2D" ] } }
The API will respond with a 201 Created status code if the call is successful.
The most important entry of this response is the “id” which you will use to refer to the job in following steps.
{ "settings": { "outputs": [ { "name": "objects2D" } ], "inputs": [ { "name": "photos", "realityDataId": "PhotosID" }, { "name": "photoObjectDetector", "realityDataId": "DetectorID" } ] }, "state": "unsubmitted", "dataCenter": "EastUs", "createdDateTime": "2022-03-08T10: 55: 24Z", "lastModifiedDateTime": "2022-03-08T10: 55: 24Z", "id": "JobID", "email": "name@bentley.com", "costEstimation": { "estimatedCost": -1.0 }, "type": "objects2D", "name": "O2D job sample", "iTwinId": "ProjectID" }
6. Estimate job cost
You can optionally provide relevant quantities about the input to get an estimation of the job cost before deciding to submit it.
Send an HTTP PATCH message to https://api.bentley.com/realitydataanalysis/jobs/JobID endpoint with the payload describing the quantities. Be sure to replace JWT_TOKEN and JobID by their values.
PATCH https://api.bentley.com/realitydataanalysis/jobs/JobID HTTP/1.1 Authorization: Bearer JWT_TOKEN Content-Type: application/json
For an 2D object detection job type, these quantities are the number of photos and the total number of gigapixels that will be analyzed.
{ "costEstimation": { "numberOfPhotos": 3, "gigaPixels": 12 } }
The API will respond with a 201 Created status code if the call is successful.
The estimatedCost is now set.
{ "settings": { "outputs": [ { "name": "objects2D" } ], "inputs": [ { "name": "photos", "realityDataId": "PhotosID" }, { "name": "photoObjectDetector", "realityDataId": "DetectorID" } ] }, "state": "unsubmitted", "dataCenter": "EastUs", "createdDateTime": "2022-03-08T10: 55: 24Z", "lastModifiedDateTime": "2022-03-08T13: 55: 24Z", "id": "JobID", "email": "name@bentley.com", "costEstimation": { "estimatedCost": 2.4 }, "type": "objects2D", "name": "O2D job sample", "iTwinId": "ProjectID" }
7. Submit job
To submit your job, simply change its state to active.
Send an HTTP PATCH message to https://api.bentley.com/realitydataanalysis/jobs/JobID endpoint with the payload to change its state to active. Be sure to replace JWT_TOKEN and JobID by their values.
PATCH https://api.bentley.com/realitydataanalysis/jobs/JobID HTTP/1.1 Authorization: Bearer JWT_TOKEN Content-Type: application/json
The only entry to patch is the state of the job.
{ "state": "active" }
The API will respond with a 200 OK status code if the call is successful.
The response indicates that the state has changed to active and adds an executionInformation section, filled so far with a submission time.
{ "settings": { "outputs": [ { "name": "objects2D" } ], "inputs": [ { "name": "photos", "realityDataId": "PhotosID" }, { "name": "photoObjectDetector", "realityDataId": "DetectorID" } ] }, "state": "active", "executionInformation": { "submissionDateTime": "2022-03-08T13:23:52Z" }, "dataCenter": "EastUs", "createdDateTime": "2022-03-08T10: 55: 24Z", "lastModifiedDateTime": "2022-03-08T10: 55: 24Z", "id": "JobID", "email": "name@bentley.com", "costEstimation": { "estimatedCost": 2.4 }, "type": "objects2D", "name": "O2D job sample", "iTwinId": "ProjectID" }
8. Monitor job
You will probably want to monitor your job to check its state and advancement.
To monitor the job, send an HTTP GET message to https://api.bentley.com/realitydataanalysis/jobs/JobID/progress endpoint. Be sure to replace JWT_TOKEN and JobID by their values.
PATCH https://api.bentley.com/realitydataanalysis/jobs/JobID/progress HTTP/1.1 Authorization: Bearer JWT_TOKEN Content-Type: application/json
The response give you the job advancement, state and a step name.
{ "progress": { "percentage": 25, "state": "active", "step": "Run_Production" } }
At any time, you can get all job information sending an HTTP GET message to https://api.bentley.com/realitydataanalysis/jobs/JobID endpoint.
PATCH https://api.bentley.com/realitydataanalysis/jobs/JobID HTTP/1.1 Authorization: Bearer JWT_TOKEN Content-Type: application/json
Once the job is over, your can get its final state and check success. The executionInformation section is also filled with more useful data. Your will also get an id in Reality Management API for every output entry. This will allow you to download the results using the Reality Management API.
{ "settings": { "outputs": [ { "name": "objects2D", "realityDataId": "SomeIDProvidedByTheService" } ], "inputs": [ { "name": "photos", "realityDataId": "PhotosID" }, { "name": "photoObjectDetector", "realityDataId": "DetectorID" } ] }, "state": "success", "executionInformation": { "exitCode": 0, "estimatedUnits": 3.0, "startedDateTime": "2022-03-08T14:20:44Z", "endedDateTime": "2022-03-08T14:27:31Z", "submissionDateTime": "2022-03-08T13:23:52Z" }, "dataCenter": "EastUs", "createdDateTime": "2022-03-08T10: 55: 24Z", "lastModifiedDateTime": "2022-03-08T10: 55: 24Z", "id": "JobID", "email": "name@bentley.com", "costEstimation": { "estimatedCost": 2.4 }, "type": "objects2D", "name": "O2D job sample", "iTwinId": "ProjectID" }
You should obtain a ContextScene file with 2D objects.
{ "version" : "5.0", "PhotoCollection" : { "Photos" : { "0" : { "ImagePath" : "0:IMAGE_1059.JPG" }, "1" : { "ImagePath" : "0:IMAGE_1060.JPG" }, "2" : { "ImagePath" : "0:IMAGE_1061.JPG" } } }, "Annotations" : { "Labels" : { "3" : { "Name" : "car" }, "4" : { "Name" : "motorcycle" } }, "Objects2D" : { "0" : { "e256276a-d478-476d-a654-195dbc23d1d9" : { "LabelInfo" : { "Confidence" : 0.998534977436066, "LabelId" : 3 }, "Box2D" : { "xmin" : 0.0319100245833397, "ymin" : 0.537032723426819, "xmax" : 0.374318659305573, "ymax" : 0.66499537229538 } }, "03f2b2e5-5df5-49a0-8515-62d6df7739bd" : { "LabelInfo" : { "Confidence" : 0.996562480926514, "LabelId" : 3 }, "Box2D" : { "xmin" : 0.877566039562225, "ymin" : 0.4940065741539, "xmax" : 1, "ymax" : 0.62068098783493 } } }, "1" : { "b61c36dd-98d2-4a2a-902e-9d12d9472f15" : { "LabelInfo" : { "Confidence" : 0.983914494514465, "LabelId" : 3 }, "Box2D" : { "xmin" : 0.854629874229431, "ymin" : 0.483299434185028, "xmax" : 0.938638925552368, "ymax" : 0.547508895397186 } }, ...
10. Other requests
In this tutorial, you have seen almost all the API requests except two:
- deleting a job: this is only possible if the job has not been submitted yet. The job will be deleted from the service database.
- canceling a job: it can be done while active. The job will stay in the service database but will not be processed anymore. Please refer to the API Reference.
As mentioned in the documentation, Reality Analysis API is batch oriented to allow concurrent analysis on large datasets. This is why the input and output ContextScenes go through an upload/download to/from Reality Management API. To simplify use cases where these files are light enough to go be transferred via the API requests, future versions will allow direct json transfers as a possible choice.
Continue learning
Congratulations for completing the Get started with the Reality Analysis API tutorial! You should now be able to detect 2D objects in a set of photos. Stay tuned for more tutorials. Meanwhile, you might want to try photo segmentation jobs.