API docs by Redocly

Starburst Manager (v2)

Download OpenAPI specification:

About

APIs to manage your system

⬇️ Download OpenAPI specification

Roles

All APIs support the standard Trino role header of X-Trino-Role. Set a role for the API request by setting the header to system=ROLE{role-name}.

Uploads

The combined size of an uploaded set of text files, binary files, or secrets cannot exceed 10MB and each individual file or secret cannot exceed 1MB.

Manager Service

System management

resumeInstance

Resume a cron batch job instance

Authorizations:
accessToken
path Parameters
instanceId
required
string
  • An instance in the system
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

batch

Shutdown and remove all batch jobs

Authorizations:
accessToken

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

allInstanceLogs

Get all the logs of an instance as a single Zip file

Authorizations:
accessToken
path Parameters
instanceId
required
string
  • An instance in the system
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

userInstance

List instances available to the current user

Authorizations:
accessToken
query Parameters
showAll
boolean

If true, show terminated instances as well as active instances

Responses

Response samples

Content type
application/json
[
  • {
    }
]

connect

Shutdown and remove all connect servers

Authorizations:
accessToken

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

shutdownInstance

Shutdown a notebook server instance

Authorizations:
accessToken
path Parameters
instanceId
required
string
  • An instance in the system
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

connectInstance

Start a new Spark Connect driver

Authorizations:
accessToken
query Parameters
doNotSaveConfiguration
boolean

Use this to disable saving the configuration for this instance. Normally, the configuration used to start instances is saved so that it can be examined and/or duplicated later. If for some reason you do not want this (due to sensitive information, for example), then add this query parameter.

Request Body schema: application/json
required
required
object (Common)

Responses

Request samples

Content type
application/json
{
  • "common": {
    }
}

Response samples

Content type
application/json
{
  • "instanceId": "string"
}

createPresignedUrl

Create pre-signed URLs for the given bucket, key and credentials

Authorizations:
accessToken
Request Body schema: application/json
required
accessKey
required
string

The proxy AccessKey provided by your administrator

secretKey
required
string

The proxy SecretKey provided by your administrator

region
required
string

The S3 region of the bucket

bucket
required
string

The bucket for creating the pre-signed URLs

key
required
string

The key for creating the pre-signed URLs

Responses

Request samples

Content type
application/json
{
  • "accessKey": "string",
  • "secretKey": "string",
  • "region": "string",
  • "bucket": "string",
  • "key": "string"
}

Response samples

Content type
application/json
{
  • "presignedUrls": {
    }
}

batchInstance

Submit and run a batch job

Authorizations:
accessToken
query Parameters
doNotSaveConfiguration
boolean

Use this to disable saving the configuration for this instance. Normally, the configuration used to start instances is saved so that it can be examined and/or duplicated later. If for some reason you do not want this (due to sensitive information, for example), then add this query parameter.

Request Body schema: application/json
required
applicationResource
required
string

The application resource to run - must be on an accessible object store

mainClass
string

The main class of the batch job

cronTimeZone
string

Optional CRON timezone. If provided, specify a time zone for the CRON job. Default is UTC.

See List of Time Zones.

cronSchedule
string

Optional CRON schedule. If provided, runs the job on the given schedule.

See Wikipedia's CRON article for details on CRON schedules.

ttlSecondsAfterFinished
integer <int32>

Optional.

ttlSecondsAfterFinished specifies the number of seconds after completion that the driver is eligible to be deleted/cleaned. Default is 1 seconds.

applicationArguments
Array of strings

Any application arguments

required
object (Common)

Responses

Request samples

Content type
application/json
{
  • "applicationResource": "string",
  • "mainClass": "string",
  • "cronTimeZone": "string",
  • "cronSchedule": "string",
  • "ttlSecondsAfterFinished": 0,
  • "applicationArguments": [
    ],
  • "common": {
    }
}

Response samples

Content type
application/json
{
  • "instanceId": "string"
}

restartInstance

Restart a notebook server instance

Authorizations:
accessToken
path Parameters
instanceId
required
string
  • An instance in the system
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

listEvent

List system events

Authorizations:
accessToken

Responses

Response samples

Content type
application/json
[
  • {
    }
]

duplicateInstance

Create/start a duplicate instance of the any instance

Authorizations:
accessToken
path Parameters
instanceId
required
string
  • An instance in the system
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)
query Parameters
doNotSaveConfiguration
boolean

Use this to disable saving the configuration for this instance. Normally, the configuration used to start instances is saved so that it can be examined and/or duplicated later. If for some reason you do not want this (due to sensitive information, for example), then add this query parameter.

Request Body schema: application/json
required
newName
string

The application name. If not provided a name will be generated.

resourcePool
string

Optional - the resource pool to use (you must have permission to use it)

Responses

Request samples

Content type
application/json
{
  • "newName": "string",
  • "resourcePool": "string"
}

Response samples

Content type
application/json
{
  • "instanceId": "string"
}

updateRole

Change the role associated with the token used in this API call

Authorizations:
accessToken
path Parameters
roleId
required
string

A role

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

getInstanceStatus

Get the status of an instance

Authorizations:
accessToken
path Parameters
instanceId
required
string
  • An instance in the system
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)
query Parameters
showAll
boolean

If true, show terminated instances as well as active instances

Responses

Response samples

Content type
application/json
{
  • "status": {
    }
}

getInstanceLogs

Get the log output of a batch job, a connect driver or a notebook server

Authorizations:
accessToken
path Parameters
instanceId
required
string
  • An instance in the system
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

notebookInstance

Start a new notebook server

Authorizations:
accessToken
query Parameters
doNotSaveConfiguration
boolean

Use this to disable saving the configuration for this instance. Normally, the configuration used to start instances is saved so that it can be examined and/or duplicated later. If for some reason you do not want this (due to sensitive information, for example), then add this query parameter.

Request Body schema: application/json
required
notebookName
string

The notebook name. If not provided a name will be generated.

resourcePool
string

Optional - the resource pool to use (you must have permission to use it)

instanceImageId
string

Instance image id to use (you must have permission to use it)

description
string

Optional - the description of the notebook instance

Responses

Request samples

Content type
application/json
{
  • "notebookName": "string",
  • "resourcePool": "string",
  • "instanceImageId": "string",
  • "description": "string"
}

Response samples

Content type
application/json
{
  • "instanceId": "string"
}

getLogs

Get the system logs

Authorizations:
accessToken
path Parameters
logsId
required
string

Logs from an instance

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

listInstance

List all known instances

Authorizations:
accessToken
query Parameters
showAll
boolean

If true, show terminated instances as well as active instances

Responses

Response samples

Content type
application/json
[
  • {
    }
]

suspendInstance

Suspend a cron batch job instance

Authorizations:
accessToken
path Parameters
instanceId
required
string
  • An instance in the system
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

getLogsWithIndex

Get the log output of an instance. logsId is the log index to return: Some cluster types have multiple nodes/workers. Pass 0 to get the main logs and then increase the index to get other logs.

Authorizations:
accessToken
path Parameters
instanceId
required
string
  • An instance in the system
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)
logsId
required
string

Logs from an instance

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

notebook

Shutdown and remove all notebook servers

Authorizations:
accessToken
query Parameters
preserveUserData
boolean

Preserve user data when deleting the notebook servers

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

deleteInstance

Shutdown and remove an instance

Authorizations:
accessToken
path Parameters
instanceId
required
string
  • An instance in the system
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)
query Parameters
preserveUserData
boolean

Preserve user data when deleting the notebook server

force
boolean

Use with caution! Immediately deletes instance without waiting for normal shutdown. This will likely result in loss of final logs, events, etc.

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

Resources Service

Resource pool management

getResourcePool

Return the current resource pool set including the total available memory and cores

Authorizations:
accessToken

Responses

Response samples

Content type
application/json
{
  • "totalMemory": "string",
  • "totalCores": "string",
  • "totalGpus": "string",
  • "resourcePools": {
    }
}

updateResourcePool

Update the set of available resource pools (replace the entire set)

Authorizations:
accessToken
Request Body schema: application/json
required
required
Array of objects (ResourcePool)

The set of resource pools

Responses

Request samples

Content type
application/json
{
  • "resourcePools": [
    ]
}

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

createResourcePool

Create new resource pool

Authorizations:
accessToken
Request Body schema: application/json
required
resourcePoolId
required
string

The name of this resource pool (must be unique)

priority
integer <int32>

The priority of this pool. Pools with larger/higher priority numbers have priority over pools with smaller/lower priority numbers. If not specified, the priority is "0".

minMemory
required
string

Minimum memory (as a Spark quantity string)

minCores
required
string

Minimum virtual cores (as a CPU unit)

maxMemory
required
string

Maximum memory (as a Spark quantity string)

maxCores
required
string

Maximum virtual cores (as a CPU unit)

maxGpus
string

Maximum gpus (as a Spark quantity string)

applicationMaxGpus
string

Maximum gpus per application (as a Spark quantity string)

Responses

Request samples

Content type
application/json
{
  • "resourcePoolId": "string",
  • "priority": 0,
  • "minMemory": "string",
  • "minCores": "string",
  • "maxMemory": "string",
  • "maxCores": "string",
  • "maxGpus": "string",
  • "applicationMaxGpus": "string"
}

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

resources.updateResourcePool

Edit resource pool (update all parameters for a single resource pool)

Authorizations:
accessToken
path Parameters
resourcePoolId
required
string

A resource pool that describes a minimum and maximum resource set of memory and cores

Request Body schema: application/json
required
resourcePoolId
required
string

The name of this resource pool (must be unique)

priority
integer <int32>

The priority of this pool. Pools with larger/higher priority numbers have priority over pools with smaller/lower priority numbers. If not specified, the priority is "0".

minMemory
required
string

Minimum memory (as a Spark quantity string)

minCores
required
string

Minimum virtual cores (as a CPU unit)

maxMemory
required
string

Maximum memory (as a Spark quantity string)

maxCores
required
string

Maximum virtual cores (as a CPU unit)

maxGpus
string

Maximum gpus (as a Spark quantity string)

applicationMaxGpus
string

Maximum gpus per application (as a Spark quantity string)

Responses

Request samples

Content type
application/json
{
  • "resourcePoolId": "string",
  • "priority": 0,
  • "minMemory": "string",
  • "minCores": "string",
  • "maxMemory": "string",
  • "maxCores": "string",
  • "maxGpus": "string",
  • "applicationMaxGpus": "string"
}

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

deleteResourcePool

Delete resource pool

Authorizations:
accessToken
path Parameters
resourcePoolId
required
string

A resource pool that describes a minimum and maximum resource set of memory and cores

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

patchUpdateResourcePool

Update single resource pool parameters

Only include fields you wish to update. Missing or unrecognized fields are ignored.

Authorizations:
accessToken
path Parameters
resourcePoolId
required
string

A resource pool that describes a minimum and maximum resource set of memory and cores

Request Body schema: application/json
required
resourcePoolId
string

The name of this resource pool (must be unique)

priority
integer <int32>

The priority of this pool. Pools with larger/higher priority numbers have priority over pools with smaller/lower priority numbers. If not specified, the priority is "0".

minMemory
string

Minimum memory (as a Spark quantity string)

minCores
string

Minimum virtual cores (as a CPU unit)

maxMemory
string

Maximum memory (as a Spark quantity string)

maxCores
string

Maximum virtual cores (as a CPU unit)

maxGpus
string

Maximum gpus (as a Spark quantity string)

applicationMaxGpus
string

Maximum gpus per application (as a Spark quantity string)

Responses

Request samples

Content type
application/json
{
  • "resourcePoolId": "string",
  • "priority": 0,
  • "minMemory": "string",
  • "minCores": "string",
  • "maxMemory": "string",
  • "maxCores": "string",
  • "maxGpus": "string",
  • "applicationMaxGpus": "string"
}

Response samples

Content type
application/json
{
  • "resourcePoolId": "string",
  • "priority": 0,
  • "minMemory": "string",
  • "minCores": "string",
  • "maxMemory": "string",
  • "maxCores": "string",
  • "maxGpus": "string",
  • "applicationMaxGpus": "string"
}

Admin Service

Administration

getLogRetentionSettings

Get log retention settings

Authorizations:
accessToken

Responses

Response samples

Content type
application/json
{
  • "sparkEventLogsCleanerEnabled": true,
  • "sparkEventLogsCleanerInterval": "string",
  • "sparkEventLogsCleanerMaxAge": "string",
  • "sparkEventLogsCompactionEnabled": true,
  • "sparkEventLogsCompactionMaxFilesToRetain": 0,
  • "instanceLogsCleanerEnabled": true,
  • "instanceLogsCleanerInterval": "string",
  • "instanceLogsCleanerMaxAge": "string"
}

updateLogRetentionSettings

Update log retention settings

Authorizations:
accessToken
Request Body schema: application/json
required
sparkEventLogsCleanerEnabled
boolean

Enable Spark event logs cleaner

sparkEventLogsCleanerInterval
string

Interval for Spark event logs cleaner

sparkEventLogsCleanerMaxAge
string

Max age for Spark event logs to retain

sparkEventLogsCompactionEnabled
boolean

Enable Spark event logs compaction

sparkEventLogsCompactionMaxFilesToRetain
string

Max files to retain after Spark event logs compaction

instanceLogsCleanerEnabled
boolean

Enable instance logs cleaner

instanceLogsCleanerInterval
string

Interval for instance logs cleaner

instanceLogsCleanerMaxAge
string

Max age for instance logs to retain

Responses

Request samples

Content type
application/json
{
  • "sparkEventLogsCleanerEnabled": true,
  • "sparkEventLogsCleanerInterval": "string",
  • "sparkEventLogsCleanerMaxAge": "string",
  • "sparkEventLogsCompactionEnabled": true,
  • "sparkEventLogsCompactionMaxFilesToRetain": "string",
  • "instanceLogsCleanerEnabled": true,
  • "instanceLogsCleanerInterval": "string",
  • "instanceLogsCleanerMaxAge": "string"
}

Response samples

Content type
application/json
{
  • "sparkEventLogsCleanerEnabled": true,
  • "sparkEventLogsCleanerInterval": "string",
  • "sparkEventLogsCleanerMaxAge": "string",
  • "sparkEventLogsCompactionEnabled": true,
  • "sparkEventLogsCompactionMaxFilesToRetain": 0,
  • "instanceLogsCleanerEnabled": true,
  • "instanceLogsCleanerInterval": "string",
  • "instanceLogsCleanerMaxAge": "string"
}

deleteLogRetentionSettings

Reset log retention settings to defaults

Authorizations:
accessToken

Responses

Response samples

Content type
application/json
{
  • "sparkEventLogsCleanerEnabled": true,
  • "sparkEventLogsCleanerInterval": "string",
  • "sparkEventLogsCleanerMaxAge": "string",
  • "sparkEventLogsCompactionEnabled": true,
  • "sparkEventLogsCompactionMaxFilesToRetain": 0,
  • "instanceLogsCleanerEnabled": true,
  • "instanceLogsCleanerInterval": "string",
  • "instanceLogsCleanerMaxAge": "string"
}

getGlobalStorageUsage

Get global storage usage

Authorizations:
accessToken
query Parameters
moreInstances
boolean

Gets top 1000 instances usage

allResourcePools
boolean

Gets all resource pools usage

Responses

Response samples

Content type
application/json
{
  • "totalUsedBytes": 0,
  • "totalObjectsCount": 0,
  • "logsUsedBytes": 0,
  • "logObjectsCount": 0,
  • "historyServerEventsUsedBytes": 0,
  • "historyServerEventsObjectsCount": 0,
  • "notebookWorkspacesUsedBytes": 0,
  • "notebookWorkspacesObjectsCount": 0,
  • "instanceUsages": [
    ],
  • "resourcePoolUsages": [
    ],
  • "lastRefreshedAt": "2019-08-24T14:15:22Z"
}

listSystemMaintenanceHistory

Show system maintenance history

Authorizations:
accessToken

Responses

Response samples

Content type
application/json
[
  • {
    }
]

statusInstanceLogsCleaner

Get instance logs cleaner status

Authorizations:
accessToken

Responses

Response samples

Content type
application/json
{
  • "running": true,
  • "lastCleanedAt": "2019-08-24T14:15:22Z"
}

stopInstanceLogsCleaner

Stop instance logs cleaner

Authorizations:
accessToken

Responses

Response samples

Content type
application/json
null

refreshGlobalStorageUsage

Refresh global storage usage

Authorizations:
accessToken

Responses

Response samples

Content type
application/json
null

getSystemMaintenance

Check if the system maintenance is currently activated

Authorizations:
accessToken

Responses

Response samples

Content type
application/json
{
  • "status": "Deactivated"
}

createSystemMaintenance

Activate system maintenance

Authorizations:
accessToken

Responses

Response samples

Content type
application/json
{
  • "status": "Deactivated"
}

deleteSystemMaintenance

Deactivate system maintenance

Authorizations:
accessToken

Responses

Response samples

Content type
application/json
{
  • "status": "Deactivated"
}

runInstanceLogsCleaner

Run instance logs cleaner

Authorizations:
accessToken

Responses

Response samples

Content type
application/json
null

Uploads Service

File and secret upload management

getFileUpload

Get a file upload

Authorizations:
accessToken
path Parameters
uploadId
required
string

A text or binary file. See Uploads for more details and limits.

Responses

Response samples

Content type
application/json
{
  • "comment": "string",
  • "textData": {
    },
  • "binaryData": {
    }
}

updateFileUpload

Update a file upload

Authorizations:
accessToken
path Parameters
uploadId
required
string

A text or binary file. See Uploads for more details and limits.

Request Body schema: application/json
required
comment
required
string

Comment or description. Used only for your own reference purposes.

required
object

Map of name-to-text files/data

required
object

Map of name-to-binary files/data. Data must be Base64 encoded.

Responses

Request samples

Content type
application/json
{
  • "comment": "string",
  • "textData": {
    },
  • "binaryData": {
    }
}

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

deleteFileUpload

Delete a file upload

Authorizations:
accessToken
path Parameters
uploadId
required
string

A text or binary file. See Uploads for more details and limits.

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

listFileUploads

List file uploads

Authorizations:
accessToken

Responses

Response samples

Content type
application/json
[
  • {
    }
]

createFileUpload

Create a new file upload

Authorizations:
accessToken
Request Body schema: application/json
required
comment
required
string

Comment or description. Used only for your own reference purposes.

required
object

Map of name-to-text files/data

required
object

Map of name-to-binary files/data. Data must be Base64 encoded.

Responses

Request samples

Content type
application/json
{
  • "comment": "string",
  • "textData": {
    },
  • "binaryData": {
    }
}

Response samples

Content type
application/json
{
  • "uploadId": "string",
  • "comment": "string",
  • "textNames": [
    ],
  • "binaryNames": [
    ]
}

listSecretUploads

List file uploads

Authorizations:
accessToken

Responses

Response samples

Content type
application/json
[
  • {
    }
]

createSecretUpload

Create a new secret upload

Authorizations:
accessToken
Request Body schema: application/json
required
comment
required
string

Comment or description. Used only for your own reference purposes.

required
object

Map of name-to-binary secrets. Data must be Base64 encoded.

When the uploaded secret is used in an instance this map of secrets/values are set as environment variables. Thus, the secret name must be valid environment variable identifier.

See Uploads for more details and limits.

Responses

Request samples

Content type
application/json
{
  • "comment": "string",
  • "secrets": {
    }
}

Response samples

Content type
application/json
{
  • "uploadId": "string",
  • "comment": "string",
  • "secretNames": [
    ]
}

getSecretUpload

Get a secret upload

Authorizations:
accessToken
path Parameters
uploadId
required
string

A text or binary file. See Uploads for more details and limits.

Responses

Response samples

Content type
application/json
{
  • "comment": "string",
  • "secretNames": [
    ]
}

updateSecretUpload

Update a secret upload

Authorizations:
accessToken
path Parameters
uploadId
required
string

A text or binary file. See Uploads for more details and limits.

Request Body schema: application/json
required
comment
required
string

Comment or description. Used only for your own reference purposes.

required
object

Map of name-to-binary secrets. Data must be Base64 encoded.

When the uploaded secret is used in an instance this map of secrets/values are set as environment variables. Thus, the secret name must be valid environment variable identifier.

See Uploads for more details and limits.

Responses

Request samples

Content type
application/json
{
  • "comment": "string",
  • "secrets": {
    }
}

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

deleteSecretUpload

Delete a secret upload

Authorizations:
accessToken
path Parameters
uploadId
required
string

A text or binary file. See Uploads for more details and limits.

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

InstanceImageImports Service

Instance image imports management

listInstanceImageImport

Get all instance image imports

Authorizations:
accessToken
query Parameters
pageToken
string

Pagination token

pageSize
integer <int32>

Page size or 0 for default (current maximum is 100)

Responses

Response samples

Content type
application/json
{
  • "nextPageToken": "string",
  • "result": [
    ]
}

createInstanceImageImport

Submit and run a instance image import

Authorizations:
accessToken
Request Body schema: application/json
required
name
required
string

Image name

repository
required
string

Registry with repository name to import

tag
required
string

Image tag to import

description
string

Optional image description

allowInsecure
required
boolean

Allow insecure registry connection

isPrivate
required
boolean

Registry require credentials for accessing

object (Credentials)

Credentials required for accessing registry

Responses

Request samples

Content type
application/json
{
  • "name": "string",
  • "repository": "string",
  • "tag": "string",
  • "description": "string",
  • "allowInsecure": true,
  • "isPrivate": true,
  • "credentials": {
    }
}

Response samples

Content type
application/json
{
  • "instanceImageImportId": "string"
}

getInstanceImageImport

Get image import details

Authorizations:
accessToken
path Parameters
instanceImageImportId
required
string

An instance image import

Responses

Response samples

Content type
application/json
{
  • "id": "string",
  • "name": "string",
  • "repository": "string",
  • "tag": "string",
  • "description": "string",
  • "allowInsecure": true,
  • "isPrivate": true,
  • "status": "New",
  • "statusDescription": "string",
  • "createdBy": "string",
  • "createdAt": "2019-08-24T14:15:22Z",
  • "updatedAt": "2019-08-24T14:15:22Z"
}

InstanceImages Service

Instance image management

updateInstanceImageInstanceImageStatus

Update instance image status

Authorizations:
accessToken
path Parameters
instanceImageId
required
string

An instance image

Request Body schema: application/json
required
status
required
string
Enum: "Init" "Pending" "Verified" "Rejected" "MarkedForDelete" "Deleted"

Image status

statusDescription
string

Status description

Responses

Request samples

Content type
application/json
{
  • "status": "Init",
  • "statusDescription": "string"
}

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

getInstanceImage

Get instance image details

Authorizations:
accessToken
path Parameters
instanceImageId
required
string

An instance image

Responses

Response samples

Content type
application/json
{
  • "instanceImageInfo": {
    },
  • "layers": [
    ],
  • "history": [
    ]
}

userlistInstanceImage

Get available instance images for users

Authorizations:
accessToken
query Parameters
pageToken
string

Pagination token

pageSize
integer <int32>

Page size or 0 for default (current maximum is 100)

Responses

Response samples

Content type
application/json
{
  • "nextPageToken": "string",
  • "result": [
    ]
}

listInstanceImage

Get all instance image

Authorizations:
accessToken
query Parameters
pageToken
string

Pagination token

pageSize
integer <int32>

Page size or 0 for default (current maximum is 100)

Responses

Response samples

Content type
application/json
{
  • "nextPageToken": "string",
  • "result": [
    ]
}

Model Definitions

SystemMaintenance

status
required
string
Enum: "Deactivated" "WaitingForJobsTermination" "ReadyForSafeUpgrade"

System maintenance status

{
  • "status": "Deactivated"
}

Instance

instanceId
required
string

The instance Id

{
  • "instanceId": "string"
}

InstanceImageOriginInfo

repository
required
string

Origin repository

tag
required
string

Origin tag

isPrivate
required
boolean

Origin required credentials

taskId
string

Origin import task id

{
  • "repository": "string",
  • "tag": "string",
  • "isPrivate": true,
  • "taskId": "string"
}

PaginatedInstanceImageInfo

nextPageToken
string

The next page token to use or "" if there are no more pages.

Array of objects (InstanceImageInfo)

A page of results.

{
  • "nextPageToken": "string",
  • "result": [
    ]
}

InstanceImageDetailsInfo

required
object (InstanceImageInfo)

An instance image

required
Array of objects (InstanceLayerInfo)

Layers

required
Array of objects (InstanceHistoryInfo)

History

{
  • "instanceImageInfo": {
    },
  • "layers": [
    ],
  • "history": [
    ]
}

GlobalStorageUsage

totalUsedBytes
required
integer <int64>

Total storage used (bytes)

totalObjectsCount
required
integer <int64>

Total number of objects stored

logsUsedBytes
required
integer <int64>

Storage used by logs (bytes)

logObjectsCount
required
integer <int64>

Number of log objects

historyServerEventsUsedBytes
required
integer <int64>

Storage used by history server events (bytes)

historyServerEventsObjectsCount
required
integer <int64>

Number of objects stored by history server events

notebookWorkspacesUsedBytes
required
integer <int64>

Storage used by notebook workspaces (bytes)

notebookWorkspacesObjectsCount
required
integer <int64>

Number of objects stored in notebook workspaces

required
Array of objects (InstanceStorageUsage)

Storage usage details per instance

required
Array of objects (ResourcePoolStorageUsage)

Storage usage details per resource pool

lastRefreshedAt
required
string <date-time>

Last refreshed at

{
  • "totalUsedBytes": 0,
  • "totalObjectsCount": 0,
  • "logsUsedBytes": 0,
  • "logObjectsCount": 0,
  • "historyServerEventsUsedBytes": 0,
  • "historyServerEventsObjectsCount": 0,
  • "notebookWorkspacesUsedBytes": 0,
  • "notebookWorkspacesObjectsCount": 0,
  • "instanceUsages": [
    ],
  • "resourcePoolUsages": [
    ],
  • "lastRefreshedAt": "2019-08-24T14:15:22Z"
}

InstanceImageTypeInfo

kind
required
string
Enum: "Spark" "Notebook"

An instance image type

version
required
string

An instance image version

{
  • "kind": "Spark",
  • "version": "string"
}

ResourcePoolSet

required
Array of objects (ResourcePool)

The set of resource pools

{
  • "resourcePools": [
    ]
}

InstanceImageStatus

status
required
string
Enum: "Init" "Pending" "Verified" "Rejected" "MarkedForDelete" "Deleted"

Image status

statusDescription
string

Status description

{
  • "status": "Init",
  • "statusDescription": "string"
}

SparkBatchJob

applicationResource
required
string

The application resource to run - must be on an accessible object store

mainClass
string

The main class of the batch job

cronTimeZone
string

Optional CRON timezone. If provided, specify a time zone for the CRON job. Default is UTC.

See List of Time Zones.

cronSchedule
string

Optional CRON schedule. If provided, runs the job on the given schedule.

See Wikipedia's CRON article for details on CRON schedules.

ttlSecondsAfterFinished
integer <int32>

Optional.

ttlSecondsAfterFinished specifies the number of seconds after completion that the driver is eligible to be deleted/cleaned. Default is 1 seconds.

applicationArguments
Array of strings

Any application arguments

required
object (Common)
{
  • "applicationResource": "string",
  • "mainClass": "string",
  • "cronTimeZone": "string",
  • "cronSchedule": "string",
  • "ttlSecondsAfterFinished": 0,
  • "applicationArguments": [
    ],
  • "common": {
    }
}

EventV2

time
required
string

Time of the event

type
required
string

Event type

reason
required
string

Event reason

name
required
string

Event name

action
required
string

Event action

{
  • "time": "string",
  • "type": "string",
  • "reason": "string",
  • "name": "string",
  • "action": "string"
}

PaginatedInstanceImageImportInfo

nextPageToken
string

The next page token to use or "" if there are no more pages.

Array of objects (InstanceImageImportInfo)

A page of results.

{
  • "nextPageToken": "string",
  • "result": [
    ]
}

ResourcePoolsInfo

totalMemory
required
string

Total available memory (as a Spark quantity string)

totalCores
required
string

Total available virtual cores

totalGpus
required
string

Total available gpus (as quantity string)

required
object (ResourcePoolSet)
{
  • "totalMemory": "string",
  • "totalCores": "string",
  • "totalGpus": "string",
  • "resourcePools": {
    }
}

StatusV2

required
object

The status

{
  • "status": {
    }
}

InstanceLayerInfo

mime
required
string

Mime type

digest
required
string

Digest

size
required
integer <int64>

Size

annotations
string

Annotations

{
  • "mime": "string",
  • "digest": "string",
  • "size": 0,
  • "annotations": "string"
}

ResourcePoolStorageUsage

resourcePoolId
required
string

The name of the resource pool

usedBytes
required
integer <int64>

Storage bytes used by this resource pool

objectsCount
required
integer <int64>

Number of objects stored by this resource pool

{
  • "resourcePoolId": "string",
  • "usedBytes": 0,
  • "objectsCount": 0
}

InstanceHistoryInfo

created
required
string <date-time>

Created

command
required
string

Command

comment
required
string

Comment

emptyLayer
required
boolean

Is layer empty

{
  • "created": "2019-08-24T14:15:22Z",
  • "command": "string",
  • "comment": "string",
  • "emptyLayer": true
}

RedactedSecretUpload

comment
required
string

Comment or description. Used only for your own reference purposes.

secretNames
required
Array of strings

Secret names

{
  • "comment": "string",
  • "secretNames": [
    ]
}

RunInstanceLogsCleaner

any (RunInstanceLogsCleaner)

Manage instance logs cleaner

null

ResourcePoolPatch

resourcePoolId
string

The name of this resource pool (must be unique)

priority
integer <int32>

The priority of this pool. Pools with larger/higher priority numbers have priority over pools with smaller/lower priority numbers. If not specified, the priority is "0".

minMemory
string

Minimum memory (as a Spark quantity string)

minCores
string

Minimum virtual cores (as a CPU unit)

maxMemory
string

Maximum memory (as a Spark quantity string)

maxCores
string

Maximum virtual cores (as a CPU unit)

maxGpus
string

Maximum gpus (as a Spark quantity string)

applicationMaxGpus
string

Maximum gpus per application (as a Spark quantity string)

{
  • "resourcePoolId": "string",
  • "priority": 0,
  • "minMemory": "string",
  • "minCores": "string",
  • "maxMemory": "string",
  • "maxCores": "string",
  • "maxGpus": "string",
  • "applicationMaxGpus": "string"
}

NotebookInstance

notebookName
string

The notebook name. If not provided a name will be generated.

resourcePool
string

Optional - the resource pool to use (you must have permission to use it)

instanceImageId
string

Instance image id to use (you must have permission to use it)

description
string

Optional - the description of the notebook instance

{
  • "notebookName": "string",
  • "resourcePool": "string",
  • "instanceImageId": "string",
  • "description": "string"
}

InstanceLogsCleanerStatus

running
required
boolean

Instance log cleaning is being run

lastCleanedAt
string <date-time>

Last cleaned at

{
  • "running": true,
  • "lastCleanedAt": "2019-08-24T14:15:22Z"
}

InstanceImageImport

instanceImageImportId
required
string

The image import Id

{
  • "instanceImageImportId": "string"
}

InstanceStorageUsage

instanceId
required
string

The instance id

usedBytes
required
integer <int64>

Storage bytes used by this instance

objectsCount
required
integer <int64>

Number of objects stored by this instance

{
  • "instanceId": "string",
  • "usedBytes": 0,
  • "objectsCount": 0
}

LogRetentionSettings

sparkEventLogsCleanerEnabled
required
boolean

Enable Spark event logs cleaner

sparkEventLogsCleanerInterval
required
string

Interval for Spark event logs cleaner

sparkEventLogsCleanerMaxAge
required
string

Max age for Spark event logs to retain

sparkEventLogsCompactionEnabled
required
boolean

Enable Spark event logs compaction

sparkEventLogsCompactionMaxFilesToRetain
required
integer <int32>

Max files to retain after Spark event logs compaction

instanceLogsCleanerEnabled
required
boolean

Enable instance logs cleaner

instanceLogsCleanerInterval
required
string

Interval for instance logs cleaner

instanceLogsCleanerMaxAge
required
string

Max age for instance logs to retain

{
  • "sparkEventLogsCleanerEnabled": true,
  • "sparkEventLogsCleanerInterval": "string",
  • "sparkEventLogsCleanerMaxAge": "string",
  • "sparkEventLogsCompactionEnabled": true,
  • "sparkEventLogsCompactionMaxFilesToRetain": 0,
  • "instanceLogsCleanerEnabled": true,
  • "instanceLogsCleanerInterval": "string",
  • "instanceLogsCleanerMaxAge": "string"
}

InstanceImageInfo

id
required
string

The instance image Id

name
required
string

Imported image name

tag
required
string

Image tag

description
string

Optional image description

required
object (InstanceImageOriginInfo)

An instance image origin

baseImageId
string

Image Id of base image

digest
required
string

Image digest

required
object (InstanceImageTypeInfo)

Image type

status
required
string
Enum: "Init" "Pending" "Verified" "Rejected" "MarkedForDelete" "Deleted"

Image status

statusDescription
string

Status description

createdBy
required
string

User that created instance image

createdAt
required
string <date-time>

Instance creation timestamp

updatedAt
required
string <date-time>

Instance last update timestamp

{
  • "id": "string",
  • "name": "string",
  • "tag": "string",
  • "description": "string",
  • "origin": {
    },
  • "baseImageId": "string",
  • "digest": "string",
  • "type": {
    },
  • "status": "Init",
  • "statusDescription": "string",
  • "createdBy": "string",
  • "createdAt": "2019-08-24T14:15:22Z",
  • "updatedAt": "2019-08-24T14:15:22Z"
}

InstanceInfo

instanceId
required
string

The instance Id

name
required
string

The instance name

type
required
string

The instance type

createdBy
required
string

User that created the instance

ownerRole
string

Role that owns the instance

required
object

Any additional details about the instance

{
  • "instanceId": "string",
  • "name": "string",
  • "type": "string",
  • "createdBy": "string",
  • "ownerRole": "string",
  • "details": {
    }
}

InstanceImageImportCreate

name
required
string

Image name

repository
required
string

Registry with repository name to import

tag
required
string

Image tag to import

description
string

Optional image description

allowInsecure
required
boolean

Allow insecure registry connection

isPrivate
required
boolean

Registry require credentials for accessing

object (Credentials)

Credentials required for accessing registry

{
  • "name": "string",
  • "repository": "string",
  • "tag": "string",
  • "description": "string",
  • "allowInsecure": true,
  • "isPrivate": true,
  • "credentials": {
    }
}

PaginatedUserInstanceImageInfo

nextPageToken
string

The next page token to use or "" if there are no more pages.

Array of objects (UserInstanceImageInfo)

A page of results.

{
  • "nextPageToken": "string",
  • "result": [
    ]
}

Common

applicationName
string

The application name. If not provided a name will be generated.

object

Any Spark configuration/properties to set arguments

jars
Array of strings

Any jars to pass in the --jars Spark argument

pythonFiles
Array of strings

Any python files to pass in the --py-files Spark argument

files
Array of strings

Any files to pass in the --files Spark argument

archives
Array of strings

Any archives to pass in the --archives Spark argument

object

Any environment variables to set

resourcePool
string

Optional - the resource pool to use (you must have permission to use it)

instanceImageId
string

Instance image id to use (you must have permission to use it)

secretUploadIds
Array of strings

Optional - secret uploads

Secrets will be set as environment variables in the Spark driver and executors.

secretUploads
Array of strings

DEPRECATED - use secretUploadIds

Array of objects (UploadMapping)

Optional - file uploads (read only)

Array of objects (InlineUploadMapping)

Optional - inline file uploads. See Uploads for more details and limits. (read only)

options
Array of strings
Items Value: "EncryptCommunication"

Details:

  • "EncryptCommunication": Enable encryption for communication between the driver and executors
{
  • "applicationName": "string",
  • "sparkProperties": {
    },
  • "jars": [
    ],
  • "pythonFiles": [
    ],
  • "files": [
    ],
  • "archives": [
    ],
  • "environmentVariables": {
    },
  • "resourcePool": "string",
  • "instanceImageId": "string",
  • "secretUploadIds": [
    ],
  • "secretUploads": [
    ],
  • "fileUploads": [
    ],
  • "inlineFileUploads": [
    ],
  • "options": [
    ]
}

PresignedUrlRequest

accessKey
required
string

The proxy AccessKey provided by your administrator

secretKey
required
string

The proxy SecretKey provided by your administrator

region
required
string

The S3 region of the bucket

bucket
required
string

The bucket for creating the pre-signed URLs

key
required
string

The key for creating the pre-signed URLs

{
  • "accessKey": "string",
  • "secretKey": "string",
  • "region": "string",
  • "bucket": "string",
  • "key": "string"
}

UploadInfo

uploadId
required
string

The ID of this upload

comment
required
string

Comment or description. Used only for your own reference purposes.

textNames
required
Array of strings

Names of text data in the upload

binaryNames
required
Array of strings

Names of binary data in the upload

{
  • "uploadId": "string",
  • "comment": "string",
  • "textNames": [
    ],
  • "binaryNames": [
    ]
}

ResourcePool

resourcePoolId
required
string

The name of this resource pool (must be unique)

priority
integer <int32>

The priority of this pool. Pools with larger/higher priority numbers have priority over pools with smaller/lower priority numbers. If not specified, the priority is "0".

minMemory
required
string

Minimum memory (as a Spark quantity string)

minCores
required
string

Minimum virtual cores (as a CPU unit)

maxMemory
required
string

Maximum memory (as a Spark quantity string)

maxCores
required
string

Maximum virtual cores (as a CPU unit)

maxGpus
string

Maximum gpus (as a Spark quantity string)

applicationMaxGpus
string

Maximum gpus per application (as a Spark quantity string)

{
  • "resourcePoolId": "string",
  • "priority": 0,
  • "minMemory": "string",
  • "minCores": "string",
  • "maxMemory": "string",
  • "maxCores": "string",
  • "maxGpus": "string",
  • "applicationMaxGpus": "string"
}

DuplicateInstance

newName
string

The application name. If not provided a name will be generated.

resourcePool
string

Optional - the resource pool to use (you must have permission to use it)

{
  • "newName": "string",
  • "resourcePool": "string"
}

SparkConnectDriver

required
object (Common)
{
  • "common": {
    }
}

LogRetentionSettingsUpdate

sparkEventLogsCleanerEnabled
boolean

Enable Spark event logs cleaner

sparkEventLogsCleanerInterval
string

Interval for Spark event logs cleaner

sparkEventLogsCleanerMaxAge
string

Max age for Spark event logs to retain

sparkEventLogsCompactionEnabled
boolean

Enable Spark event logs compaction

sparkEventLogsCompactionMaxFilesToRetain
string

Max files to retain after Spark event logs compaction

instanceLogsCleanerEnabled
boolean

Enable instance logs cleaner

instanceLogsCleanerInterval
string

Interval for instance logs cleaner

instanceLogsCleanerMaxAge
string

Max age for instance logs to retain

{
  • "sparkEventLogsCleanerEnabled": true,
  • "sparkEventLogsCleanerInterval": "string",
  • "sparkEventLogsCleanerMaxAge": "string",
  • "sparkEventLogsCompactionEnabled": true,
  • "sparkEventLogsCompactionMaxFilesToRetain": "string",
  • "instanceLogsCleanerEnabled": true,
  • "instanceLogsCleanerInterval": "string",
  • "instanceLogsCleanerMaxAge": "string"
}

UserInstanceImageInfo

id
required
string

The instance image Id

name
required
string

Imported image name

tag
required
string

Image tag

description
string

Optional image description

required
object (UserInstanceImageTypeInfo)

Image type

status
required
string
Enum: "Init" "Pending" "Verified" "Rejected" "MarkedForDelete" "Deleted"

Image status

{
  • "id": "string",
  • "name": "string",
  • "tag": "string",
  • "description": "string",
  • "type": {
    },
  • "status": "Init"
}

UserInstanceImageTypeInfo

kind
required
string
Enum: "Spark" "Notebook"

An instance image type

version
required
string

An instance image version

{
  • "kind": "Spark",
  • "version": "string"
}

SecretUpload

comment
required
string

Comment or description. Used only for your own reference purposes.

required
object

Map of name-to-binary secrets. Data must be Base64 encoded.

When the uploaded secret is used in an instance this map of secrets/values are set as environment variables. Thus, the secret name must be valid environment variable identifier.

See Uploads for more details and limits.

{
  • "comment": "string",
  • "secrets": {
    }
}

PresignedUrl

required
object

The pre-signed URLs. The key is an HTTP verb (GET, PUT, POST, DELETE). The value is the pre-signed URL

{
  • "presignedUrls": {
    }
}

UploadMapping

uploadId
required
string

The upload ID

mountPath
required
string

The path to mount the files at. This must be a partial path as an uploads directory will be prefixed to it. E.g. if mountPath is /my/mount/path then the actual directory will be /opt/spark/uploads/my/mount/path

{
  • "uploadId": "string",
  • "mountPath": "string"
}

Credentials

username
required
string

Username

password
required
string

Password

{
  • "username": "string",
  • "password": "string"
}

InstanceImageImportInfo

id
required
string

The instance image Id

name
required
string

Imported image name

repository
required
string

Registry with repository name to import

tag
required
string

Image tag to import

description
string

Optional image description

allowInsecure
required
boolean

Allow insecure registry connection

isPrivate
required
boolean

Registry require credentials for accessing

status
required
string
Enum: "New" "InProgress" "Completed" "Failed"

Instance image import status

statusDescription
string

Optional status description

createdBy
required
string

User that created instance image import

createdAt
required
string <date-time>

Instance import creation timestamp

updatedAt
required
string <date-time>

Instance import last update timestamp

{
  • "id": "string",
  • "name": "string",
  • "repository": "string",
  • "tag": "string",
  • "description": "string",
  • "allowInsecure": true,
  • "isPrivate": true,
  • "status": "New",
  • "statusDescription": "string",
  • "createdBy": "string",
  • "createdAt": "2019-08-24T14:15:22Z",
  • "updatedAt": "2019-08-24T14:15:22Z"
}

Upload

comment
required
string

Comment or description. Used only for your own reference purposes.

required
object

Map of name-to-text files/data

required
object

Map of name-to-binary files/data. Data must be Base64 encoded.

{
  • "comment": "string",
  • "textData": {
    },
  • "binaryData": {
    }
}

SecretUploadInfo

uploadId
required
string

The ID of this upload

comment
required
string

Comment or description. Used only for your own reference purposes.

secretNames
required
Array of strings

Secret names

{
  • "uploadId": "string",
  • "comment": "string",
  • "secretNames": [
    ]
}

InlineUploadMapping

comment
required
string

Comment or description. Used only for your own reference purposes.

fileName
required
string

The name/file name of the upload

uploadType
required
string
Enum: "Text" "Binary"

Whether this is a text or binary upload

data
required
string

The upload data. For binary, it must be Base64 encoded.

mountPath
required
string

The path to mount the files at. This must be a partial path as an uploads directory will be prefixed to it. E.g. if mountPath is /my/mount/path then the actual directory will be /opt/spark/uploads/my/mount/path

{
  • "comment": "string",
  • "fileName": "string",
  • "uploadType": "Text",
  • "data": "string",
  • "mountPath": "string"
}

SystemMaintenanceHistory

activatedAt
required
string

The system maintenance enabled at

activatedBy
required
string

User that activated the system maintenance

activatedVersion
required
string

The version of the system for which the maintenance mode has been activated

deactivatedAt
string

The system maintenance deactivated at

deactivatedBy
string

User that deactivated the system maintenance

deactivatedVersion
string

The version of the system for which the maintenance mode has been deactivated

{
  • "activatedAt": "string",
  • "activatedBy": "string",
  • "activatedVersion": "string",
  • "deactivatedAt": "string",
  • "deactivatedBy": "string",
  • "deactivatedVersion": "string"
}

RefreshGlobalStorageUsageResponse

any (RefreshGlobalStorageUsageResponse)

Refresh global storage usage response

null

Responses

NotFound

message
required
string

Name of the resource

description
string

Violation description

fields
Array of strings

Any fields/values that weren't found

{
  • "message": "string",
  • "description": "string",
  • "fields": [
    ]
}

Forbidden

message
string
{
  • "message": "string"
}

BadRequest

message
string

Description of the error

fields
Array of strings

Any fields/values that are invalid or incorrect

{
  • "message": "string",
  • "fields": [
    ]
}

Aborted

message
string
{
  • "message": "string"
}

Unauthorized

message
string
{
  • "message": "string"
}

Exists

message
string
{
  • "message": "string"
}

InternalError

message
string

Description of the error

{
  • "message": "string"
}

Unavailable

message
required
string

Error message

{
  • "message": "string"
}