API docs by Redocly

Starburst Manager (v2)

Download OpenAPI specification:

About

APIs to manage your system

⬇️ Download OpenAPI specification

Roles

All APIs support the standard Trino role header of X-Trino-Role. Set a role for the API request by setting the header to system=ROLE{role-name}.

Uploads

The combined size of an uploaded set of text files, binary files, or secrets cannot exceed 10MB and each individual file or secret cannot exceed 1MB.

Manager Service

System management

allInstanceLogs

Get all the logs of an instance as a single Zip file

Authorizations:
None
path Parameters
instanceId
required
string
  • An instance in the system
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

batchInstance

Submit and run a batch job

Authorizations:
None
Request Body schema: application/json
required
applicationResource
required
string

The application resource to run - must be on an accessible object store

mainClass
string

The main class of the batch job

cronTimeZone
string

Optional CRON timezone. If provided, specify a time zone for the CRON job. Default is UTC.

See List of Time Zones.

cronSchedule
string

Optional CRON schedule. If provided, runs the job on the given schedule.

See Wikipedia's CRON article for details on CRON schedules.

ttlSecondsAfterFinished
integer <int32>

Optional.

ttlSecondsAfterFinished specifies the number of seconds after completion that the driver is eligible to be deleted/cleaned. Default is 1 seconds.

applicationArguments
Array of strings

Any application arguments

object (SparkBatchJobRestartSettings)

Optional restart settings. If provided, the job will be retried on failure up to maxRetries times.

applicationName
string

The application name. If not provided a name will be generated.

object

Any Spark configuration/properties to set arguments

jars
Array of strings

Any jars to pass in the --jars Spark argument

pythonFiles
Array of strings

Any python files to pass in the --py-files Spark argument

files
Array of strings

Any files to pass in the --files Spark argument

archives
Array of strings

Any archives to pass in the --archives Spark argument

object

Any environment variables to set

resourcePool
string

Optional - the resource pool to use (you must have permission to use it)

instanceImageId
string

Instance image id to use (you must have permission to use it)

options
Array of strings
Items Value: "EncryptCommunication"

Details:

  • "EncryptCommunication": Enable encryption for communication between the driver and executors
Array of objects

Volumes to mount to the job (read only)

Responses

Request samples

Content type
application/json
{
  • "applicationResource": "string",
  • "mainClass": "string",
  • "cronTimeZone": "string",
  • "cronSchedule": "string",
  • "ttlSecondsAfterFinished": 0,
  • "applicationArguments": [
    ],
  • "restartSettings": {
    },
  • "applicationName": "string",
  • "sparkProperties": {
    },
  • "jars": [
    ],
  • "pythonFiles": [
    ],
  • "files": [
    ],
  • "archives": [
    ],
  • "environmentVariables": {
    },
  • "resourcePool": "string",
  • "instanceImageId": "string",
  • "options": [
    ],
  • "volumeMounts": [
    ]
}

Response samples

Content type
application/json
{
  • "instanceId": "string"
}

connectInstance

Start a new Spark Connect driver

Authorizations:
None
Request Body schema: application/json
required
applicationName
string

The application name. If not provided a name will be generated.

object

Any Spark configuration/properties to set arguments

jars
Array of strings

Any jars to pass in the --jars Spark argument

pythonFiles
Array of strings

Any python files to pass in the --py-files Spark argument

files
Array of strings

Any files to pass in the --files Spark argument

archives
Array of strings

Any archives to pass in the --archives Spark argument

object

Any environment variables to set

resourcePool
string

Optional - the resource pool to use (you must have permission to use it)

instanceImageId
string

Instance image id to use (you must have permission to use it)

options
Array of strings
Items Value: "EncryptCommunication"

Details:

  • "EncryptCommunication": Enable encryption for communication between the driver and executors
Array of objects

Volumes to mount to the job (read only)

Responses

Request samples

Content type
application/json
{
  • "applicationName": "string",
  • "sparkProperties": {
    },
  • "jars": [
    ],
  • "pythonFiles": [
    ],
  • "files": [
    ],
  • "archives": [
    ],
  • "environmentVariables": {
    },
  • "resourcePool": "string",
  • "instanceImageId": "string",
  • "options": [
    ],
  • "volumeMounts": [
    ]
}

Response samples

Content type
application/json
{
  • "instanceId": "string"
}

notebookInstance

Start a new notebook server

Authorizations:
None
Request Body schema: application/json
required
notebookName
string

The notebook name. If not provided a name will be generated.

resourcePool
string

Optional - the resource pool to use (you must have permission to use it)

instanceImageId
string

Instance image id to use (you must have permission to use it)

description
string

Optional - the description of the notebook instance

Array of objects (VolumeMount)

Volumes to mount to the server (read only)

object

Any environment variables to set

Responses

Request samples

Content type
application/json
{
  • "notebookName": "string",
  • "resourcePool": "string",
  • "instanceImageId": "string",
  • "description": "string",
  • "volumeMounts": [
    ],
  • "environmentVariables": {
    }
}

Response samples

Content type
application/json
{
  • "instanceId": "string"
}

createPresignedUrl

Create pre-signed URLs for the given bucket, key and credentials

Authorizations:
None
Request Body schema: application/json
required
accessKey
required
string

The proxy AccessKey provided by your administrator

secretKey
required
string

The proxy SecretKey provided by your administrator

region
required
string

The S3 region of the bucket

bucket
required
string

The bucket for creating the pre-signed URLs

key
required
string

The key for creating the pre-signed URLs

Responses

Request samples

Content type
application/json
{
  • "accessKey": "string",
  • "secretKey": "string",
  • "region": "string",
  • "bucket": "string",
  • "key": "string"
}

Response samples

Content type
application/json
{
  • "presignedUrls": {
    }
}

batch

Shutdown and remove all batch jobs

Authorizations:
None

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

connect

Shutdown and remove all connect servers

Authorizations:
None

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

notebook

Shutdown and remove all notebook servers

Authorizations:
None
query Parameters
preserveUserData
boolean

Preserve user data when deleting the notebook servers

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

listInstance

List all known instances

Authorizations:
None
query Parameters
showAll
boolean

If true, return all instances including deleted ones.

pageToken
string

Pagination token

pageSize
integer <int32>

Page size or 0 for default (current maximum is 100)

Responses

Response samples

Content type
application/json
{
  • "nextPageToken": "string",
  • "result": [
    ]
}

listEvent

List system events

Authorizations:
None

Responses

Response samples

Content type
application/json
[
  • {
    }
]

userInstance

List instances available to the current user

Authorizations:
None
query Parameters
pageToken
string

Pagination token

pageSize
integer <int32>

Page size or 0 for default (current maximum is 100)

Responses

Response samples

Content type
application/json
{
  • "nextPageToken": "string",
  • "result": [
    ]
}

getInstanceLogs

Get the log output of a batch job, a connect driver or a notebook server

Authorizations:
None
path Parameters
instanceId
required
string
  • An instance in the system
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

getLogsWithIndex

Get the log output of an instance. logsId is the log index to return: Some cluster types have multiple nodes/workers. Pass 0 to get the main logs and then increase the index to get other logs.

Authorizations:
None
path Parameters
instanceId
required
string
  • An instance in the system
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)
logsId
required
string

Logs from an instance

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

restartInstance

Restart a notebook server instance

Authorizations:
None
path Parameters
instanceId
required
string
  • An instance in the system
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

resumeInstance

Resume a cron batch job instance

Authorizations:
None
path Parameters
instanceId
required
string
  • An instance in the system
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

shutdownInstance

Shutdown a notebook server instance

Authorizations:
None
path Parameters
instanceId
required
string
  • An instance in the system
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

softdeleteInstance

Marks the instance as deleted and thus not visible on the listings

Authorizations:
None
path Parameters
instanceId
required
string
  • An instance in the system
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

getInstanceStatus

Get the status of an instance

Authorizations:
None
path Parameters
instanceId
required
string
  • An instance in the system
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)
query Parameters
showAll
boolean

If true, return all instances including deleted ones.

Responses

Response samples

Content type
application/json
{
  • "status": {
    }
}

suspendInstance

Suspend a cron batch job instance

Authorizations:
None
path Parameters
instanceId
required
string
  • An instance in the system
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

getLogs

Get the system logs

Authorizations:
None
path Parameters
logsId
required
string

Logs from an instance

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

deleteInstance

Terminate an instance

Authorizations:
None
path Parameters
instanceId
required
string
  • An instance in the system
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)
query Parameters
preserveUserData
boolean

Preserve user data when deleting the notebook server

force
boolean

Use with caution! Immediately deletes instance without waiting for normal shutdown. This will likely result in loss of final logs, events, etc.

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

updateRole

Change the role associated with the token used in this API call

Authorizations:
None
path Parameters
roleId
required
string

A role

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

Resources Service

Resource pool management

getResourcePool

Return the current resource pool set including the total available memory and cores

Authorizations:
None

Responses

Response samples

Content type
application/json
{
  • "totalMemory": "string",
  • "totalCores": "string",
  • "totalGpus": "string",
  • "resourcePools": [
    ]
}

resources.updateResourcePool

Update the set of available resource pools (replace the entire set)

Authorizations:
None
Request Body schema: application/json
required
required
Array of objects (ResourcePool)

The set of resource pools

Responses

Request samples

Content type
application/json
{
  • "resourcePools": [
    ]
}

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

createResourcePool

Create new resource pool

Authorizations:
None
Request Body schema: application/json
required
resourcePoolId
required
string

The name of this resource pool (must be unique)

priority
integer <int32>

The priority of this pool. Pools with larger/higher priority numbers have priority over pools with smaller/lower priority numbers. If not specified, the priority is "0".

minMemory
required
string

Minimum memory (as a Spark quantity string)

minCores
required
string

Minimum virtual cores (as a CPU unit)

maxMemory
required
string

Maximum memory (as a Spark quantity string)

maxCores
required
string

Maximum virtual cores (as a CPU unit)

maxGpus
string

Maximum gpus (as a Spark quantity string)

applicationMaxGpus
string

Maximum gpus per application (as a Spark quantity string)

Responses

Request samples

Content type
application/json
{
  • "resourcePoolId": "string",
  • "priority": 0,
  • "minMemory": "string",
  • "minCores": "string",
  • "maxMemory": "string",
  • "maxCores": "string",
  • "maxGpus": "string",
  • "applicationMaxGpus": "string"
}

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

updateResourcePool

Edit resource pool (update all parameters for a single resource pool)

Authorizations:
None
path Parameters
resourcePoolId
required
string

A resource pool that describes a minimum and maximum resource set of memory and cores

Request Body schema: application/json
required
resourcePoolId
required
string

The name of this resource pool (must be unique)

priority
integer <int32>

The priority of this pool. Pools with larger/higher priority numbers have priority over pools with smaller/lower priority numbers. If not specified, the priority is "0".

minMemory
required
string

Minimum memory (as a Spark quantity string)

minCores
required
string

Minimum virtual cores (as a CPU unit)

maxMemory
required
string

Maximum memory (as a Spark quantity string)

maxCores
required
string

Maximum virtual cores (as a CPU unit)

maxGpus
string

Maximum gpus (as a Spark quantity string)

applicationMaxGpus
string

Maximum gpus per application (as a Spark quantity string)

Responses

Request samples

Content type
application/json
{
  • "resourcePoolId": "string",
  • "priority": 0,
  • "minMemory": "string",
  • "minCores": "string",
  • "maxMemory": "string",
  • "maxCores": "string",
  • "maxGpus": "string",
  • "applicationMaxGpus": "string"
}

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

deleteResourcePool

Delete resource pool

Authorizations:
None
path Parameters
resourcePoolId
required
string

A resource pool that describes a minimum and maximum resource set of memory and cores

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

patchUpdateResourcePool

Update single resource pool parameters

Only include fields you wish to update. Missing or unrecognized fields are ignored.

Authorizations:
None
path Parameters
resourcePoolId
required
string

A resource pool that describes a minimum and maximum resource set of memory and cores

Request Body schema: application/json
required
resourcePoolId
string

The name of this resource pool (must be unique)

priority
integer <int32>

The priority of this pool. Pools with larger/higher priority numbers have priority over pools with smaller/lower priority numbers. If not specified, the priority is "0".

minMemory
string

Minimum memory (as a Spark quantity string)

minCores
string

Minimum virtual cores (as a CPU unit)

maxMemory
string

Maximum memory (as a Spark quantity string)

maxCores
string

Maximum virtual cores (as a CPU unit)

maxGpus
string

Maximum gpus (as a Spark quantity string)

applicationMaxGpus
string

Maximum gpus per application (as a Spark quantity string)

Responses

Request samples

Content type
application/json
{
  • "resourcePoolId": "string",
  • "priority": 0,
  • "minMemory": "string",
  • "minCores": "string",
  • "maxMemory": "string",
  • "maxCores": "string",
  • "maxGpus": "string",
  • "applicationMaxGpus": "string"
}

Response samples

Content type
application/json
{
  • "resourcePoolId": "string",
  • "priority": 0,
  • "minMemory": "string",
  • "minCores": "string",
  • "maxMemory": "string",
  • "maxCores": "string",
  • "maxGpus": "string",
  • "applicationMaxGpus": "string"
}

Admin Service

Administration

getSystemMaintenance

Check if the system maintenance is currently activated

Authorizations:
None

Responses

Response samples

Content type
application/json
{
  • "status": "Deactivated"
}

createSystemMaintenance

Activate system maintenance

Authorizations:
None

Responses

Response samples

Content type
application/json
{
  • "status": "Deactivated"
}

deleteSystemMaintenance

Deactivate system maintenance

Authorizations:
None

Responses

Response samples

Content type
application/json
{
  • "status": "Deactivated"
}

getGlobalStorageUsage

Get global storage usage

Authorizations:
None
query Parameters
moreInstances
boolean

Gets top 1000 instances usage

allResourcePools
boolean

Gets all resource pools usage

Responses

Response samples

Content type
application/json
{
  • "totalUsedBytes": 0,
  • "totalObjectsCount": 0,
  • "logsUsedBytes": 0,
  • "logObjectsCount": 0,
  • "historyServerEventsUsedBytes": 0,
  • "historyServerEventsObjectsCount": 0,
  • "notebookWorkspacesUsedBytes": 0,
  • "notebookWorkspacesObjectsCount": 0,
  • "instanceUsages": [
    ],
  • "resourcePoolUsages": [
    ],
  • "lastRefreshedAt": "2019-08-24T14:15:22Z"
}

statusInstanceLogsCleaner

Get instance logs cleaner status

Authorizations:
None

Responses

Response samples

Content type
application/json
{
  • "running": true,
  • "lastCleanedAt": "2019-08-24T14:15:22Z"
}

getLogRetentionSettings

Get log retention settings

Authorizations:
None

Responses

Response samples

Content type
application/json
{
  • "sparkEventLogsCleanerEnabled": true,
  • "sparkEventLogsCleanerInterval": "string",
  • "sparkEventLogsCleanerMaxAge": "string",
  • "sparkEventLogsCompactionEnabled": true,
  • "sparkEventLogsCompactionMaxFilesToRetain": 0,
  • "instanceLogsCleanerEnabled": true,
  • "instanceLogsCleanerInterval": "string",
  • "instanceLogsCleanerMaxAge": "string"
}

updateLogRetentionSettings

Update log retention settings

Authorizations:
None
Request Body schema: application/json
required
sparkEventLogsCleanerEnabled
boolean

Enable Spark event logs cleaner

sparkEventLogsCleanerInterval
string

Interval for Spark event logs cleaner

sparkEventLogsCleanerMaxAge
string

Max age for Spark event logs to retain

sparkEventLogsCompactionEnabled
boolean

Enable Spark event logs compaction

sparkEventLogsCompactionMaxFilesToRetain
string

Max files to retain after Spark event logs compaction

instanceLogsCleanerEnabled
boolean

Enable instance logs cleaner

instanceLogsCleanerInterval
string

Interval for instance logs cleaner

instanceLogsCleanerMaxAge
string

Max age for instance logs to retain

Responses

Request samples

Content type
application/json
{
  • "sparkEventLogsCleanerEnabled": true,
  • "sparkEventLogsCleanerInterval": "string",
  • "sparkEventLogsCleanerMaxAge": "string",
  • "sparkEventLogsCompactionEnabled": true,
  • "sparkEventLogsCompactionMaxFilesToRetain": "string",
  • "instanceLogsCleanerEnabled": true,
  • "instanceLogsCleanerInterval": "string",
  • "instanceLogsCleanerMaxAge": "string"
}

Response samples

Content type
application/json
{
  • "sparkEventLogsCleanerEnabled": true,
  • "sparkEventLogsCleanerInterval": "string",
  • "sparkEventLogsCleanerMaxAge": "string",
  • "sparkEventLogsCompactionEnabled": true,
  • "sparkEventLogsCompactionMaxFilesToRetain": 0,
  • "instanceLogsCleanerEnabled": true,
  • "instanceLogsCleanerInterval": "string",
  • "instanceLogsCleanerMaxAge": "string"
}

deleteLogRetentionSettings

Reset log retention settings to defaults

Authorizations:
None

Responses

Response samples

Content type
application/json
{
  • "sparkEventLogsCleanerEnabled": true,
  • "sparkEventLogsCleanerInterval": "string",
  • "sparkEventLogsCleanerMaxAge": "string",
  • "sparkEventLogsCompactionEnabled": true,
  • "sparkEventLogsCompactionMaxFilesToRetain": 0,
  • "instanceLogsCleanerEnabled": true,
  • "instanceLogsCleanerInterval": "string",
  • "instanceLogsCleanerMaxAge": "string"
}

refreshGlobalStorageUsage

Refresh global storage usage

Authorizations:
None

Responses

Response samples

Content type
application/json
null

runInstanceLogsCleaner

Run instance logs cleaner

Authorizations:
None

Responses

Response samples

Content type
application/json
null

stopInstanceLogsCleaner

Stop instance logs cleaner

Authorizations:
None

Responses

Response samples

Content type
application/json
null

listSystemMaintenanceHistory

Show system maintenance history

Authorizations:
None

Responses

Response samples

Content type
application/json
[
  • {
    }
]

InstanceImageImports Service

Instance image imports management

listInstanceImageImport

Get all instance image imports

Authorizations:
None
query Parameters
pageToken
string

Pagination token

pageSize
integer <int32>

Page size or 0 for default (current maximum is 100)

orderBy
string

Sort/ordering in standard SQL ORDER BY format. Allowed values: id, createdAt

Responses

Response samples

Content type
application/json
{
  • "nextPageToken": "string",
  • "result": [
    ]
}

createInstanceImageImport

Submit and run a instance image import

Authorizations:
None
Request Body schema: application/json
required
name
required
string

Image name

repository
required
string

Registry with repository name to import

tag
required
string

Image tag to import

description
string

Optional image description

allowInsecure
required
boolean

Allow insecure registry connection

isPrivate
required
boolean

Registry require credentials for accessing

object (Credentials)

Credentials required for accessing registry

Responses

Request samples

Content type
application/json
{
  • "name": "string",
  • "repository": "string",
  • "tag": "string",
  • "description": "string",
  • "allowInsecure": true,
  • "isPrivate": true,
  • "credentials": {
    }
}

Response samples

Content type
application/json
{
  • "instanceImageImportId": "string"
}

getInstanceImageImport

Get image import details

Authorizations:
None
path Parameters
instanceImageImportId
required
string

An instance image import

Responses

Response samples

Content type
application/json
{
  • "id": "string",
  • "name": "string",
  • "repository": "string",
  • "tag": "string",
  • "description": "string",
  • "allowInsecure": true,
  • "isPrivate": true,
  • "status": "New",
  • "statusDescription": "string",
  • "createdBy": "string",
  • "createdAt": "2019-08-24T14:15:22Z",
  • "updatedAt": "2019-08-24T14:15:22Z"
}

InstanceImages Service

Instance image management

getInstanceImage

Get instance image details

Authorizations:
None
path Parameters
instanceImageId
required
string

An instance image

Responses

Response samples

Content type
application/json
{
  • "id": "string",
  • "name": "string",
  • "tag": "string",
  • "description": "string",
  • "origin": {
    },
  • "baseImageId": "string",
  • "digest": "string",
  • "type": {
    },
  • "status": "Init",
  • "statusDescription": "string",
  • "createdBy": "string",
  • "createdAt": "2019-08-24T14:15:22Z",
  • "updatedAt": "2019-08-24T14:15:22Z",
  • "layers": [
    ],
  • "history": [
    ]
}

listInstanceImage

Get all instance image

Authorizations:
None
query Parameters
pageToken
string

Pagination token

pageSize
integer <int32>

Page size or 0 for default (current maximum is 100)

orderBy
string

Sort/ordering in standard SQL ORDER BY format. Allowed values: id, createdAt

Responses

Response samples

Content type
application/json
{
  • "nextPageToken": "string",
  • "result": [
    ]
}

userlistInstanceImage

Get available instance images for users

Authorizations:
None
query Parameters
pageToken
string

Pagination token

pageSize
integer <int32>

Page size or 0 for default (current maximum is 100)

orderBy
string

Sort/ordering in standard SQL ORDER BY format. Allowed values: id, createdAt

Responses

Response samples

Content type
application/json
{
  • "nextPageToken": "string",
  • "result": [
    ]
}

updateInstanceImageInstanceImageStatus

Update instance image status

Authorizations:
None
path Parameters
instanceImageId
required
string

An instance image

Request Body schema: application/json
required
status
required
string
Enum: "Init" "Pending" "Verified" "Rejected" "MarkedForDelete" "Deleted"

Image status

statusDescription
string

Status description

Responses

Request samples

Content type
application/json
{
  • "status": "Init",
  • "statusDescription": "string"
}

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

Uploads Service

File upload management

activateUpload

Mark the upload for activation. Once activation has completed it can be used by your workloads.

Authorizations:
None
path Parameters
uploadId
required
string
  • A file upload. Uploads can be a single file or a collection of files.
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

listUploads

List all of your uploads

Authorizations:
None
query Parameters
pageToken
string

Pagination token

pageSize
integer <int32>

Page size or 0 for default (current maximum is 100)

kind
string

Filter by upload kind

Responses

Response samples

Content type
application/json
{
  • "nextPageToken": "string",
  • "result": [
    ]
}

createUpload

Create a new upload. If an upload with the given name already exists and is still available for uploads, the existing upload will be returned.

Authorizations:
None
Request Body schema: application/json
required
name
required
string

Name for the upload. Must be unique in the system.

description
string

Optional description for the upload.

kind
required
string
Enum: "Inline" "Standalone"

The kind of upload upload.

  • "Inline": An in-band upload created for a single job run, bound to a specific instance
  • "Standalone": An out-of-band upload created independently, reusable across multiple job runs

Responses

Request samples

Content type
application/json
{
  • "name": "string",
  • "description": "string",
  • "kind": "Inline"
}

Response samples

Content type
application/json
{
  • "uploadId": "string",
  • "name": "string"
}

createUploadItemPresignedUrl

Generate a pre-signed URI to upload an item to the upload or read it

Authorizations:
None
path Parameters
uploadId
required
string
  • A file upload. Uploads can be a single file or a collection of files.
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)
query Parameters
forReading
boolean

If true, returns a URI for reading an item instead of for writing an item

Request Body schema: application/json
required
key
required
string

The path of the item.

Responses

Request samples

Content type
application/json
{
  • "key": "string"
}

Response samples

Content type
application/json
{
  • "uri": "string"
}

deactivateUpload

Mark the upload for deactivation. Once any workloads using it have completed it will be set to inactive.

Authorizations:
None
path Parameters
uploadId
required
string
  • A file upload. Uploads can be a single file or a collection of files.
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

getUpload

Get the details of an upload

Authorizations:
None
path Parameters
uploadId
required
string
  • A file upload. Uploads can be a single file or a collection of files.
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)

Responses

Response samples

Content type
application/json
{
  • "uploadId": "string",
  • "name": "string",
  • "description": "string",
  • "createdBy": "string",
  • "createdAt": "2019-08-24T14:15:22Z",
  • "state": "New",
  • "sealed": true,
  • "markedForDeletion": true
}

finalizeUpload

Finalize an upload. It will be available for usage and cannot be added to.

Authorizations:
None
path Parameters
uploadId
required
string
  • A file upload. Uploads can be a single file or a collection of files.
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

deleteUpload

Mark the upload for deletion. It and its contents will be deleted shortly after all uses of it have completed.

Authorizations:
None
path Parameters
uploadId
required
string
  • A file upload. Uploads can be a single file or a collection of files.
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

updateUpload

Change the name or description of an upload

Only include fields you wish to update. Missing or unrecognized fields are ignored.

Authorizations:
None
path Parameters
uploadId
required
string
  • A file upload. Uploads can be a single file or a collection of files.
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)
Request Body schema: application/json
required
name
string

Name for the upload. Must be unique in the system.

description
string

Optional description for the upload.

Responses

Request samples

Content type
application/json
{
  • "name": "string",
  • "description": "string"
}

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

deleteUploadItem

Delete an item from an upload

Authorizations:
None
path Parameters
uploadId
required
string
  • A file upload. Uploads can be a single file or a collection of files.
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

listUploadItems

Get the item details of an upload

Authorizations:
None
path Parameters
uploadId
required
string
  • A file upload. Uploads can be a single file or a collection of files.
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)
query Parameters
pageToken
string

Pagination token

pageSize
integer <int32>

Page size or 0 for default (current maximum is 100)

Responses

Response samples

Content type
application/json
{
  • "nextPageToken": "string",
  • "result": [
    ]
}

finalizeUploadItem

Finalize the upload for the given item

Authorizations:
None
path Parameters
uploadId
required
string
  • A file upload. Uploads can be a single file or a collection of files.
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)
Request Body schema: application/json
required
key
required
string

The path of the item.

Responses

Request samples

Content type
application/json
{
  • "key": "string"
}

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

listUploadUsages

List all instances using the given upload

Authorizations:
None
path Parameters
uploadId
required
string
  • A file upload. Uploads can be a single file or a collection of files.
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)

Responses

Response samples

Content type
application/json
[
  • {
    }
]

pingUpload

While uploading files to a newly created upload, you must periodically call this method to inform the system that the upload is still in progress. Failing to do so may result in the upload being marked as failed and deleted.

Authorizations:
None
path Parameters
uploadId
required
string
  • A file upload. Uploads can be a single file or a collection of files.
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)

Responses

Response samples

Content type
application/json
{
  • "uploadId": "string"
}

renameUploadItem

Rename an item in an upload

Authorizations:
None
path Parameters
uploadId
required
string
  • A file upload. Uploads can be a single file or a collection of files.
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)
Request Body schema: application/json
required
key
required
string

The path of the item.

newKey
required
string

The new path of the item.

Responses

Request samples

Content type
application/json
{
  • "key": "string",
  • "newKey": "string"
}

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

sealUpload

Mark this upload as permanently sealed. No further items can be added to it or removed from it once it is finalized.

Authorizations:
None
path Parameters
uploadId
required
string
  • A file upload. Uploads can be a single file or a collection of files.
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

Secrets Service

Secrets management

listSecrets

List all secrets

Authorizations:
None
query Parameters
pageToken
string

Pagination token

pageSize
integer <int32>

Page size or 0 for default (current maximum is 100)

orderBy
string

Sort/ordering in standard SQL ORDER BY format. Allowed values: name, description, createdAt, createdBy, provider, kind

Responses

Response samples

Content type
application/json
{
  • "nextPageToken": "string",
  • "result": [
    ]
}

createSecret

Create a new secret

Authorizations:
None
Request Body schema: application/json
required
kind
required
string
name
required
string

The secret name

description
string

The secret description

required
object

The key-value secret data

Responses

Request samples

Content type
application/json
Example
{
  • "kind": "internal",
  • "name": "string",
  • "description": "string",
  • "secretData": {
    }
}

Response samples

Content type
application/json
Example
{
  • "kind": "internal",
  • "id": "string",
  • "name": "string",
  • "description": "string",
  • "createdAt": "2019-08-24T14:15:22Z",
  • "createdBy": "string"
}

getSecret

Get secret by ID

Authorizations:
None
path Parameters
secretId
required
string
  • A managed secret
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)
query Parameters
reveal
boolean

Show actual secret values instead of redacted placeholders

Responses

Response samples

Content type
application/json
Example
{
  • "kind": "internal",
  • "id": "string",
  • "name": "string",
  • "description": "string",
  • "createdAt": "2019-08-24T14:15:22Z",
  • "createdBy": "string",
  • "secretData": {
    }
}

updateSecret

Update a secret

Authorizations:
None
path Parameters
secretId
required
string
  • A managed secret
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)
Request Body schema: application/json
required
kind
required
string
name
required
string

The secret name

description
string

The secret description

required
object

The key-value secret data

Responses

Request samples

Content type
application/json
Example
{
  • "kind": "internal",
  • "name": "string",
  • "description": "string",
  • "secretData": {
    }
}

Response samples

Content type
application/json
Example
{
  • "kind": "internal",
  • "id": "string",
  • "name": "string",
  • "description": "string",
  • "createdAt": "2019-08-24T14:15:22Z",
  • "createdBy": "string"
}

deleteSecret

Delete a secret

Authorizations:
None
path Parameters
secretId
required
string
  • A managed secret
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)

Responses

Response samples

Content type
application/json
{
  • "message": "string",
  • "fields": [
    ]
}

patchUpdateSecret

Patch a secret

Only include fields you wish to update. Missing or unrecognized fields are ignored.

Authorizations:
None
path Parameters
secretId
required
string
  • A managed secret
  • This parameter can be looked up using name instead of its Id. Use name=value instead of an Id to lookup/search using the value. value must be encoded (see RFC including =)
Request Body schema: application/json
required
kind
required
string
name
string

The secret name

description
string

The secret description

object

The key-value secret data

Responses

Request samples

Content type
application/json
Example
{
  • "kind": "internal",
  • "name": "string",
  • "description": "string",
  • "secretData": {
    }
}

Response samples

Content type
application/json
Example
{
  • "kind": "internal",
  • "id": "string",
  • "name": "string",
  • "description": "string",
  • "createdAt": "2019-08-24T14:15:22Z",
  • "createdBy": "string"
}

listSecretProvider

List external secret providers

Authorizations:
None

Responses

Response samples

Content type
application/json
[
  • {
    }
]

Model Definitions

BasicSecret

kind
required
string
id
required
string

The secret ID

name
required
string

The secret name

description
string

The secret description

createdAt
required
string <date-time>

The creation timestamp

createdBy
required
string

The creator user ID

Example
{
  • "kind": "internal",
  • "id": "string",
  • "name": "string",
  • "description": "string",
  • "createdAt": "2019-08-24T14:15:22Z",
  • "createdBy": "string"
}

BasicSecretExternal

kind
required
string
id
required
string

The secret ID

name
required
string

The secret name

description
string

The secret description

createdAt
required
string <date-time>

The creation timestamp

createdBy
required
string

The creator user ID

Example
{
  • "kind": "internal",
  • "id": "string",
  • "name": "string",
  • "description": "string",
  • "createdAt": "2019-08-24T14:15:22Z",
  • "createdBy": "string"
}

BasicSecretInternal

kind
required
string
id
required
string

The secret ID

name
required
string

The secret name

description
string

The secret description

createdAt
required
string <date-time>

The creation timestamp

createdBy
required
string

The creator user ID

Example
{
  • "kind": "internal",
  • "id": "string",
  • "name": "string",
  • "description": "string",
  • "createdAt": "2019-08-24T14:15:22Z",
  • "createdBy": "string"
}

CreatedUpload

uploadId
required
string

Upload ID

name
required
string

The name of the upload

{
  • "uploadId": "string",
  • "name": "string"
}

Credentials

username
required
string

Username

password
required
string

Password

{
  • "username": "string",
  • "password": "string"
}

EventV2

time
required
string

Time of the event

type
required
string

Event type

reason
required
string

Event reason

name
required
string

Event name

action
required
string

Event action

{
  • "time": "string",
  • "type": "string",
  • "reason": "string",
  • "name": "string",
  • "action": "string"
}

External

name
string

The secret name

description
string

The secret description

provider
string

The secret provider name

path
string

The secret path

key
string

The secret key

{
  • "name": "string",
  • "description": "string",
  • "provider": "string",
  • "path": "string",
  • "key": "string"
}

GlobalStorageUsage

totalUsedBytes
required
integer <int64>

Total storage used (bytes)

totalObjectsCount
required
integer <int64>

Total number of objects stored

logsUsedBytes
required
integer <int64>

Storage used by logs (bytes)

logObjectsCount
required
integer <int64>

Number of log objects

historyServerEventsUsedBytes
required
integer <int64>

Storage used by history server events (bytes)

historyServerEventsObjectsCount
required
integer <int64>

Number of objects stored by history server events

notebookWorkspacesUsedBytes
required
integer <int64>

Storage used by notebook workspaces (bytes)

notebookWorkspacesObjectsCount
required
integer <int64>

Number of objects stored in notebook workspaces

required
Array of objects (InstanceStorageUsage)

Storage usage details per instance

required
Array of objects (ResourcePoolStorageUsage)

Storage usage details per resource pool

lastRefreshedAt
required
string <date-time>

Last refreshed at

{
  • "totalUsedBytes": 0,
  • "totalObjectsCount": 0,
  • "logsUsedBytes": 0,
  • "logObjectsCount": 0,
  • "historyServerEventsUsedBytes": 0,
  • "historyServerEventsObjectsCount": 0,
  • "notebookWorkspacesUsedBytes": 0,
  • "notebookWorkspacesObjectsCount": 0,
  • "instanceUsages": [
    ],
  • "resourcePoolUsages": [
    ],
  • "lastRefreshedAt": "2019-08-24T14:15:22Z"
}

Instance

instanceId
required
string

The instance Id

{
  • "instanceId": "string"
}

InstanceHistoryInfo

created
required
string <date-time>

Created

command
required
string

Command

comment
required
string

Comment

emptyLayer
required
boolean

Is layer empty

{
  • "created": "2019-08-24T14:15:22Z",
  • "command": "string",
  • "comment": "string",
  • "emptyLayer": true
}

InstanceImageDetailsInfo

id
required
string

The instance image Id

name
required
string

Imported image name

tag
required
string

Image tag

description
string

Optional image description

required
object

An instance image origin

baseImageId
string

Image Id of base image

digest
required
string

Image digest

required
object

Image type

status
required
string
Enum: "Init" "Pending" "Verified" "Rejected" "MarkedForDelete" "Deleted"

Image status

statusDescription
string

Status description

createdBy
required
string

User that created instance image

createdAt
required
string <date-time>

Instance creation timestamp

updatedAt
required
string <date-time>

Instance last update timestamp

required
Array of objects (InstanceLayerInfo)

Layers

required
Array of objects (InstanceHistoryInfo)

History

{
  • "id": "string",
  • "name": "string",
  • "tag": "string",
  • "description": "string",
  • "origin": {
    },
  • "baseImageId": "string",
  • "digest": "string",
  • "type": {
    },
  • "status": "Init",
  • "statusDescription": "string",
  • "createdBy": "string",
  • "createdAt": "2019-08-24T14:15:22Z",
  • "updatedAt": "2019-08-24T14:15:22Z",
  • "layers": [
    ],
  • "history": [
    ]
}

InstanceImageImport

instanceImageImportId
required
string

The image import Id

{
  • "instanceImageImportId": "string"
}

InstanceImageImportCreate

name
required
string

Image name

repository
required
string

Registry with repository name to import

tag
required
string

Image tag to import

description
string

Optional image description

allowInsecure
required
boolean

Allow insecure registry connection

isPrivate
required
boolean

Registry require credentials for accessing

object (Credentials)

Credentials required for accessing registry

{
  • "name": "string",
  • "repository": "string",
  • "tag": "string",
  • "description": "string",
  • "allowInsecure": true,
  • "isPrivate": true,
  • "credentials": {
    }
}

InstanceImageImportInfo

id
required
string

The instance image Id

name
required
string

Imported image name

repository
required
string

Registry with repository name to import

tag
required
string

Image tag to import

description
string

Optional image description

allowInsecure
required
boolean

Allow insecure registry connection

isPrivate
required
boolean

Registry require credentials for accessing

status
required
string
Enum: "New" "InProgress" "Completed" "Failed"

Instance image import status

statusDescription
string

Optional status description

createdBy
required
string

User that created instance image import

createdAt
required
string <date-time>

Instance import creation timestamp

updatedAt
required
string <date-time>

Instance import last update timestamp

{
  • "id": "string",
  • "name": "string",
  • "repository": "string",
  • "tag": "string",
  • "description": "string",
  • "allowInsecure": true,
  • "isPrivate": true,
  • "status": "New",
  • "statusDescription": "string",
  • "createdBy": "string",
  • "createdAt": "2019-08-24T14:15:22Z",
  • "updatedAt": "2019-08-24T14:15:22Z"
}

InstanceImageInfo

id
required
string

The instance image Id

name
required
string

Imported image name

tag
required
string

Image tag

description
string

Optional image description

required
object (InstanceImageOriginInfo)

An instance image origin

baseImageId
string

Image Id of base image

digest
required
string

Image digest

required
object (InstanceImageTypeInfo)

Image type

status
required
string
Enum: "Init" "Pending" "Verified" "Rejected" "MarkedForDelete" "Deleted"

Image status

statusDescription
string

Status description

createdBy
required
string

User that created instance image

createdAt
required
string <date-time>

Instance creation timestamp

updatedAt
required
string <date-time>

Instance last update timestamp

{
  • "id": "string",
  • "name": "string",
  • "tag": "string",
  • "description": "string",
  • "origin": {
    },
  • "baseImageId": "string",
  • "digest": "string",
  • "type": {
    },
  • "status": "Init",
  • "statusDescription": "string",
  • "createdBy": "string",
  • "createdAt": "2019-08-24T14:15:22Z",
  • "updatedAt": "2019-08-24T14:15:22Z"
}

InstanceImageOriginInfo

repository
required
string

Origin repository

tag
required
string

Origin tag

isPrivate
required
boolean

Origin required credentials

taskId
string

Origin import task id

{
  • "repository": "string",
  • "tag": "string",
  • "isPrivate": true,
  • "taskId": "string"
}

InstanceImageStatus

status
required
string
Enum: "Init" "Pending" "Verified" "Rejected" "MarkedForDelete" "Deleted"

Image status

statusDescription
string

Status description

{
  • "status": "Init",
  • "statusDescription": "string"
}

InstanceImageTypeInfo

kind
required
string
Enum: "Spark" "Notebook"

An instance image type

version
required
string

An instance image version

{
  • "kind": "Spark",
  • "version": "string"
}

InstanceInfo

instanceId
required
string

The instance Id

name
required
string

The instance name

type
required
string

The instance type

createdBy
required
string

User that created the instance

ownerRole
string

Role that owns the instance

required
object

Any additional details about the instance

{
  • "instanceId": "string",
  • "name": "string",
  • "type": "string",
  • "createdBy": "string",
  • "ownerRole": "string",
  • "details": {
    }
}

InstanceLayerInfo

mime
required
string

Mime type

digest
required
string

Digest

size
required
integer <int64>

Size

annotations
string

Annotations

{
  • "mime": "string",
  • "digest": "string",
  • "size": 0,
  • "annotations": "string"
}

InstanceLogsCleanerStatus

running
required
boolean

Instance log cleaning is being run

lastCleanedAt
string <date-time>

Last cleaned at

{
  • "running": true,
  • "lastCleanedAt": "2019-08-24T14:15:22Z"
}

InstanceStorageUsage

instanceId
required
string

The instance id

usedBytes
required
integer <int64>

Storage bytes used by this instance

objectsCount
required
integer <int64>

Number of objects stored by this instance

{
  • "instanceId": "string",
  • "usedBytes": 0,
  • "objectsCount": 0
}

Internal

name
string

The secret name

description
string

The secret description

object

The key-value secret data

{
  • "name": "string",
  • "description": "string",
  • "secretData": {
    }
}

LogRetentionSettings

sparkEventLogsCleanerEnabled
required
boolean

Enable Spark event logs cleaner

sparkEventLogsCleanerInterval
required
string

Interval for Spark event logs cleaner

sparkEventLogsCleanerMaxAge
required
string

Max age for Spark event logs to retain

sparkEventLogsCompactionEnabled
required
boolean

Enable Spark event logs compaction

sparkEventLogsCompactionMaxFilesToRetain
required
integer <int32>

Max files to retain after Spark event logs compaction

instanceLogsCleanerEnabled
required
boolean

Enable instance logs cleaner

instanceLogsCleanerInterval
required
string

Interval for instance logs cleaner

instanceLogsCleanerMaxAge
required
string

Max age for instance logs to retain

{
  • "sparkEventLogsCleanerEnabled": true,
  • "sparkEventLogsCleanerInterval": "string",
  • "sparkEventLogsCleanerMaxAge": "string",
  • "sparkEventLogsCompactionEnabled": true,
  • "sparkEventLogsCompactionMaxFilesToRetain": 0,
  • "instanceLogsCleanerEnabled": true,
  • "instanceLogsCleanerInterval": "string",
  • "instanceLogsCleanerMaxAge": "string"
}

LogRetentionSettingsUpdate

sparkEventLogsCleanerEnabled
boolean

Enable Spark event logs cleaner

sparkEventLogsCleanerInterval
string

Interval for Spark event logs cleaner

sparkEventLogsCleanerMaxAge
string

Max age for Spark event logs to retain

sparkEventLogsCompactionEnabled
boolean

Enable Spark event logs compaction

sparkEventLogsCompactionMaxFilesToRetain
string

Max files to retain after Spark event logs compaction

instanceLogsCleanerEnabled
boolean

Enable instance logs cleaner

instanceLogsCleanerInterval
string

Interval for instance logs cleaner

instanceLogsCleanerMaxAge
string

Max age for instance logs to retain

{
  • "sparkEventLogsCleanerEnabled": true,
  • "sparkEventLogsCleanerInterval": "string",
  • "sparkEventLogsCleanerMaxAge": "string",
  • "sparkEventLogsCompactionEnabled": true,
  • "sparkEventLogsCompactionMaxFilesToRetain": "string",
  • "instanceLogsCleanerEnabled": true,
  • "instanceLogsCleanerInterval": "string",
  • "instanceLogsCleanerMaxAge": "string"
}

NewSecret

kind
required
string
name
required
string

The secret name

description
string

The secret description

required
object

The key-value secret data

Example
{
  • "kind": "internal",
  • "name": "string",
  • "description": "string",
  • "secretData": {
    }
}

NewSecretExternal

kind
required
string
name
required
string

The secret name

description
string

The secret description

required
object

The key-value secret data

Example
{
  • "kind": "internal",
  • "name": "string",
  • "description": "string",
  • "secretData": {
    }
}

NewSecretInternal

kind
required
string
name
required
string

The secret name

description
string

The secret description

required
object

The key-value secret data

Example
{
  • "kind": "internal",
  • "name": "string",
  • "description": "string",
  • "secretData": {
    }
}

NewSecretPatchPatch

kind
required
string
name
string

The secret name

description
string

The secret description

object

The key-value secret data

Example
{
  • "kind": "internal",
  • "name": "string",
  • "description": "string",
  • "secretData": {
    }
}

NewSecretPatchPatchExternalPatch

kind
required
string
name
string

The secret name

description
string

The secret description

object

The key-value secret data

Example
{
  • "kind": "internal",
  • "name": "string",
  • "description": "string",
  • "secretData": {
    }
}

NewSecretPatchPatchInternalPatch

kind
required
string
name
string

The secret name

description
string

The secret description

object

The key-value secret data

Example
{
  • "kind": "internal",
  • "name": "string",
  • "description": "string",
  • "secretData": {
    }
}

NewUpload

name
required
string

Name for the upload. Must be unique in the system.

description
string

Optional description for the upload.

kind
required
string
Enum: "Inline" "Standalone"

The kind of upload upload.

  • "Inline": An in-band upload created for a single job run, bound to a specific instance
  • "Standalone": An out-of-band upload created independently, reusable across multiple job runs
{
  • "name": "string",
  • "description": "string",
  • "kind": "Inline"
}

NotebookInstance

notebookName
string

The notebook name. If not provided a name will be generated.

resourcePool
string

Optional - the resource pool to use (you must have permission to use it)

instanceImageId
string

Instance image id to use (you must have permission to use it)

description
string

Optional - the description of the notebook instance

Array of objects (VolumeMount)

Volumes to mount to the server (read only)

object

Any environment variables to set

{
  • "notebookName": "string",
  • "resourcePool": "string",
  • "instanceImageId": "string",
  • "description": "string",
  • "volumeMounts": [
    ],
  • "environmentVariables": {
    }
}

PaginatedBasicSecret

nextPageToken
string

The next page token to use or "" if there are no more pages.

Array of objects (BasicSecret)

A page of results.

{
  • "nextPageToken": "string",
  • "result": [
    ]
}

PaginatedInstanceImageImportInfo

nextPageToken
string

The next page token to use or "" if there are no more pages.

Array of objects (InstanceImageImportInfo)

A page of results.

{
  • "nextPageToken": "string",
  • "result": [
    ]
}

PaginatedInstanceImageInfo

nextPageToken
string

The next page token to use or "" if there are no more pages.

Array of objects (InstanceImageInfo)

A page of results.

{
  • "nextPageToken": "string",
  • "result": [
    ]
}

PaginatedInstanceInfo

nextPageToken
string

The next page token to use or "" if there are no more pages.

Array of objects (InstanceInfo)

A page of results.

{
  • "nextPageToken": "string",
  • "result": [
    ]
}

PaginatedUploadBase

nextPageToken
string

The next page token to use or "" if there are no more pages.

Array of objects (UploadBase)

A page of results.

{
  • "nextPageToken": "string",
  • "result": [
    ]
}

PaginatedUploadItem

nextPageToken
string

The next page token to use or "" if there are no more pages.

Array of objects (UploadItem)

A page of results.

{
  • "nextPageToken": "string",
  • "result": [
    ]
}

PaginatedUserInstanceImageInfo

nextPageToken
string

The next page token to use or "" if there are no more pages.

Array of objects (UserInstanceImageInfo)

A page of results.

{
  • "nextPageToken": "string",
  • "result": [
    ]
}

PresignedUrl

required
object

The pre-signed URLs. The key is an HTTP verb (GET, PUT, POST, DELETE). The value is the pre-signed URL

{
  • "presignedUrls": {
    }
}

PresignedUrlRequest

accessKey
required
string

The proxy AccessKey provided by your administrator

secretKey
required
string

The proxy SecretKey provided by your administrator

region
required
string

The S3 region of the bucket

bucket
required
string

The bucket for creating the pre-signed URLs

key
required
string

The key for creating the pre-signed URLs

{
  • "accessKey": "string",
  • "secretKey": "string",
  • "region": "string",
  • "bucket": "string",
  • "key": "string"
}

RefreshGlobalStorageUsageResponse

any (RefreshGlobalStorageUsageResponse)

Refresh global storage usage response

null

RenameUploadItemName

key
required
string

The path of the item.

newKey
required
string

The new path of the item.

{
  • "key": "string",
  • "newKey": "string"
}

ResourcePool

resourcePoolId
required
string

The name of this resource pool (must be unique)

priority
integer <int32>

The priority of this pool. Pools with larger/higher priority numbers have priority over pools with smaller/lower priority numbers. If not specified, the priority is "0".

minMemory
required
string

Minimum memory (as a Spark quantity string)

minCores
required
string

Minimum virtual cores (as a CPU unit)

maxMemory
required
string

Maximum memory (as a Spark quantity string)

maxCores
required
string

Maximum virtual cores (as a CPU unit)

maxGpus
string

Maximum gpus (as a Spark quantity string)

applicationMaxGpus
string

Maximum gpus per application (as a Spark quantity string)

{
  • "resourcePoolId": "string",
  • "priority": 0,
  • "minMemory": "string",
  • "minCores": "string",
  • "maxMemory": "string",
  • "maxCores": "string",
  • "maxGpus": "string",
  • "applicationMaxGpus": "string"
}

ResourcePool2

resourcePoolId
required
string

The name of this resource pool (must be unique)

priority
integer <int32>

The priority of this pool. Pools with larger/higher priority numbers have priority over pools with smaller/lower priority numbers. If not specified, the priority is "0".

minMemory
required
string

Minimum memory (as a Spark quantity string)

minCores
required
string

Minimum virtual cores (as a CPU unit)

maxMemory
required
string

Maximum memory (as a Spark quantity string)

maxCores
required
string

Maximum virtual cores (as a CPU unit)

maxGpus
string

Maximum gpus (as a Spark quantity string)

applicationMaxGpus
string

Maximum gpus per application (as a Spark quantity string)

{
  • "resourcePoolId": "string",
  • "priority": 0,
  • "minMemory": "string",
  • "minCores": "string",
  • "maxMemory": "string",
  • "maxCores": "string",
  • "maxGpus": "string",
  • "applicationMaxGpus": "string"
}

ResourcePoolPatch

resourcePoolId
string

The name of this resource pool (must be unique)

priority
integer <int32>

The priority of this pool. Pools with larger/higher priority numbers have priority over pools with smaller/lower priority numbers. If not specified, the priority is "0".

minMemory
string

Minimum memory (as a Spark quantity string)

minCores
string

Minimum virtual cores (as a CPU unit)

maxMemory
string

Maximum memory (as a Spark quantity string)

maxCores
string

Maximum virtual cores (as a CPU unit)

maxGpus
string

Maximum gpus (as a Spark quantity string)

applicationMaxGpus
string

Maximum gpus per application (as a Spark quantity string)

{
  • "resourcePoolId": "string",
  • "priority": 0,
  • "minMemory": "string",
  • "minCores": "string",
  • "maxMemory": "string",
  • "maxCores": "string",
  • "maxGpus": "string",
  • "applicationMaxGpus": "string"
}

ResourcePoolSet

required
Array of objects (ResourcePool)

The set of resource pools

{
  • "resourcePools": [
    ]
}

ResourcePoolStorageUsage

resourcePoolId
required
string

The name of the resource pool

usedBytes
required
integer <int64>

Storage bytes used by this resource pool

objectsCount
required
integer <int64>

Number of objects stored by this resource pool

{
  • "resourcePoolId": "string",
  • "usedBytes": 0,
  • "objectsCount": 0
}

ResourcePoolsInfo

totalMemory
required
string

Total available memory (as a Spark quantity string)

totalCores
required
string

Total available virtual cores

totalGpus
required
string

Total available gpus (as quantity string)

required
Array of objects

The set of resource pools

{
  • "totalMemory": "string",
  • "totalCores": "string",
  • "totalGpus": "string",
  • "resourcePools": [
    ]
}

RunInstanceLogsCleaner

any (RunInstanceLogsCleaner)

Manage instance logs cleaner

null

Secret

kind
required
string
id
required
string

The secret ID

name
required
string

The secret name

description
string

The secret description

createdAt
required
string <date-time>

The creation timestamp

createdBy
required
string

The creator user ID

required
object

The key-value secret data

Example
{
  • "kind": "internal",
  • "id": "string",
  • "name": "string",
  • "description": "string",
  • "createdAt": "2019-08-24T14:15:22Z",
  • "createdBy": "string",
  • "secretData": {
    }
}

SecretExternal

kind
required
string
id
required
string

The secret ID

name
required
string

The secret name

description
string

The secret description

createdAt
required
string <date-time>

The creation timestamp

createdBy
required
string

The creator user ID

required
object

The key-value secret data

Example
{
  • "kind": "internal",
  • "id": "string",
  • "name": "string",
  • "description": "string",
  • "createdAt": "2019-08-24T14:15:22Z",
  • "createdBy": "string",
  • "secretData": {
    }
}

SecretInternal

kind
required
string
id
required
string

The secret ID

name
required
string

The secret name

description
string

The secret description

createdAt
required
string <date-time>

The creation timestamp

createdBy
required
string

The creator user ID

required
object

The key-value secret data

Example
{
  • "kind": "internal",
  • "id": "string",
  • "name": "string",
  • "description": "string",
  • "createdAt": "2019-08-24T14:15:22Z",
  • "createdBy": "string",
  • "secretData": {
    }
}

SecretProvider

name
required
string

The secret provider name

{
  • "name": "string"
}

SparkBatchJob

applicationResource
required
string

The application resource to run - must be on an accessible object store

mainClass
string

The main class of the batch job

cronTimeZone
string

Optional CRON timezone. If provided, specify a time zone for the CRON job. Default is UTC.

See List of Time Zones.

cronSchedule
string

Optional CRON schedule. If provided, runs the job on the given schedule.

See Wikipedia's CRON article for details on CRON schedules.

ttlSecondsAfterFinished
integer <int32>

Optional.

ttlSecondsAfterFinished specifies the number of seconds after completion that the driver is eligible to be deleted/cleaned. Default is 1 seconds.

applicationArguments
Array of strings

Any application arguments

object (SparkBatchJobRestartSettings)

Optional restart settings. If provided, the job will be retried on failure up to maxRetries times.

applicationName
string

The application name. If not provided a name will be generated.

object

Any Spark configuration/properties to set arguments

jars
Array of strings

Any jars to pass in the --jars Spark argument

pythonFiles
Array of strings

Any python files to pass in the --py-files Spark argument

files
Array of strings

Any files to pass in the --files Spark argument

archives
Array of strings

Any archives to pass in the --archives Spark argument

object

Any environment variables to set

resourcePool
string

Optional - the resource pool to use (you must have permission to use it)

instanceImageId
string

Instance image id to use (you must have permission to use it)

options
Array of strings
Items Value: "EncryptCommunication"

Details:

  • "EncryptCommunication": Enable encryption for communication between the driver and executors
Array of objects

Volumes to mount to the job (read only)

{
  • "applicationResource": "string",
  • "mainClass": "string",
  • "cronTimeZone": "string",
  • "cronSchedule": "string",
  • "ttlSecondsAfterFinished": 0,
  • "applicationArguments": [
    ],
  • "restartSettings": {
    },
  • "applicationName": "string",
  • "sparkProperties": {
    },
  • "jars": [
    ],
  • "pythonFiles": [
    ],
  • "files": [
    ],
  • "archives": [
    ],
  • "environmentVariables": {
    },
  • "resourcePool": "string",
  • "instanceImageId": "string",
  • "options": [
    ],
  • "volumeMounts": [
    ]
}

SparkBatchJobRestartSettings

maxRetries
required
integer <int32>

Maximum number of retries after failure. Must be at least 1.

{
  • "maxRetries": 0
}

SparkConnectDriver

applicationName
string

The application name. If not provided a name will be generated.

object

Any Spark configuration/properties to set arguments

jars
Array of strings

Any jars to pass in the --jars Spark argument

pythonFiles
Array of strings

Any python files to pass in the --py-files Spark argument

files
Array of strings

Any files to pass in the --files Spark argument

archives
Array of strings

Any archives to pass in the --archives Spark argument

object

Any environment variables to set

resourcePool
string

Optional - the resource pool to use (you must have permission to use it)

instanceImageId
string

Instance image id to use (you must have permission to use it)

options
Array of strings
Items Value: "EncryptCommunication"

Details:

  • "EncryptCommunication": Enable encryption for communication between the driver and executors
Array of objects

Volumes to mount to the job (read only)

{
  • "applicationName": "string",
  • "sparkProperties": {
    },
  • "jars": [
    ],
  • "pythonFiles": [
    ],
  • "files": [
    ],
  • "archives": [
    ],
  • "environmentVariables": {
    },
  • "resourcePool": "string",
  • "instanceImageId": "string",
  • "options": [
    ],
  • "volumeMounts": [
    ]
}

StatusV2

required
object

The status

{
  • "status": {
    }
}

SystemMaintenance

status
required
string
Enum: "Deactivated" "WaitingForJobsTermination" "ReadyForSafeUpgrade"

System maintenance status

{
  • "status": "Deactivated"
}

SystemMaintenanceHistory

activatedAt
required
string

The system maintenance enabled at

activatedBy
required
string

User that activated the system maintenance

activatedVersion
required
string

The version of the system for which the maintenance mode has been activated

deactivatedAt
string

The system maintenance deactivated at

deactivatedBy
string

User that deactivated the system maintenance

deactivatedVersion
string

The version of the system for which the maintenance mode has been deactivated

{
  • "activatedAt": "string",
  • "activatedBy": "string",
  • "activatedVersion": "string",
  • "deactivatedAt": "string",
  • "deactivatedBy": "string",
  • "deactivatedVersion": "string"
}

UpdateUploadPatch

name
string

Name for the upload. Must be unique in the system.

description
string

Optional description for the upload.

{
  • "name": "string",
  • "description": "string"
}

UploadBase

uploadId
required
string

Upload ID

name
required
string

The name of the upload

description
string

The description of the upload

createdBy
required
string

Who created the upload

createdAt
required
string <date-time>

When the upload was created

state
required
string
Enum: "New" "Inactive" "Activating" "Active"

The current state of the upload

sealed
required
boolean

When true, no further items can be added or removed

markedForDeletion
required
boolean

When true, the upload will be deleted when it is no longer in use

{
  • "uploadId": "string",
  • "name": "string",
  • "description": "string",
  • "createdBy": "string",
  • "createdAt": "2019-08-24T14:15:22Z",
  • "state": "New",
  • "sealed": true,
  • "markedForDeletion": true
}

UploadItem

key
required
string

The path of the item

size
required
integer <int64>

The size of the item

eTag
required
string

The eTag of the item

lastModified
required
string <date-time>

The date/time the item was last modified in UTC

{
  • "key": "string",
  • "size": 0,
  • "eTag": "string",
  • "lastModified": "2019-08-24T14:15:22Z"
}

UploadItemName

key
required
string

The path of the item.

{
  • "key": "string"
}

UploadItemUri

uri
required
string

A pre-signed URI for uploading an item.

{
  • "uri": "string"
}

UploadResult

uploadId
required
string

The affected upload ID

{
  • "uploadId": "string"
}

UserInstanceImageInfo

id
required
string

The instance image Id

name
required
string

Imported image name

tag
required
string

Image tag

description
string

Optional image description

required
object (UserInstanceImageTypeInfo)

Image type

status
required
string
Enum: "Init" "Pending" "Verified" "Rejected" "MarkedForDelete" "Deleted"

Image status

{
  • "id": "string",
  • "name": "string",
  • "tag": "string",
  • "description": "string",
  • "type": {
    },
  • "status": "Init"
}

UserInstanceImageTypeInfo

kind
required
string
Enum: "Spark" "Notebook"

An instance image type

version
required
string

An instance image version

{
  • "kind": "Spark",
  • "version": "string"
}

VolumeMount

uploadId
string

ID of the upload to mount. Use either uploadId or uploadName but not both.

uploadName
string

Name of the upload to mount. Use either uploadId or uploadName but not both.

mountPath
string

Optional mount path. If not specified, the volume is mounted to the working directory using the upload name.

bindToInstance
required
boolean

Can only be used with new uploads. Ties the upload to the workload being created so that when the workload is deleted, the upload is also deleted. The upload cannot be used with any other workload.

{
  • "uploadId": "string",
  • "uploadName": "string",
  • "mountPath": "string",
  • "bindToInstance": true
}

Responses

Aborted

message
string
{
  • "message": "string"
}

BadRequest

message
string

Description of the error

fields
Array of strings

Any fields/values that are invalid or incorrect

{
  • "message": "string",
  • "fields": [
    ]
}

Busy

message
string
{
  • "message": "string"
}

Exists

message
string
{
  • "message": "string"
}

Forbidden

message
string
{
  • "message": "string"
}

InternalError

message
string

Description of the error

{
  • "message": "string"
}

NotFound

message
required
string

Name of the resource

description
string

Violation description

fields
Array of strings

Any fields/values that weren't found

{
  • "message": "string",
  • "description": "string",
  • "fields": [
    ]
}

Unauthorized

message
string
{
  • "message": "string"
}

Unavailable

message
required
string

Error message

{
  • "message": "string"
}