API docs by Redocly

Spark Manager (v1)

Download OpenAPI specification:Download

About

APIs to manage your Spark jobs and clusters

⬇️ Download OpenAPI specification

Uploads

The combined size of an uploaded set of text files, binary files, or secrets cannot exceed 10MB and each individual file or secret cannot exceed 1MB.

Manager Service

Spark management

connectSpark

Start a new Spark Connect driver

Authorizations:
accessToken
Request Body schema: application/json
applicationName
string

The application name. If not provided a name will be generated.

object

Any Spark configuration/properties to set arguments

jars
Array of strings

Any jars to pass in the --jars Spark argument

pythonFiles
Array of strings

Any python files to pass in the --py-files Spark argument

files
Array of strings

Any files to pass in the --files Spark argument

archives
Array of strings

Any archives to pass in the --archives Spark argument

object

Any environment variables to set

resourcePool
string

Optional - the resource pool to use (you must have permission to use it)

secretUploads
Array of strings

Optional - secret uploads

Secrets will be set as environment variables in the Spark driver and executors.

Array of objects

Optional - file uploads (read only)

Array of objects

Optional - inline file uploads. See Uploads for more details and limits. (read only)

options
Array of strings
Items Value: "EncryptCommunication"

Details:

  • "EncryptCommunication": Enable encryption for communication between the driver and executors

Responses

Request samples

Content type
application/json
{
  • "applicationName": "string",
  • "sparkProperties": {
    },
  • "jars": [
    ],
  • "pythonFiles": [
    ],
  • "files": [
    ],
  • "archives": [
    ],
  • "environmentVariables": {
    },
  • "resourcePool": "string",
  • "secretUploads": [
    ],
  • "fileUploads": [
    ],
  • "inlineFileUploads": [
    ],
  • "options": [
    ]
}

Response samples

Content type
application/json
{
  • "sparkId": "string"
}

listEvent

List system events

Authorizations:
accessToken

Responses

Response samples

Content type
application/json
[
  • {
    }
]

batchSpark

Submit and run a batch job

Authorizations:
accessToken
Request Body schema: application/json
applicationResource
required
string

The application resource to run - must be on an accessible object store

mainClass
string

The main class of the batch job

cronTimeZone
string

Optional CRON timezone. If provided, specify a time zone for the CRON job. Default is UTC.

See List of Time Zones.

cronSchedule
string

Optional CRON schedule. If provided, runs the job on the given schedule.

See Wikipedia's CRON article for details on CRON schedules.

ttlSecondsAfterFinished
integer <int32>

Optional.

ttlSecondsAfterFinished specifies the number of seconds after completion that the driver is eligible to be deleted/cleaned. Default is 1 seconds.

applicationArguments
Array of strings

Any application arguments

applicationName
string

The application name. If not provided a name will be generated.

object

Any Spark configuration/properties to set arguments

jars
Array of strings

Any jars to pass in the --jars Spark argument

pythonFiles
Array of strings

Any python files to pass in the --py-files Spark argument

files
Array of strings

Any files to pass in the --files Spark argument

archives
Array of strings

Any archives to pass in the --archives Spark argument

object

Any environment variables to set

resourcePool
string

Optional - the resource pool to use (you must have permission to use it)

secretUploads
Array of strings

Optional - secret uploads

Secrets will be set as environment variables in the Spark driver and executors.

Array of objects

Optional - file uploads (read only)

Array of objects

Optional - inline file uploads. See Uploads for more details and limits. (read only)

options
Array of strings
Items Value: "EncryptCommunication"

Details:

  • "EncryptCommunication": Enable encryption for communication between the driver and executors

Responses

Request samples

Content type
application/json
{
  • "applicationResource": "string",
  • "mainClass": "string",
  • "cronTimeZone": "string",
  • "cronSchedule": "string",
  • "ttlSecondsAfterFinished": 0,
  • "applicationArguments": [
    ],
  • "applicationName": "string",
  • "sparkProperties": {
    },
  • "jars": [
    ],
  • "pythonFiles": [
    ],
  • "files": [
    ],
  • "archives": [
    ],
  • "environmentVariables": {
    },
  • "resourcePool": "string",
  • "secretUploads": [
    ],
  • "fileUploads": [
    ],
  • "inlineFileUploads": [
    ],
  • "options": [
    ]
}

Response samples

Content type
application/json
{
  • "sparkId": "string"
}

getSparkLogs

Get the log output of a batch job or a connect driver

Authorizations:
accessToken
path Parameters
sparkId
required
string

A spark instance (batch job, connect driver, etc)

Responses

Response samples

Content type
application/json
{
  • "fieldViolations": [
    ]
}

createPresignedUrl

Create pre-signed URLs for the given bucket, key and credentials

Authorizations:
accessToken
Request Body schema: application/json
accessKey
required
string

The proxy AccessKey provided by your administrator

secretKey
required
string

The proxy SecretKey provided by your administrator

region
required
string

The S3 region of the bucket

bucket
required
string

The bucket for creating the pre-signed URLs

key
required
string

The key for creating the pre-signed URLs

Responses

Request samples

Content type
application/json
{
  • "accessKey": "string",
  • "secretKey": "string",
  • "region": "string",
  • "bucket": "string",
  • "key": "string"
}

Response samples

Content type
application/json
{
  • "presignedUrls": {
    }
}

getSparkLogsWithIndex

Get the log output of a batch job or a connect driver. logsId is the log index to return: Some cluster types have multiple nodes/workers. Pass 0 to get the main logs and then increase the index to get other logs.

Authorizations:
accessToken
path Parameters
sparkId
required
string

A spark instance (batch job, connect driver, etc)

logsId
required
string

Logs from a Spark instance

Responses

Response samples

Content type
application/json
{
  • "fieldViolations": [
    ]
}

getSparkStatus

Get the status of a batch job or a connect driver

Authorizations:
accessToken
path Parameters
sparkId
required
string

A spark instance (batch job, connect driver, etc)

query Parameters
showAll
boolean

If true, show terminated instances as well as active instances

Responses

Response samples

Content type
application/json
{
  • "status": {
    }
}

deleteSpark

Shutdown and remove a Spark instance

Authorizations:
accessToken
path Parameters
sparkId
required
string

A spark instance (batch job, connect driver, etc)

Responses

Response samples

Content type
application/json
{
  • "fieldViolations": [
    ]
}

userSpark

List instances available to the current user

Authorizations:
accessToken
query Parameters
showAll
boolean

If true, show terminated instances as well as active instances

Responses

Response samples

Content type
application/json
[
  • {
    }
]

getLogs

Get the system logs

Authorizations:
accessToken
path Parameters
logsId
required
string

Logs from a Spark instance

Responses

Response samples

Content type
application/json
{
  • "fieldViolations": [
    ]
}

allSparkLogs

Get all the logs of a batch job or a connect driver (driver and any executors) as a single Zip file

Authorizations:
accessToken
path Parameters
sparkId
required
string

A spark instance (batch job, connect driver, etc)

Responses

Response samples

Content type
application/json
{
  • "fieldViolations": [
    ]
}

listSpark

List all known instances

Authorizations:
accessToken
query Parameters
showAll
boolean

If true, show terminated instances as well as active instances

Responses

Response samples

Content type
application/json
[
  • {
    }
]

Resources Service

Resource pool management

getResourcePool

Return the current resource pool set including the total available memory and cores

Authorizations:
accessToken

Responses

Response samples

Content type
application/json
{
  • "totalMemory": "string",
  • "totalCores": "string",
  • "resourcePools": [
    ]
}

updateResourcePool

Update the set of available resource pools

Authorizations:
accessToken
Request Body schema: application/json
required
Array of objects (ResourcePools)

The set of resource pools

Responses

Request samples

Content type
application/json
{
  • "resourcePools": [
    ]
}

Response samples

Content type
application/json
{
  • "resourceName": "string",
  • "description": "string"
}

Uploads Service

File and secret upload management

listSecretUploads

List file uploads

Authorizations:
accessToken

Responses

Response samples

Content type
application/json
[
  • {
    }
]

createSecretUpload

Create a new secret upload

Authorizations:
accessToken
Request Body schema: application/json
comment
required
string

Comment or description. Used only for your own reference purposes.

required
object

Map of name-to-binary secrets. Data must be Base64 encoded.

When the uploaded secret is used in a Spark Connect, batch job, etc. this map of secrets/values are set as environment variables. Thus, the secret name must be valid environment variable identifier.

See Uploads for more details and limits.

Responses

Request samples

Content type
application/json
{
  • "comment": "string",
  • "secrets": {
    }
}

Response samples

Content type
application/json
{
  • "uploadId": "string",
  • "comment": "string",
  • "secretNames": [
    ]
}

listFileUploads

List file uploads

Authorizations:
accessToken

Responses

Response samples

Content type
application/json
[
  • {
    }
]

createFileUpload

Create a new file upload

Authorizations:
accessToken
Request Body schema: application/json
comment
required
string

Comment or description. Used only for your own reference purposes.

required
object

Map of name-to-text files/data

required
object

Map of name-to-binary files/data. Data must be Base64 encoded.

Responses

Request samples

Content type
application/json
{
  • "comment": "string",
  • "textData": {
    },
  • "binaryData": {
    }
}

Response samples

Content type
application/json
{
  • "uploadId": "string",
  • "comment": "string",
  • "textNames": [
    ],
  • "binaryNames": [
    ]
}

getSecretUpload

Get a secret upload

Authorizations:
accessToken
path Parameters
uploadId
required
string

A text or binary file. See Uploads for more details and limits.

Responses

Response samples

Content type
application/json
{
  • "comment": "string",
  • "secretNames": [
    ]
}

updateSecretUpload

Update a secret upload

Authorizations:
accessToken
path Parameters
uploadId
required
string

A text or binary file. See Uploads for more details and limits.

Request Body schema: application/json
comment
required
string

Comment or description. Used only for your own reference purposes.

required
object

Map of name-to-binary secrets. Data must be Base64 encoded.

When the uploaded secret is used in a Spark Connect, batch job, etc. this map of secrets/values are set as environment variables. Thus, the secret name must be valid environment variable identifier.

See Uploads for more details and limits.

Responses

Request samples

Content type
application/json
{
  • "comment": "string",
  • "secrets": {
    }
}

Response samples

Content type
application/json
{
  • "resourceName": "string",
  • "description": "string"
}

deleteSecretUpload

Delete a secret upload

Authorizations:
accessToken
path Parameters
uploadId
required
string

A text or binary file. See Uploads for more details and limits.

Responses

Response samples

Content type
application/json
{
  • "fieldViolations": [
    ]
}

getFileUpload

Get a file upload

Authorizations:
accessToken
path Parameters
uploadId
required
string

A text or binary file. See Uploads for more details and limits.

Responses

Response samples

Content type
application/json
{
  • "comment": "string",
  • "textData": {
    },
  • "binaryData": {
    }
}

updateFileUpload

Update a file upload

Authorizations:
accessToken
path Parameters
uploadId
required
string

A text or binary file. See Uploads for more details and limits.

Request Body schema: application/json
comment
required
string

Comment or description. Used only for your own reference purposes.

required
object

Map of name-to-text files/data

required
object

Map of name-to-binary files/data. Data must be Base64 encoded.

Responses

Request samples

Content type
application/json
{
  • "comment": "string",
  • "textData": {
    },
  • "binaryData": {
    }
}

Response samples

Content type
application/json
{
  • "resourceName": "string",
  • "description": "string"
}

deleteFileUpload

Delete a file upload

Authorizations:
accessToken
path Parameters
uploadId
required
string

A text or binary file. See Uploads for more details and limits.

Responses

Response samples

Content type
application/json
{
  • "fieldViolations": [
    ]
}

Model Definitions

Responses