Release notes

New releases of the Dell Data Processing Engine are created whenever significant new features or bug fixes are available. Each DDPE release is based on a corresponding SEP release, indicated in parentheses.

1.9 (480-e) (September 2026)

• Added support for uploads to bundle and attach files to Spark jobs and Spark Connect servers.

• Added support for a secrets manager for storing and managing credentials used by Spark jobs and Spark Connect servers.

• Added support for automatic retry on failure for Spark batch jobs.

• Added support for creating and managing data sources in Starburst Metastore via the Starburst Control Plane API.

• Simplified the job creation UI to a single dialog.

• Updated the CLI with the following changes:

  • Added uploads commands: add, list, info, activate, deactivate, seal, update, delete, delete-item, get-item, and rename-item.

  • Added secret commands: create, list, get, update, delete, and list-providers.

1.8 (479-e) (May 2026)

• Added support for custom Spark images.

• Added support for maintenance mode.

• Added log retention settings for Spark jobs.

• Updated the CLI with the following changes:

  • Added the update command for resource pools.

  • Added instance-images commands for managing custom instance images.

  • Added log-retention commands for configuring Spark event log and instance log retention settings.

  • Added system-maintenance commands for enabling, disabling, and viewing maintenance mode history.

  • Added global-storage commands for viewing and refreshing global storage usage.

1.6 (476-e) (November 2025)

• Added support for Jupyter Notebook servers.

• Added support for resource pool-enabled GPU acceleration with Spark RAPIDS.

• Added support for pip install in your Spark jobs.

• Updated the API to version 2.

• Updated the CLI with the following changes:

  • Added the add, delete, and edit resource pool commands.

  • Added the duplicate, restart, suspend, resume, and shutdown instance commands.

  • Added the start command for Jupyter Notebooks.

1.5 (474-e) (May 2025)

• Added full built-in access control integration.

• Added support for object storage governance.

• Added UI support for job submissions.

• Added UI support for resource pools.

• Updated the CLI with the following changes:

  • The --logLevel, list-all-instances, and --zipFile CLI commands have been deprecated. Use --log-level, list-instances, and --zip-file instead.

  • The default value for the --ttl-seconds-after-finished command has changed from 0 to 1.

  • Added history-server-logs command.

  • Added no-browser login command option.

  • Added login-with-credentials command.

  • Updated set config command description.

474-e.3 (July 2025)

• Fixed an access control issue affecting Spark Iceberg write operations when using object storage governance.

1.4 (468-e) (February 2025)

• The Dell Data Processing Engine API is now available.