Manage secrets#
Dell Data Analytics Engine, powered by Starburst Enterprise platform (SEP) manages configuration details in static properties files. This configuration needs to include values such as usernames, passwords, and other strings, that are often required to be kept secret. Only a few selected administrators or the provisioning system has access to the actual value.
The secrets support in SEP allows you to use environment variables as values
for any configuration property. All properties files used by SEP, including
config.properties and catalog properties files, are supported. When loading
the properties, SEP replaces the reference to the environment variable with
the value of the environment variable.
Environment variables are the most widely supported means of setting and retrieving values. Environment variables can be set in the scope of the task being performed, preventing external access. Most provisioning and configuration management systems include support for setting environment variables. This includes systems such as Ansible, often used for virtual machines, and Kubernetes for container usage. You can also manually set an environment variable on the command line.
export DB_PASSWORD=my-super-secret-pwd
To use this variable in the properties file, you reference it with the syntax
${ENV:VARIABLE}. For example, if you want to use the password in a catalog
properties file like etc/catalog/db.properties, add the following line:
connection-password=${ENV:DB_PASSWORD}
With this setup in place, the secret is managed by the provisioning system or by the administrators handling the machines. No secret is stored in the SEP configuration files on the filesystem or wherever they are managed.
Configuration providers#
SEP allows you to use the following configuration providers:
You can also use one or more files as your configuration provider.
Note
Configuration providers only resolve references in catalog properties.
To enable additional configuration providers, use the
configuration-provider.config-files configuration property. Insert a
comma-separated list of the configuration provider property files.
Vault configuration provider#
To enable Vault by Hashicorp as your configuration provider, set the following configuration properties:
configuration-provider.name=vault
address=http://${ENV:VAULT_ADDRESS}:8200
token=${ENV:VAULT_TOKEN}
Reference values from Vault using the syntax ${VAULT:path:key}, where path
is a secret path and key is a secret key.
The following configuration properties are available:
Property |
Description |
|---|---|
|
The address of the Vault server instance. This is a required property. |
|
The Vault authentication token. This is a required property. |
|
The global namespace of the Vault server instance. |
|
The KV Secrets Engine version of the Vault server instance. |
|
The path of a file containing an RSA private key, in unencrypted PEM format with UTF-8 encoding. |
|
The path of a file containing an X.509 certificate, in unencrypted PEM format with UTF-8 encoding. |
|
A JKS keystore file, containing a client certificate registered with Vault’s TLS Certificate auth backend. |
|
The password to the keystore. |
|
A JKS truststore file, containing the Vault server’s X509 certificate. |
|
A Boolean indicating whether to verify the SSL certificate used by Vault with HTTPS connections. |
|
The timeout value, in seconds, for establishing an HTTP(S) connection to the Vault server instance. |
|
The timeout value, in seconds, for downloading data after establishing an HTTP(S) connection. |
|
The depth of the prefix file path. |
|
The number of retry attempts for API operations after a failure. |
|
The wait time, in seconds, between retry attempts for API operations after a failure. |
AWS Secret Manager configuration provider#
To enable AWS Secrets Manager as your configuration provider, set the following configuration property:
configuration-provider.name=asm
Reference values from AWS Secrets Manager using the syntax ${ASM:secret-id} or
${ASM:secret-id:key}.
If your secret contains a JSON object, reference a property by using a JSON pointer expression.
The following configuration properties are available:
Property |
Description |
|---|---|
|
The AWS access key used for authenticating with AWS Secrets Manager. |
|
The AWS secret key used for authenticating with AWS Secrets Manager. |
|
The endpoint URL for your AWS Secrets Manager service. |
|
The AWS region for your AWS Secrets Manager service. |
|
The ARN of the IAM role used when connecting to AWS Secrets Manager. |
|
The role session name used when connecting to AWS Secrets Manager. |
|
External ID for the IAM role trust policy used when connecting to AWS Secrets Manager. |
|
The endpoint URL for the AWS Security Token Service (STS) used for authenticating with AWS Secrets Manager. |
|
The AWS region for the AWS STS service. |
|
The maximum number of concurrent connections to AWS Secrets Manager. |
|
The URL of the HTTP proxy server used for connecting to AWS Secrets Manager. |
|
Boolean indicating whether the HTTP proxy uses TLS. |
File configuration provider#
To use one or more files as your configuration provider, set the following configuration properties:
configuration-provider.name=file
allowed-file-variable-dir=/<path>
You must specify a directory using the allowed-file-variable-dir property.
This ensures that you only allow access to a specific directory on the host.
Note
Your files must be accessible on all nodes.
Reference values from a file using the syntax ${FILE:path}. path must
begin with the directory you specify in allowed-file-variable-dir. For
example:
connector.name=postgresql
connection-url=jdbc:postgres://pg-instance-hostname:5432
connection-user=${FILE:/<path>/pg_user}
connection-password=${FILE:/<path>/pg_password}