Dynamic secrets: database secrets engine
Vault can generate secrets on-demand for some systems. For example, when an app needs to access an Amazon S3 bucket, it asks Vault for AWS credentials. Vault will generate an AWS credential granting permissions to access the S3 bucket. In addition, Vault will automatically revoke this credential after the time-to-live (TTL) expires.
Note
The Getting Started tutorial walks you through the generation of dynamic AWS credentials. This tutorial demonstrates the use of database secrets engine to dynamically generate database credentials.
Challenge
Data protection is a top priority, and database credential rotation is a critical part of any data protection initiative. Each role has a different set of permissions granted to access the database. When a system is attacked by hackers, continuous credential rotation becomes necessary and needs to be automated.
Solution
Applications ask Vault for database credentials rather than setting them as environment variables. The administrator specifies the TTL of the database credentials to enforce its validity so that they are automatically revoked when they are no longer used.
Each app instance can get unique credentials that they don't have to share. By making those credentials short-lived, you reduce the chance that they might be compromised. If an app was compromised, the credentials used by the app can be revoked rather than changing more global sets of credentials.
Launch Terminal
This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.
Personas
The end-to-end scenario described in this tutorial involves two personas:
Prerequisites
This lab was tested on macOS using an x86_64 based processor. If you are running macOS on an Apple silicon-based processor, click the Start interactive lab button or perform the steps using a x86_64 based Linux virtual machine in your preferred cloud provider.
To perform the tasks described in this tutorial, you need to have:
- HCP or Vault Community Edition environment
- Docker to run a PostgreSQL container
- jq
- ngrok installed and configured with an auth token (HCP Vault Dedicated only)
Policy requirements
Each persona requires a different set of capabilities. These are expressed in policies. If you are not familiar with policies, complete the policies tutorial.
The admin tasks require these capabilities.
The apps tasks require these capabilities.
Lab setup
Start Postgres
The tutorial requires a Postgres database. Docker provides a Postgres server image that satisfies this requirement.
Note
This tutorial works for an existing Postgres database given appropriate credentials and connection information.
Pull a Postgres server image with
docker
.Create a Postgres database with a root user named
root
with the passwordrootpassword
.Verify that the postgres container is running.
The credentials generated by the Vault role in the Create a role step requires a role named
ro
that has been granted the ability to read all tables.Create a role named
ro
.Grant the ability to read all tables to the role named
ro
.The database is available and the role is created with the appropriate permissions.
Start Vault
Note
If you do not have access to an HCP Vault Dedicated cluster, visit the Create a Vault Cluster on HCP tutorial.
Launch the HCP Portal and login.
Click Vault in the left navigation pane.
In the Vault clusters pane, click vault-cluster.
Under Cluster URLs, click Public Cluster URL.
In a terminal, set the
VAULT_ADDR
environment variable to the copied address.Return to the Overview page and click Generate token.
Within a few moments, a new token will be generated.
Copy the Admin Token.
Return to the terminal and set the
VAULT_TOKEN
environment variable.Set the
VAULT_NAMESPACE
environment variable toadmin
.The
admin
namespace is the top-level namespace automatically created by HCP Vault. All CLI operations default to use the namespace defined in this environment variable.Type
vault status
to verify your connectivity to the Vault cluster.For Vault Dedicated to interact with resources running on your local machine, a tunnel needs to be established.
In another terminal, start ngrok and connect to PostgreSQL.
Example output:
Copy the ngrok forwarding address.
Return to the terminal where you set the
VAULT_ADDR
environment variable and set an environment variable for the ngrok address. Do not includetcp://
.
The Vault Dedicated server is ready to proceed with the lab.
Scenario Introduction
In this tutorial, you are going to configure the PostgreSQL secrets engine, and create a read-only database role. The Vault-generated PostgreSQL credentials will only have read permission.
- Enable the database secrets engine
- Configure PostgreSQL secrets engine
- Create a role
- Request PostgreSQL credentials
- Manage leases
- Define a password policy
- Define a username template
Enable the database secrets engine
(Persona: admin)
The database secrets engine generates database credentials dynamically based on configured roles.
Enable the database secrets engine at the database/
path.
The database secrets engine is enabled.
Configure PostgreSQL secrets engine
(Persona: admin)
The database secrets engine supports many databases through a plugin interface.
To use a Postgres database with the secrets engine requires further
configuration with the postgresql-database-plugin
plugin and connection
information.
Note
This task uses the connection information defined in the Start Postgres step.
Configure the database secrets engine with the connection credentials for the Postgres database.
The secrets engine is configured to work with Postgres.
Note
Users of Vault version 1.11.0 and beyond can specify multiple comma-separated postgresql server URLs in the value of connection_url
, and Vault will retry communication with each server in the list until it can connect to one that is actively handling requests.
Read the Database Root Credential Rotation tutorial to learn about rotating the root credential immediately after the initial configuration of each database.
Create a role
(Persona: admin)
In configure Postgresql secrets engine step, you configured the
PostgreSQL secrets engine with the allowed role named readonly
. A role is a
logical name within Vault that maps to database credentials. These credentials
are expressed as SQL statements and assigned to the Vault role.
Define the SQL used to create credentials.
The SQL contains the templatized fields {{name}}
, {{password}}
, and
{{expiration}}
. These values are provided by Vault when the credentials are
created. This creates a new role and then grants that role the permissions
defined in the Postgres role named ro
. This Postgres role was created when
Postgres was started.
Create the role named readonly
that creates credentials with the
readonly.sql
.
The role generates database credentials with a default TTL of 1 hour and max TTL of 24 hours.
Request PostgreSQL credentials
(Persona: apps)
The applications that require the database credentials read them from the secret engine's readonly role.
Read credentials from the readonly
database role.
The Postgres credentials are displayed as username
and password
. The
credentials are identified within Vault by the lease_id
.
Validation
Connect to the Postgres database and list all database users.
The output displays a table of all the database credentials generated. The credentials that were recently generated appear in this list.
Manage leases
(Persona: admin)
The credentials are managed by the lease ID and remain valid for the lease duration (TTL) or until revoked. Once revoked the credentials are no longer valid.
List the existing leases.
All valid leases for database credentials are displayed.
Create a variable that stores the first lease ID.
Renew the lease for the database credential by passing its lease ID.
The TTL of the renewed lease is set to 1h
.
Revoke the lease without waiting for its expiration.
List the existing leases.
The lease is no longer valid and is not displayed.
Read new credentials from the readonly
database role.
All leases associated with a path may be removed.
Revoke all the leases with the prefix database/creds/readonly
.
The prefix
flag matches all valid leases with the path prefix of
database/creds/readonly
.
List the existing leases.
All the leases with this path as a prefix have been revoked.
Define a password policy
To perform the steps in this section, you need Vault 1.6 or later. Refer to the password policy tutorial for more details.
The database secret engines generate passwords that adhere to a default pattern that may be overridden with a new password policy. A policy defines the rules and requirements that the password must adhere to and can provide that password directly through a new endpoint or within secrets engines.
The passwords you want to generate adhere to these requirements.
- length of 20 characters
- at least 1 uppercase character
- at least 1 lowercase character
- at least 1 number
- at least 1 symbol
Define this password policy in a file named example_policy.hcl
.
The policy is written in HashiCorp Configuration Language (HCL). The length
field sets the length of the password returned to 20
characters. Each rule
stanza defines a character set and the minimum number of occurrences those
characters need to appear in the generated password. These rules are cumulative
so each one adds more requirements on the password generated.
Create a Vault password policy named example
with the password policy
rules defined in example_policy.hcl
.
This policy can now be accessed directly to generate a password or referenced
by its name example
when configuring supported secrets engines.
Generate a password from the example
password policy.
The password generated adheres to the defined requirements.
Apply the password policy
Configure the database secrets engine with the example
password policy.
The same connection information is used to establish the connection with the
database server. The difference is that the password_policy
has been set to
the example
policy.
Read credentials from the readonly
database role.
The credentials display the username
and password
generated. The password
generated adheres to the example password policy defined in the secrets
engine's configuration.
Define a username template
To perform the steps in this section, you need Vault 1.7 or later.
The database secret engines generate usernames that adhere to a default pattern. A customized username template may be provided to meet the needs of your organization.
Ensure that custom username templates include enough randomness to prevent the same username being generated multiple times.
Read credentials from the readonly
database role.
The generated username, v-token-readonly-wGLPkpDyc6AgqBfMZTD3-1604195404
, uses
the default pattern expressed as a Go template, {{ printf "v-%s-%s-%s-%s" (.DisplayName | truncate 8) (.RoleName | truncate 8) (random 20) (unix_time) | truncate 63 }}
.
Refer to the Username Templating documentation to learn more functions that can be applied.
Configure the database secrets engine with the username template.
This username template is prefixed with myorg-
, uses the name of role,
readonly
, the unix timestamp in seconds, and a random sequence of 8
characters.
Read credentials from the readonly
database role.
The username generated adheres to the template provided to the configuration.
Next Step
There are some tools available to help integrate your applications with Vault's database secrets engine. Using those tools, the existing applications may require minimum to no code change to work with Vault.
Refer to the Vault Agent caching tutorial.