Using Hashicorp Vault to enable credential cycling

David Thor - 05/28/20

Every software engineer knows the importance of data security, and therefore the importance of protecting the credentials that have access to data. At any given moment applications will handle potentially sensitive customer data before writing it to some storage solution, but the application will keep the credentials it uses to access this storage indefinitely. While the data has inherent business value, the credentials themselves are a big liability and require the utmost care to ensure their protection.

Despite the importance of these credentials, proper handling is often an afterthought to app/service creators. While creating their service, developers are likely to sandbox using private storage littered with dummy data (often their own personal information or pen name) that they expect to delete frequently. It's not until the feature is complete and the service is ready for production that customer data will be accessed. Only then is credential security considered more thoughtfully, and at the tail end of a feature release developers race to consider how this protection will work:

  • Who creates the credentials and how are they created?
  • How are the credentials stored so they don't get lost?
  • Who is responsible for querying the credentials and giving them to my application?
  • How do we revoke the credentials if they are compromised?
  • How do we cycle the credentials to protect ourselves from attacks?

There's a lot of questions to consider, and more importantly these questions are going to be asked by every developer creating services that require sensitive credentials! The broad impact this problem has on the day-to-day lives of software engineers is why SecOps teams spend so much time coming up with standards for credential management.

But SecOps can't handle the problem on their own. They may know how to store the credentials safely and how to generate them, but they don't know what credentials need to be generated or how to provide them to applications and services. Credential management takes collaboration to handle properly, so its important that all engineers know their role and how to prepare their application for production and security.

Setting up a secret manager

Offline storage of credentials is not something you want to handle yourself. Credential storage is a problem that all businesses face, and as such there are a number of solutions available to store sensitive credentials like AWS Secrets Manager, GCP Secret Manager, and Hashicorp Vault. These services not only provide proper encryption for credentials at rest, but also handle rich access control policies that can dictate which people or applications are able to access each credential. While credentials to dev environments may be open to all developers, production credentials may only be accessible to the applications running in the production environment and a handful of dedicated production engineers. This kind of access control is critical to preventing credential exposure.

Throughout this tutorial, we'll be using Hashicorp Vault – both for its ability to be run locally and for its ability to proxy to other secret managers. Hashicorp vault has a few great tutorials on how to run a vault server locally and create/manage secrets using their CLI, so in this tutorial we're going to focus on consuming secrets from an application. If you don't know how to setup a vault server locally or create secrets, we recommend reading the following posts first:

Providing credentials to applications

Now that we know how to run Vault and create secrets, let's create an application that can read from our vault instance. In this tutorial we'll use Node.js, though Vault can be accessed from any of the official or community supported Vault SDKs.

First we're going to be connecting our application to a postgres instance. Before we can begin, we need that postgres instance to exist and then we'll put its credentials into our vault. You can create a postgres instance easily with Docker using the command below:

# Your postgres instance will be available on localhost:5432 once complete
$ docker run -d -p 5432:5432 \
    -e POSTGRES_USER=postgres \
    -e POSTGRES_PASSWORD=password \
    -e POSTGRES_DB=mydatabase \
    postgres:11

Once we have the instance running we need to put the credentials into our vault:

$ vault kv put secret/database user=postgres password=password name=mydatabase
Key              Value
---              -----
created_time     2020-05-27T13:50:05.784242Z
deletion_time    n/a
destroyed        false
version          1

If the command was successful, you should see an output like the above confirming the creation time.

Assuming that was successful, you're now ready to create your application! In a new directory, run the following to bootstrap your application and install the vault SDK:

$ npm init
$ npm install node-vault --save

# In this tutorial we'll be using the credentials to connect to a postgres DB.
# To do so we also need to install the postgres client for node
$ npm install pg --save

Now, let's initialize the vault SDK in a file called index.js:

const VaultSDK = require('node-vault');

const vault_client = VaultSDK({
  apiVersion: 'v1',
  endpoint: process.env.VAULT_ADDR || 'http://localhost:8200',
  token: process.env.VAULT_TOKEN || '<your vault token>',
});

Next up is querying the secret we just created from out vault instance. Let's go ahead and read from our vault and parse the secret data into a postgres client:

// We're wrapping our vault and postgres calls in an async function so we can
// use the async/await pattern
const run = async () => {
  const secret = await vault_client.read('secret/data/database');

  const postgres_config = {
    host: process.env.DATABASE_HOST || 'localhost',
    port: Number(process.env.DATABASE_PORT) || 5432,
    user: secret.data.data.user,
    password: secret.data.data.password,
    database: secret.data.data.name,
  };
  const postgres_client = new PostgresClient(postgres_config);
};

Great! We've now retrieved our secrets from the vault and have created a postgres_client that can read from our postgres instance. Let's go ahead and put it all together by creating a database table and filling it with data:

const VaultSDK = require('node-vault');
const { Client: PostgresClient } = require('pg');

const vault_client = VaultSDK({
  apiVersion: 'v1',
  endpoint: process.env.VAULT_ADDR || 'http://localhost:8200',
  token: process.env.VAULT_TOKEN || '<your vault token>',
});

// We're wrapping our vault and postgres calls in an async function so we can
// use the async/await pattern
const run = async () => {
  const secret = await vault_client.read('secret/data/database');

  const postgres_config = {
    host: process.env.DATABASE_HOST || 'localhost',
    port: Number(process.env.DATABASE_PORT) || 5432,
    user: secret.data.data.user,
    password: secret.data.data.password,
    database: secret.data.data.name,
  };
  const postgres_client = new PostgresClient(postgres_config);

  postgres_client.connect();

  // Create and seed table
  await postgres_client.query(
    'CREATE TABLE users (id int, last_name varchar(255), first_name varchar(255))',
  );
  await postgres_client.query(
    "INSERT INTO users (id, last_name, first_name) VALUES (1, 'User', 'Test')",
  );

  // Query seeded content
  const result = await postgres_client.query('SELECT * FROM users');
  console.log(result.rows);

  // Drop table and kill connection
  await postgres_client.query('DROP TABLE users');
  await postgres_client.end();
};

run();

Congratulations! You've now created an application that can read from a secret manager! More importantly, the application is setup to query credentials from the secret manager on-demand and not simply loading them up once. This is the... key... to ensuring that credentials can be recycled – allowing a SecOps team to control if/when credentials need to be changed from a central source.

Secret managers can be powerful tools to enable developers to work collaboratively with their security teams and properly prepare their applications for production, but this isn't the only benefit of a secret manager. Stay tuned to learn how secret managers can be configured to dynamically generate credentials which further enables application portability!

Improve performance & security with credential leasing

Now that you've successfully setup your application to read from your secret manager, you may be wondering if/why you have to read from the secret manager every time a request happens. Ensuring that you are frequently requesting new credentials is an important mechanic to ensure that credentials are able to be refreshed, but you certainly don't have to request them every minute or second. The structured practice for requesting new secrets on an interval is called a lease in Hashicorp's documentation and vault syntax. Leases are easiest to manage via dynamic credentials because each new request for a credential generates and issues a unique set. You can learn more about dynamic credentials in another post of ours.