Introduction

direnv is an environment variable manager for your shell. It knows how to hook into bash, zsh and fish shell to load or unload environment variables depending on your current directory. This allows you to have project-specific environment variables and not clutter the “ /.profile” file.

Before each prompt it checks for the existence of an .envrc file in the current and parent directories. If the file exists, it is loaded into a bash sub-shell and all exported variables are then captured by direnv and then made available to your current shell, while unset variables are removed.

Because direnv is compiled into a single static executable it is fast enough to be unnoticeable on each prompt. It is also language agnostic and can be used to build solutions similar to rbenv, pyenv, phpenv, …

Links

• https://direnv.net/

Baskstory

Like many people I use a different directory per project. And many of my projects each have their own virtual python enviroments. I was using a terrible hack that consisted of setting an environment variable in ~/.bashrc:

export FOCUSED_PROJECT="typhoon"

Then using if statements to configure the environment based on the project:

if [ $FOCUSED_PROJECT == "kubespray" ]; then
    ssh-add $HOME/Downloads/pem/some.pem
    cd /data/projects/ic1/kubespray
    export AWS_REGION="us-east-1"
    export AWS_PROFILE=gloop
    export NAMESPACE=default
    export PKI_PRIVATE_PEM=$HOME/Downloads/pem/some.pem
    export SSH_USER=centos
    export SSM_BINARY_DIR=/data/projects/dva/amazon-ssm-agent/bin
fi

I had ten of those if statements in ~/.bashrc. It was ungainly.

Then I learned about direnv. Now I have a .envrc file in each directory. My environment is automatically configured as I switch directories.

Installation

Add the follow to the end of your .bashrc file.

show_virtual_env() {
  if [[ -n "$VIRTUAL_ENV" && -n "$DIRENV_DIR" ]]; then
    echo "($(basename $VIRTUAL_ENV)) "
  fi
}
export -f show_virtual_env
PS1='$(show_virtual_env)'$PS1

eval "$(direnv hook bash)"

Now start a new shell.

Configure To Use venv.

Use the following commands to create a direnvrc file. Notice that variable interpolation is not used in the HEREDOC.

mkdir -p ~/.config/direnv

cat <<-'EOF' >$HOME/.config/direnv/direnvrc
realpath() {
    [[ $1 = /* ]] && echo "$1" || echo "$PWD/${1#./}"
}
layout_python-venv() {
    local python=${1:-python3}
    [[ $# -gt 0 ]] && shift
    unset PYTHONHOME
    if [[ -n $VIRTUAL_ENV ]]; then
        VIRTUAL_ENV=$(realpath "${VIRTUAL_ENV}")
    else
        local python_version
        python_version=$("$python" -c "import platform; print(platform.python_version())")
        if [[ -z $python_version ]]; then
            log_error "Could not detect Python version"
            return 1
        fi
        VIRTUAL_ENV=$PWD/.direnv/python-venv-$python_version
    fi
    export VIRTUAL_ENV
    if [[ ! -d $VIRTUAL_ENV ]]; then
        log_status "no venv found; creating $VIRTUAL_ENV"
        "$python" -m venv "$VIRTUAL_ENV"
    fi

    PATH="${VIRTUAL_ENV}/bin:${PATH}"
    export PATH
}
EOF

Create .envrc File

Switch to a directory which needs a python virtual environment.

cat <<EOF >.envrc
export VIRTUAL_ENV=venv
layout python-venv python3.8
EOF

Note that direnv is smart enough to deactivate the virtual directory when you change out of it.

After creating a .envrc file, you will notice that direnv complains about the .envrc being blocked. This is the security mechanism to avoid loading new files automatically. Otherwise any git repo that you pull, or tar archive that you unpack, would be able to wipe your hard drive once you cd into it.

Run direnv allow and watch direnv loading your new environment. Note that direnv edit . is a handy shortcut that open the file in your $EDITOR and automatically allows it if the file’s modification time has changed.

Configure git To Ignore direnv Files

Because direnv is essentially a personal tool for now, I recommend that you hide the direnv files and folders so that you don’t have to set them in all your project’s .gitignore:

Note that this setting is personal and needs to be run by each git user. Also note that editors, like Visual Code Studio, might not know about this setting and there might not handle it properly.

git config --global core.excludesfile "~/.gitignore_global"
cat <<EOF >> ~/.gitignore_global
# Direnv Files
.direnv
.envrc
# Python Virtual Environment
venv
EOF

Create Scheduled Job To Backup Files

Create /etc/cron.hourly/backup-direnv-configuration-files.sh with the following content to have your .envrc files backed hourly. This seems like a good ida to me.

#!/bin/bash

#
# Find all the .envrc files in my projects directory and its sub-directories.
#

PROJECT_DIR=/data/projects
BACKUP_DIR=/data/direnv-backups

for f in $(find $PROJECT_DIR -name .envrc -type f); do
  BASE_DIR=$(echo $f | rev | cut -b8- | rev)
  ABSOLUTE_DIR=$BACKUP_DIR$BASE_DIR
  mkdir -p $ABSOLUTE_DIR
  cp $f $ABSOLUTE_DIR
  echo "Created backup: $f"
done

Table of Contents

• Introduction
• File Descriptions
• Files
  • accounts.txt
  • encrypted_password.*.txt
  • iam-group-membership.tf
  • iam-groups.tf
  • iam-user-*.tf
  • main.tf
  • terraform.tfvars
  • tfa.sh
  • tfd.sh
  • user-add.sh
  • variables.tf

Introduction

See https://github.com/medined/create-iam-users-with-terraform-and-keybase for the copies of the files discussed here.

GOAL: Provide a cryptographically secure way to distribute first-time passwords to a team. This means the system administrator (me!) should not know their passwords nor have any way to find them out.

Recently I needed to create a few IAM users for a team. I choose to use Terraform to provision them. My first lesson was that while learning Terraform, create a separate directory for each project. For example, with my current knowledge I would not provision server resources and IAM resources in the same Terraform project.

File Descriptions

Since there are several files involved, first I will list them with a description.

No secrets are stored in any of the files below. However, some teams might consider the permission list for each IAM role to be security-relevant. Check with your security people before storing the password files in a code repository though.

Files

accounts.txt

This file should not be in your code repository. Below is an example. Change to fit your needs. Lines that start with a pound sign are ignored. Note that is prefectly find to use one KeyBase account for multiple IAM users. Whoever owns the KeyBase account can decrypt the passwords. This is a great feature for a tester than needs differently permissioned IAM users.

#aws_account,keybase_account,group
admin,medined,administrators
programmer01,medined,developers
programmer02,medined,developers
tester01,medined,console_users
tester02,medined,console_users

encrypted_password.*.txt

The terraform apply process will create any AWS resources needed and produce a local file containing the encrypted password. Below is a elided example of the files that will be created. If the user pastes the file content into the decryption form on the KeyBase site, they can decrypt the information to get their temporary AWS console password.

-----BEGIN PGP MESSAGE-----
Comment: https://keybase.io/download
Version: Keybase Go 1.0.10 (linux)

wcF...ncA
-----END PGP MESSAGE-----

iam-group-membership.tf

The user-add.sh script produces this file using information from accounts.txt. This file should not be part of your code repository.

resource "aws_iam_group_membership" "administrators" {
  name = "group-membership-administrators"
  users = [
    aws_iam_user.admin.name
  ]
  group = aws_iam_group.administrators.name
}
resource "aws_iam_group_membership" "console_users" {
  name = "group-membership-console-users"
  users = [
    aws_iam_user.tester01.name,aws_iam_user.tester02.name
  ]
  group = aws_iam_group.console_users.name
}
resource "aws_iam_group_membership" "developers" {
  name = "group-membership-developers"
  users = [
    aws_iam_user.programmer01.name,aws_iam_user.programmer02.name
  ]
  group = aws_iam_group.developers.name
}

iam-groups.tf

This file has three sections. They are groups, policies, and group policy attachments. Each group and policy combination has its own attachment resource. This provides granular management. Delete an attachment to remove the policy from a group.

Please use some prefix in your group names so that several projects can be managed at the same time. Otherwise, “administrators” in project “abcde” will conflict with project “zyxw”. Only group names need to be namespaced in this way.

resource "aws_iam_group" "administrators" {
  name = "abcde_administrators"
  path = "/"
}
resource "aws_iam_group" "developers" {
  name = "abcde_developers"
  path = "/"
}


data "aws_iam_policy" "AdministratorAccess" {
  arn = "arn:aws:iam::aws:policy/AdministratorAccess"
}
data "aws_iam_policy" "AmazonS3FullAccess" {
  arn = "arn:aws:iam::aws:policy/AmazonS3FullAccess"
}
data "aws_iam_policy" "AmazonEC2FullAccess" {
  arn = "arn:aws:iam::aws:policy/AmazonEC2FullAccess"
}


## ADMINISTRATORS
resource "aws_iam_group_policy_attachment" "administrator" {
  group = aws_iam_group.administrators.name
  policy_arn = data.aws_iam_policy.AdministratorAccess.arn
}
## DEVELOPERS
resource "aws_iam_group_policy_attachment" "developers_AmazonS3FullAccess" {
  group = aws_iam_group.developers.name
  policy_arn = data.aws_iam_policy.AmazonS3FullAccess.arn
}
resource "aws_iam_group_policy_attachment" "developers_AmazonEC2FullAccess" {
  group = aws_iam_group.developers.name
  policy_arn = data.aws_iam_policy.AmazonEC2FullAccess.arn
}

iam-user-*.tf

The user-add.sh script produces these files using information from accounts.txt. These files should not be part of your code repository. Here is an example:

##     _    ____  __  __ ___ _   _
##    / \  |  _ \|  \/  |_ _| \ | |
##   / _ \ | | | | |\/| || ||  \| |
##  / ___ \| |_| | |  | || || |\  |
## /_/   \_\____/|_|  |_|___|_| \_|
##

resource "aws_iam_user" "admin" {
  name = "admin"
}
resource "aws_iam_access_key" "admin" {
  user = aws_iam_user.admin.id
}
resource "aws_iam_user_login_profile" "admin" {
  user    = aws_iam_user.admin.id
  pgp_key = "keybase:medined"
}
resource "local_file" "admin_password" {
  sensitive_content = "-----BEGIN PGP MESSAGE-----\nComment: https://keybase.io/download\nVersion: Keybase Go 1.0.10 (linux)\n\n${aws_iam_user_login_profile.admin.encrypted_password}\n-----END PGP MESSAGE-----\n"
  filename = "encrypted_password.admin.txt"
  file_permission = "0600"
}

main.tf

Define region and aws_profile_name in terraform.tfvars. Otherwise, this is pretty basic.

provider "aws" {
  region  = var.region
  profile = var.aws_profile_name
  version    = "~> 2.70"
}

terraform.tfvars

Don’t add this file to your code repository. It’s simple for this project but sometimes it will contain sensitive information.

aws_profile_name = "bluejay"
region           = "us-east-1"

tfa.sh

## Make sure the user files are up to date.
./user-add.sh

export TF_LOG=TRACE
export TF_LOG_PATH="/tmp/terraform-apply-$(date "+%Y-%m-%d_%H:%M").log"
terraform apply --auto-approve
ls -ltr /tmp/terraform-apply*.log | tail -n 1

tfd.sh

export TF_LOG=TRACE
export TF_LOG_PATH="/tmp/terraform-destroy-$(date "+%Y-%m-%d_%H:%M").log"
terraform destroy --auto-approve
ls -ltr /tmp/terraform-destroy*.log | tail -n 1

user-add.sh

Please install figlet if you don’t have that tool. Or remove the usage of it from the file below.

This script is used like this:

./user-add <iam_user> <keybase_account> <iam_group>

The group must be in the VALID_IAM_GROUPS. Please adjust as needed.

The script is straightforward. It uses a template to produce the user-XXXX.tf file.

The code is:

## A script this complex should be written in Python so that the list of
## valid IAM groups could be automatically generated from the
## iam-groups.tf file.

## Remove the set of user files. This will ensure that users removed from
## the accounts.txt file are also deleted from AWS.
rm -f iam-group-membership.tf iam-user-*.if

VALID_IAM_GROUPS="administrators,console_users,developers"

for LINE in $(cat accounts.txt)
do
  echo "$LINE" | grep --silent "^#"
  [[ $? == 0 ]] && continue
  IAM_USER=$(echo $LINE | cut -d',' -f1)
  KEYBASE_ACCOUNT=$(echo $LINE | cut -d',' -f2)
  IAM_GROUP=$(echo $LINE | cut -d',' -f3)

  echo $VALID_IAM_GROUPS | grep --silent $IAM_GROUP
  if [ $? != 0 ]
  then
    echo "Invalid IAM Group. Please use one of these: $VALID_IAM_GROUPS"
    exit 1
  fi

  BANNER=$(figlet -f standard $(echo $IAM_USER | tr '[:lower:]' '[:upper:]') | sed -e 's/^/# /')
  USER_TF_FILE="iam-user-$IAM_USER.tf"

  echo "Processed: $IAM_USER"

  cat <<EOF > $USER_TF_FILE
$BANNER

resource "aws_iam_user" "$IAM_USER" {
  name = "$IAM_USER"
}
resource "aws_iam_access_key" "$IAM_USER" {
  user = aws_iam_user.$IAM_USER.id
}
resource "aws_iam_user_login_profile" "$IAM_USER" {
  user    = aws_iam_user.$IAM_USER.id
  pgp_key = "keybase:$KEYBASE_ACCOUNT"
}
resource "local_file" "${IAM_USER}_password" {
  sensitive_content = "-----BEGIN PGP MESSAGE-----\nComment: https://keybase.io/download\nVersion: Keybase Go 1.0.10 (linux)\n\n\${aws_iam_user_login_profile.${IAM_USER}.encrypted_password}\n-----END PGP MESSAGE-----\n"
  filename = "encrypted_password.$IAM_USER.txt"
  file_permission = "0600"
}
EOF

done

ADMINISTRATORS=""
CONSOLE_USERS=""
DEVELOPERS=""

for LINE in $(cat accounts.txt)
do
  echo "$LINE" | grep --silent "^#"
  [[ $? == 0 ]] && continue
  IAM_USER=$(echo $LINE | cut -d',' -f1)
  KEYBASE_ACCOUNT=$(echo $LINE | cut -d',' -f2)
  IAM_GROUP=$(echo $LINE | cut -d',' -f3)

  if [ $IAM_GROUP == "administrators" ]; then
    if [ "$ADMINISTRATORS" == "" ]; then
      ADMINISTRATORS="aws_iam_user.$IAM_USER.name"
    else
      ADMINISTRATORS="$ADMINISTRATORS,aws_iam_user.$IAM_USER.name"
    fi
  fi

  if [ $IAM_GROUP == "console_users" ]; then
    if [ "$CONSOLE_USERS" == "" ]; then
      CONSOLE_USERS="aws_iam_user.$IAM_USER.name"
    else
      CONSOLE_USERS="$CONSOLE_USERS,aws_iam_user.$IAM_USER.name"
    fi
  fi

  if [ $IAM_GROUP == "developers" ]; then
    if [ "$DEVELOPERS" == "" ]; then
      DEVELOPERS="aws_iam_user.$IAM_USER.name"
    else
      DEVELOPERS="$DEVELOPERS,aws_iam_user.$IAM_USER.name"
    fi
  fi

done

cat <<EOF > iam-group-membership.tf
resource "aws_iam_group_membership" "administrators" {
  name = "group-membership-administrators"
  users = [
    $ADMINISTRATORS
  ]
  group = aws_iam_group.administrators.name
}
resource "aws_iam_group_membership" "console_users" {
  name = "group-membership-console-users"
  users = [
    $CONSOLE_USERS
  ]
  group = aws_iam_group.console_users.name
}
resource "aws_iam_group_membership" "developers" {
  name = "group-membership-developers"
  users = [
    $DEVELOPERS
  ]
  group = aws_iam_group.developers.name
}
EOF

variables.tf

variable "aws_profile_name" {
  description = "Profile name from ~/.aws/credentials"
}

variable "region" {
  type        = string
  description = "Region of the VPC"
}

When you have the project files assembled, here are the steps.

• Initialize Terraform.

• Apply Terraform.

• Example log file.

• Fix mistakes.

• Email the encrypted files to users with a procedure they should follow. For example,

  * Log into the KeyBase website.

  * In the decryption section, paste the contents of the attached file into the form.

  * Visit https://AWS_ACCOUNT.signin.aws.amazon.com/console. Use the following information to log into the AWS console.

  Account: AWS_ACCOUNT
  Username: IAM_USER
  Password: <decrypted from above>

  You have XXXXXX access. Let me know if you need additional permissions.

  Let me know when you have logged in.