Table of Contents

• Introduction
• File Descriptions
• Files
  • accounts.txt
  • encrypted_password.*.txt
  • iam-group-membership.tf
  • iam-groups.tf
  • iam-user-*.tf
  • main.tf
  • terraform.tfvars
  • tfa.sh
  • tfd.sh
  • user-add.sh
  • variables.tf

Introduction

See https://github.com/medined/create-iam-users-with-terraform-and-keybase for the copies of the files discussed here.

GOAL: Provide a cryptographically secure way to distribute first-time passwords to a team. This means the system administrator (me!) should not know their passwords nor have any way to find them out.

Recently I needed to create a few IAM users for a team. I choose to use Terraform to provision them. My first lesson was that while learning Terraform, create a separate directory for each project. For example, with my current knowledge I would not provision server resources and IAM resources in the same Terraform project.

File Descriptions

Since there are several files involved, first I will list them with a description.

No secrets are stored in any of the files below. However, some teams might consider the permission list for each IAM role to be security-relevant. Check with your security people before storing the password files in a code repository though.

Files

accounts.txt

This file should not be in your code repository. Below is an example. Change to fit your needs. Lines that start with a pound sign are ignored. Note that is prefectly find to use one KeyBase account for multiple IAM users. Whoever owns the KeyBase account can decrypt the passwords. This is a great feature for a tester than needs differently permissioned IAM users.

#aws_account,keybase_account,group
admin,medined,administrators
programmer01,medined,developers
programmer02,medined,developers
tester01,medined,console_users
tester02,medined,console_users

encrypted_password.*.txt

The terraform apply process will create any AWS resources needed and produce a local file containing the encrypted password. Below is a elided example of the files that will be created. If the user pastes the file content into the decryption form on the KeyBase site, they can decrypt the information to get their temporary AWS console password.

-----BEGIN PGP MESSAGE-----
Comment: https://keybase.io/download
Version: Keybase Go 1.0.10 (linux)

wcF...ncA
-----END PGP MESSAGE-----

iam-group-membership.tf

The user-add.sh script produces this file using information from accounts.txt. This file should not be part of your code repository.

resource "aws_iam_group_membership" "administrators" {
  name = "group-membership-administrators"
  users = [
    aws_iam_user.admin.name
  ]
  group = aws_iam_group.administrators.name
}
resource "aws_iam_group_membership" "console_users" {
  name = "group-membership-console-users"
  users = [
    aws_iam_user.tester01.name,aws_iam_user.tester02.name
  ]
  group = aws_iam_group.console_users.name
}
resource "aws_iam_group_membership" "developers" {
  name = "group-membership-developers"
  users = [
    aws_iam_user.programmer01.name,aws_iam_user.programmer02.name
  ]
  group = aws_iam_group.developers.name
}

iam-groups.tf

This file has three sections. They are groups, policies, and group policy attachments. Each group and policy combination has its own attachment resource. This provides granular management. Delete an attachment to remove the policy from a group.

Please use some prefix in your group names so that several projects can be managed at the same time. Otherwise, “administrators” in project “abcde” will conflict with project “zyxw”. Only group names need to be namespaced in this way.

resource "aws_iam_group" "administrators" {
  name = "abcde_administrators"
  path = "/"
}
resource "aws_iam_group" "developers" {
  name = "abcde_developers"
  path = "/"
}


data "aws_iam_policy" "AdministratorAccess" {
  arn = "arn:aws:iam::aws:policy/AdministratorAccess"
}
data "aws_iam_policy" "AmazonS3FullAccess" {
  arn = "arn:aws:iam::aws:policy/AmazonS3FullAccess"
}
data "aws_iam_policy" "AmazonEC2FullAccess" {
  arn = "arn:aws:iam::aws:policy/AmazonEC2FullAccess"
}


## ADMINISTRATORS
resource "aws_iam_group_policy_attachment" "administrator" {
  group = aws_iam_group.administrators.name
  policy_arn = data.aws_iam_policy.AdministratorAccess.arn
}
## DEVELOPERS
resource "aws_iam_group_policy_attachment" "developers_AmazonS3FullAccess" {
  group = aws_iam_group.developers.name
  policy_arn = data.aws_iam_policy.AmazonS3FullAccess.arn
}
resource "aws_iam_group_policy_attachment" "developers_AmazonEC2FullAccess" {
  group = aws_iam_group.developers.name
  policy_arn = data.aws_iam_policy.AmazonEC2FullAccess.arn
}

iam-user-*.tf

The user-add.sh script produces these files using information from accounts.txt. These files should not be part of your code repository. Here is an example:

##     _    ____  __  __ ___ _   _
##    / \  |  _ \|  \/  |_ _| \ | |
##   / _ \ | | | | |\/| || ||  \| |
##  / ___ \| |_| | |  | || || |\  |
## /_/   \_\____/|_|  |_|___|_| \_|
##

resource "aws_iam_user" "admin" {
  name = "admin"
}
resource "aws_iam_access_key" "admin" {
  user = aws_iam_user.admin.id
}
resource "aws_iam_user_login_profile" "admin" {
  user    = aws_iam_user.admin.id
  pgp_key = "keybase:medined"
}
resource "local_file" "admin_password" {
  sensitive_content = "-----BEGIN PGP MESSAGE-----\nComment: https://keybase.io/download\nVersion: Keybase Go 1.0.10 (linux)\n\n${aws_iam_user_login_profile.admin.encrypted_password}\n-----END PGP MESSAGE-----\n"
  filename = "encrypted_password.admin.txt"
  file_permission = "0600"
}

main.tf

Define region and aws_profile_name in terraform.tfvars. Otherwise, this is pretty basic.

provider "aws" {
  region  = var.region
  profile = var.aws_profile_name
  version    = "~> 2.70"
}

terraform.tfvars

Don’t add this file to your code repository. It’s simple for this project but sometimes it will contain sensitive information.

aws_profile_name = "bluejay"
region           = "us-east-1"

tfa.sh

## Make sure the user files are up to date.
./user-add.sh

export TF_LOG=TRACE
export TF_LOG_PATH="/tmp/terraform-apply-$(date "+%Y-%m-%d_%H:%M").log"
terraform apply --auto-approve
ls -ltr /tmp/terraform-apply*.log | tail -n 1

tfd.sh

export TF_LOG=TRACE
export TF_LOG_PATH="/tmp/terraform-destroy-$(date "+%Y-%m-%d_%H:%M").log"
terraform destroy --auto-approve
ls -ltr /tmp/terraform-destroy*.log | tail -n 1

user-add.sh

Please install figlet if you don’t have that tool. Or remove the usage of it from the file below.

This script is used like this:

./user-add <iam_user> <keybase_account> <iam_group>

The group must be in the VALID_IAM_GROUPS. Please adjust as needed.

The script is straightforward. It uses a template to produce the user-XXXX.tf file.

The code is:

## A script this complex should be written in Python so that the list of
## valid IAM groups could be automatically generated from the
## iam-groups.tf file.

## Remove the set of user files. This will ensure that users removed from
## the accounts.txt file are also deleted from AWS.
rm -f iam-group-membership.tf iam-user-*.if

VALID_IAM_GROUPS="administrators,console_users,developers"

for LINE in $(cat accounts.txt)
do
  echo "$LINE" | grep --silent "^#"
  [[ $? == 0 ]] && continue
  IAM_USER=$(echo $LINE | cut -d',' -f1)
  KEYBASE_ACCOUNT=$(echo $LINE | cut -d',' -f2)
  IAM_GROUP=$(echo $LINE | cut -d',' -f3)

  echo $VALID_IAM_GROUPS | grep --silent $IAM_GROUP
  if [ $? != 0 ]
  then
    echo "Invalid IAM Group. Please use one of these: $VALID_IAM_GROUPS"
    exit 1
  fi

  BANNER=$(figlet -f standard $(echo $IAM_USER | tr '[:lower:]' '[:upper:]') | sed -e 's/^/# /')
  USER_TF_FILE="iam-user-$IAM_USER.tf"

  echo "Processed: $IAM_USER"

  cat <<EOF > $USER_TF_FILE
$BANNER

resource "aws_iam_user" "$IAM_USER" {
  name = "$IAM_USER"
}
resource "aws_iam_access_key" "$IAM_USER" {
  user = aws_iam_user.$IAM_USER.id
}
resource "aws_iam_user_login_profile" "$IAM_USER" {
  user    = aws_iam_user.$IAM_USER.id
  pgp_key = "keybase:$KEYBASE_ACCOUNT"
}
resource "local_file" "${IAM_USER}_password" {
  sensitive_content = "-----BEGIN PGP MESSAGE-----\nComment: https://keybase.io/download\nVersion: Keybase Go 1.0.10 (linux)\n\n\${aws_iam_user_login_profile.${IAM_USER}.encrypted_password}\n-----END PGP MESSAGE-----\n"
  filename = "encrypted_password.$IAM_USER.txt"
  file_permission = "0600"
}
EOF

done

ADMINISTRATORS=""
CONSOLE_USERS=""
DEVELOPERS=""

for LINE in $(cat accounts.txt)
do
  echo "$LINE" | grep --silent "^#"
  [[ $? == 0 ]] && continue
  IAM_USER=$(echo $LINE | cut -d',' -f1)
  KEYBASE_ACCOUNT=$(echo $LINE | cut -d',' -f2)
  IAM_GROUP=$(echo $LINE | cut -d',' -f3)

  if [ $IAM_GROUP == "administrators" ]; then
    if [ "$ADMINISTRATORS" == "" ]; then
      ADMINISTRATORS="aws_iam_user.$IAM_USER.name"
    else
      ADMINISTRATORS="$ADMINISTRATORS,aws_iam_user.$IAM_USER.name"
    fi
  fi

  if [ $IAM_GROUP == "console_users" ]; then
    if [ "$CONSOLE_USERS" == "" ]; then
      CONSOLE_USERS="aws_iam_user.$IAM_USER.name"
    else
      CONSOLE_USERS="$CONSOLE_USERS,aws_iam_user.$IAM_USER.name"
    fi
  fi

  if [ $IAM_GROUP == "developers" ]; then
    if [ "$DEVELOPERS" == "" ]; then
      DEVELOPERS="aws_iam_user.$IAM_USER.name"
    else
      DEVELOPERS="$DEVELOPERS,aws_iam_user.$IAM_USER.name"
    fi
  fi

done

cat <<EOF > iam-group-membership.tf
resource "aws_iam_group_membership" "administrators" {
  name = "group-membership-administrators"
  users = [
    $ADMINISTRATORS
  ]
  group = aws_iam_group.administrators.name
}
resource "aws_iam_group_membership" "console_users" {
  name = "group-membership-console-users"
  users = [
    $CONSOLE_USERS
  ]
  group = aws_iam_group.console_users.name
}
resource "aws_iam_group_membership" "developers" {
  name = "group-membership-developers"
  users = [
    $DEVELOPERS
  ]
  group = aws_iam_group.developers.name
}
EOF

variables.tf

variable "aws_profile_name" {
  description = "Profile name from ~/.aws/credentials"
}

variable "region" {
  type        = string
  description = "Region of the VPC"
}

When you have the project files assembled, here are the steps.

• Initialize Terraform.

• Apply Terraform.

• Example log file.

• Fix mistakes.

• Email the encrypted files to users with a procedure they should follow. For example,

  * Log into the KeyBase website.

  * In the decryption section, paste the contents of the attached file into the form.

  * Visit https://AWS_ACCOUNT.signin.aws.amazon.com/console. Use the following information to log into the AWS console.

  Account: AWS_ACCOUNT
  Username: IAM_USER
  Password: <decrypted from above>

  You have XXXXXX access. Let me know if you need additional permissions.

  Let me know when you have logged in.

I am seeing the following message in my Ingress Nginx Controller logs. The message has been elided so it is easier to read.

E0823 .../reflector.go:125: Failed to watch *v1.Endpoints: Get "<API URL>":
  dial tcp 10.233.0.1:443: connect: connection refused

Here is the full API URL. Notice that the IP address is not internet addressable. The second line removes the host name from the string. We’ll add the controller information later.

API_URL="https://10.233.0.1:443/api/v1/endpoints?allowWatchBookmarks=true&labelSelector=OWNER%21%3DTILLER&resourceVersion=5840136&timeout=6m24s&timeoutSeconds=384&watch=true"
API_URL=$(echo $API_URL | cut -d'/' -f4-)

The E8023 error does not provide much information. I know that 10.233.0.1 corresponds to my controller node. And I know that the Ingress Nginx Controller is running on my worker node.

NOTE This cluster has just one controller and one worker node.

• Set the namespace and service account of the pod.

NAMESPACE=ingress-nginx
SERVICE_ACCOUNT=ingress-nginx

• Get the service account’s token secret’s name. This will look like ingress-nginx-token-sc7dw.

SECRET=$(kubectl -n $NAMESPACE get serviceaccount ${SERVICE_ACCOUNT} -o json | jq -Mr '.secrets[].name | select(contains("token"))')

• Get the bearer token from the secret and decode it. This is a 941 character string. It looks like eyJhbGc...48AzXRQ.

TOKEN=$(kubectl -n $NAMESPACE get secret ${SECRET} -o json | jq -Mr '.data.token' | base64 -d)

• Make a local copy of the certificate authority file by the secret.

kubectl -n $NAMESPACE get secret ${SECRET} -o json | jq -Mr '.data["ca.crt"]' | base64 -d > /tmp/ca.crt

• Get the hostname of the API server. There are several ways to accomplish this. I choose to use the cluster-info command. The only tricky part to this techique is there are non-visible characters around the host name that need to be removed. That’s what the cut and rev commands are doing.

APISERVER=$(kubectl cluster-info  | head -n 1 | awk '{print $6}' | cut -b16- | rev | cut -b10- | rev)
echo "[$APISERVER]"

• Run the API call that caused the original error.

curl https://$APISERVER:6443/$API_URL --header "Authorization: Bearer $TOKEN" --cacert /tmp/ca.crt
{"type":"ERROR","object":{..."message":"too old resource version: 5840136 (5856237)","reason":"Expired","code":410}}

This was not an expected outcome. Looking closer at the URL being fetched shows:

api/v1/endpoints

I think that should be

openapi/v1/endpoints

Here is the result of that change.

curl https://$APISERVER:6443/openapi/v1/endpoints --header "Authorization: Bearer $TOKEN" --cacert /tmp/ca.crt
{
  "paths": [
    "/apis",
    "/apis/",
    "/apis/apiextensions.k8s.io",
    "/apis/apiextensions.k8s.io/v1",
    "/apis/apiextensions.k8s.io/v1beta1",
    "/healthz",
    "/healthz/etcd",
    "/healthz/log",
    "/healthz/ping",
    "/healthz/poststarthook/crd-informer-synced",
    "/healthz/poststarthook/generic-apiserver-start-informers",
    "/healthz/poststarthook/start-apiextensions-controllers",
    "/healthz/poststarthook/start-apiextensions-informers",
    "/livez",
    "/livez/etcd",
    "/livez/log",
    "/livez/ping",
    "/livez/poststarthook/crd-informer-synced",
    "/livez/poststarthook/generic-apiserver-start-informers",
    "/livez/poststarthook/start-apiextensions-controllers",
    "/livez/poststarthook/start-apiextensions-informers",
    "/metrics",
    "/openapi/v2",
    "/readyz",
    "/readyz/etcd",
    "/readyz/log",
    "/readyz/ping",
    "/readyz/poststarthook/crd-informer-synced",
    "/readyz/poststarthook/generic-apiserver-start-informers",
    "/readyz/poststarthook/start-apiextensions-controllers",
    "/readyz/poststarthook/start-apiextensions-informers",
    "/readyz/shutdown",
    "/version"
  ]
}