2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018

06/28/2020: Provision CentOS-based Kubernetes Cluster On AWS With KubeSpray

Let me start by saying that I will be covering no new ground. This post is only to disambiguate building a centos-based kubernetes cluster by showing the exact steps that I used.

Creating a cluster takes two steps:

  • Provisioning the AWS infrastructure.
  • Installing the Kubernetes software.

It should take less than 20 minutes to create a small cluster. I have just one controller node and one worker node. Kubespray also creates two basion nodes. I don’t mind one bastion but I don’t know why two would be helpful.

This work is being done at the request of the Enterprise Container Working Group (ECWG) of the Office of Information and Technology (OIT - https://www.oit.va.gov/) at the Department of Veteran Affairs.

Provisioning Infrastructure

  • Connect to a base project directory. I use /data/project which is on a separate partition.
cd /data/project
  • Download kubespray. These command will remove all python packages before installing the ones needed for kubespray. This is done to prevent incompatibilites.
git clone https://github.com/kubernetes-sigs/kubespray.git
cd kubespray
pip freeze | xargs pip uninstall -y
pip install -r requirements.txt
  • Change to the aws directory.
cd contrib/terraform/aws
  • Fix old syntax. This is not strictly needed but I like to avoid warning messages when possible.
    • In ./modules/elb/variables.tf, remove quotes around type.
    • In ./modules/vpc/variables.tf, remove quotes around type.
    • In ./variables.tf, remove quotes around type.
  • In ./terraform.tfvars, set variables as needed. Note that the inventory file will be created a few levels up in the directory tree.
cat <<EOF > terraform.tfvars
# Global Vars
aws_cluster_name = "flooper"

# VPC Vars
aws_vpc_cidr_block       = ""
aws_cidr_subnets_private = [ "", "" ]
aws_cidr_subnets_public  = [ "", "" ]

# Bastion Host
aws_bastion_size = "t3.medium"

# Kubernetes Cluster
aws_kube_master_num  = 1
aws_kube_master_size = "t3.medium"

aws_etcd_num  = 1
aws_etcd_size = "t3.medium"

aws_kube_worker_num  = 1
aws_kube_worker_size = "t3.medium"

# Settings AWS ELB
aws_elb_api_port                = 6443
k8s_secure_api_port             = 6443
kube_insecure_apiserver_address = ""

default_tags = {
  #  Env = "devtest"
  #  Product = "kubernetes"

inventory_file = "../../../inventory/hosts"
  • In ./credentials.tfvars, set your AWS credentials. Don’t create a cluster unless you have access to a PEM file related to the AWS_SSH_KEY_NAME EC2 key pair.
AWS_SSH_KEY_NAME = "keypair_name"
AWS_DEFAULT_REGION = "us-east-1"
  • In ./variables.tf, switch the AMI to use Centos.
data "aws_ami" "distro" {
  owners      = ["679593333241"]
  most_recent = true

  filter {
      name   = "name"
      values = ["CentOS Linux 7 x86_64 HVM EBS *"]

  filter {
      name   = "architecture"
      values = ["x86_64"]

  filter {
      name   = "root-device-type"
      values = ["ebs"]
  • Intialize terraform.
terraform init
  • Apply the terraform plan. This is create all of the AWS infrastructure that is needed; including creating a VPC.
terraform apply --var-file=credentials.tfvars --auto-approve

Install kubernetes

  • Connect to the kubespray directory.
cd /data/project/kubespray
  • Export the location of the EC2 key pair PEM file.
  • Run the ansible playbook for CentOS.
ansible-playbook \
    -i ./inventory/hosts \
    ./cluster.yml \
    -e ansible_user=centos \
    -e bootstrap_os=centos \
    -b \
    --become-user=root \
    --flush-cache \
    -e ansible_ssh_private_key_file=$PKI_PRIVATE_PEM
  • Setup kubectl so that is can connect to the new cluster. THIS OVERWRITES YOUR KUBECTL CONFIG FILE!
CONTROLLER_HOST_NAME=$(cat ./inventory/hosts | grep "\[kube-master\]" -A 1 | tail -n 1)
CONTROLLER_IP=$(cat ./inventory/hosts | grep $CONTROLLER_HOST_NAME | grep ansible_host | cut -d'=' -f2)
LB_HOST=$(cat inventory/hosts | grep apiserver_loadbalancer_domain_name | cut -d'"' -f2)

cat <<EOF
             LB_HOST: $LB_HOST

mkdir -p ~/.kube
ssh -F ssh-bastion.conf centos@$CONTROLLER_IP "sudo chmod 644 /etc/kubernetes/admin.conf"
scp -F ssh-bastion.conf centos@$CONTROLLER_IP:/etc/kubernetes/admin.conf ~/.kube/config
sed -i "s^server:.*^server: https://$LB_HOST:6443^" ~/.kube/config
kubectl get nodes

The cluster creation is complete.

Destroy Cluster

cd contrib/terraform/aws
terraform destroy --var-file=credentials.tfvars --auto-approve


  • https://www.youtube.com/watch?v=OEEr2EX8WYc

subscribe via RSS