2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2020

06/28/2020: Provision CentOS-based Kubernetes Cluster On AWS With KubeSpray

Acknowledgements
Provisioning Infrastructure
Install kubernetes
Ingress Controller
Deploy HTTP Application
Delete Certificate Manager
Deploy Certificate Manager
Install KeyCloak
Install Istio
Install Custom Docker Registry
Install Jenkins
Install Krew
Install Octant
Destroy Cluster
Links

**NOTE: The steps outlined below are outdated. However, they might still be useful when learning about KubeSpray. **

Let me start by saying that I will be covering no new ground. This post is only to disambiguate building a centos-based kubernetes cluster by showing the exact steps that I used.

Creating a cluster takes two steps:

Provisioning the AWS infrastructure.
Installing the Kubernetes software.

It should take less than 20 minutes to create a small cluster. I have just one controller node and one worker node. Kubespray also creates two basion nodes. I don’t mind one bastion but I don’t know why two would be helpful.

Watch for configuration options:

Auditing
Encryption At Rest
Kubebench
Pod Security Policy

Acknowledgements

This work is being done at the request of the Enterprise Container Working Group (ECWG) of the Office of Information and Technology (OIT - https://www.oit.va.gov/) at the Department of Veteran Affairs.

Provisioning Infrastructure

Install Helm. Eventually you’ll want to use this tool so let’s get it installed now.

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh
helm repo add stable https://kubernetes-charts.storage.googleapis.com
helm repo update
helm search repo stable

Let’s define some values ahead of need. The cluster name allows you to have multiple clusters in the same AWS account. Each will have its down VPC. And you’ll need to have a different kubespray install for each cluster. I haven’t tried to support multiple clusters.

export CLUSTER_NAME="flooper"
export IMAGE_NAME=medined/aws-encryption-provider

Encryption at Rest - If you need this, then create a Docker image of the aws-encryption-provider. I was not able to find an official version of this image so I created my own copy.

git clone https://github.com/kubernetes-sigs/aws-encryption-provider.git
cd aws-encryption-provider
docker build -t $IMAGE_NAME .
docker login
docker push $IMAGE_NAME

Encryption at Rest - If you need this, then create a KMS key. Remember the key id using a file. I don’t know a better way of tracking this information. I read that using an alias causes a problem. The tags are not displayed on the console. You can source this file to have the environment variables handy. I added AWS_DEFAULT_REGION and other values so you can find them later.

KEY_ID=$(aws kms create-key --tags TagKey=purpose,TagValue=k8s-encryption --query KeyMetadata.KeyId --output text)

export KEY_ARN=$(aws kms describe-key --key-id $KEY_ID --query KeyMetadata.Arn --output text)

cat <<EOF > $HOME/$CLUSTER_NAME-encryption-provider-kms-key.env
AWS_DEFAULT_REGION=us-east-1
CLUSTER_NAME=$CLUSTER_NAME
IMAGE_NAME=$IMAGE_NAME
KEY_ID=$KEY_ID
KEY_ARN=$KEY_ARN
EOF

Connect to a base project directory. I use /data/project which is on a separate partition.

cd /data/project

Download kubespray. These command will remove all python packages before installing the ones needed for kubespray. This is done to prevent incompatibilites.

git clone https://github.com/kubernetes-sigs/kubespray.git
cd kubespray
pip freeze | xargs pip uninstall -y
pip install -r requirements.txt

Encryption at Rest - If you need this, then update contrib/terraform/aws/modules/iam/main.tf after making a copy. I think that only “kms:ListKeys”, “kms:TagResource”, “kms:Encrypt”, “kms:DescribeKey”, and “kms:CreateKey” are needed but just in case I allow all actions. Add the following to the file.

,{
  "Effect": "Allow",
  "Action": "kms:*",
  "Resource": ["*"]
}

Auditing - In order to turn auditing on, edit the roles/kubernetes/master/defaults/main/main.yml file so the values in the file match those below:

kubernetes_audit: true
audit_log_maxbackups: 10

In order to have an always pull policy for the pods created by KubeSpray, edit roles/kubespray-defaults/defaults/main.yaml so the values in the file match those below:

k8s_image_pull_policy: Always

I prefer to use Octant (https://octant.dev/) or Lens (https://k8slens.dev/) instead of the Kubernetes Dashboard. Therefore, I don’t enable it. Edit roles/kubespray-defaults/defaults/main.yaml so the values in the file match those below:

dashboard_enabled: false

Turn on the metrics servers by editting roles/kubespray-defaults/defaults/main.yaml so the values in the file match those below. In the cluster, the “kubectl top nodes” command will be supported.

metrics_server_enabled: true

Have KubeSpray make a copy of kubeconfig in /artifacts. This means you don’t need to scp the generated admin.conf file from the controller yourself. Edit roles/kubespray-defaults/defaults/main.yaml so the values in the file match those below. After the playbook is complete, copy inventory/sample/artifacts/admin.conf to ~/.kube/config. Remember that this will overwrite any existing kubeconfig file so be careful with this copy!

kubeconfig_localhost: true

Let KubeSpray install helm by roles/kubespray-defaults/defaults/main.yaml so the values in the file match those below.

helm_enabled: true

Let KubeSpray install registry by roles/kubespray-defaults/defaults/main.yaml so the values in the file match those below.

registry_enabled: true

Let KubeSpray install cert-manager by roles/kubespray-defaults/defaults/main.yaml so the values in the file match those below.

cert_manager_enabled: true

Change to the aws directory.

cd contrib/terraform/aws

In ./terraform.tfvars, set variables as needed. Note that the inventory file will be created a few levels up in the directory tree.

cat <<EOF > terraform.tfvars
# Global Vars
aws_cluster_name = "$CLUSTER_NAME"

# VPC Vars
aws_vpc_cidr_block       = "10.250.192.0/18"
aws_cidr_subnets_private = ["10.250.192.0/20", "10.250.208.0/20"]
aws_cidr_subnets_public  = ["10.250.224.0/20", "10.250.240.0/20"]

# Bastion Host
aws_bastion_size = "t3.medium"

# Kubernetes Cluster
aws_kube_master_num  = 1
aws_kube_master_size = "t3.medium"

aws_etcd_num  = 1
aws_etcd_size = "t3.medium"

aws_kube_worker_num  = 1
aws_kube_worker_size = "t3.medium"

# Settings AWS ELB
aws_elb_api_port                = 6443
k8s_secure_api_port             = 6443
kube_insecure_apiserver_address = "0.0.0.0"

default_tags = {
  #  Env = "devtest"
  #  Product = "kubernetes"
}

inventory_file = "../../../inventory/hosts"
EOF

In ./credentials.tfvars, set your AWS credentials. Don’t create a cluster unless you have access to a PEM file related to the AWS_SSH_KEY_NAME EC2 key pair.

AWS_ACCESS_KEY_ID = "111AXLYWH3DH2FGKSOFQ"
AWS_SECRET_ACCESS_KEY = "111dvxqDOX4RXJN7BQRZI/HD02WDW2SwV5Ck8R7F"
AWS_SSH_KEY_NAME = "keypair_name"
AWS_DEFAULT_REGION = "us-east-1"

In ./variables.tf, switch the AMI to use Centos.

data "aws_ami" "distro" {
  owners      = ["679593333241"]
  most_recent = true

  filter {
      name   = "name"
      values = ["CentOS Linux 7 x86_64 HVM EBS *"]
  }

  filter {
      name   = "architecture"
      values = ["x86_64"]
  }

  filter {
      name   = "root-device-type"
      values = ["ebs"]
  }
}

Intialize terraform.

terraform init

Apply the terraform plan. This is create all of the AWS infrastructure that is needed; including creating a VPC. Note that I remove two files that might have been created by a previous apply.

rm ../../../inventory/hosts ../../../ssh-bastion.conf
time terraform apply --var-file=credentials.tfvars --auto-approve

Install kubernetes

Connect to the kubespray directory.

cd /data/projects/kubespray

Export the location of the EC2 key pair PEM file.

export PKI_PRIVATE_PEM=KEYPAIR.pem

Run the ansible playbook for CentOS.

time ansible-playbook \
  -vvvvv \
  -i ./inventory/hosts \
  ./cluster.yml \
  -e ansible_user=centos \
  -e cloud_provider=aws \
  -e bootstrap_os=centos \
  --become \
  --become-user=root \
  --flush-cache \
  -e ansible_ssh_private_key_file=$PKI_PRIVATE_PEM \
  | tee /tmp/kubespray-cluster-$(date "+%Y-%m-%d_%H:%M").log

NOTE: In /tmp, you’ll see Ansible Fact files named after the hostname. For example, /tmp/ip-10-250-192-82.ec2.internal.

NOTE: -e podsecuritypolicy_enabled=true -e kube_apiserver_enable_admission_plugins=AlwaysPullImages - I tried these parameter options but they did not work for me. I hopefully I will discover that I’ve done something wrong and can use them in the future. In the meantime, this entry has an alternative to enable Pod Security Policy.

NOTE: If you see a failure with the message target uses selinux but python bindings (libselinux-python) aren't installed., then install the selinux python page on the computer running Ansible. The command should be something like python2 -m pip install selinux. I also run the command for python3.

Setup kubectl so that is can connect to the new cluster. THIS OVERWRITES YOUR KUBECTL CONFIG FILE!

CONTROLLER_HOST_NAME=$(cat ./inventory/hosts | grep "\[kube-master\]" -A 1 | tail -n 1)
CONTROLLER_IP=$(cat ./inventory/hosts | grep $CONTROLLER_HOST_NAME | grep ansible_host | cut -d'=' -f2)
LB_HOST=$(cat inventory/hosts | grep apiserver_loadbalancer_domain_name | cut -d'"' -f2)

cat <<EOF
CONTROLLER_HOST_NAME: $CONTROLLER_HOST_NAME
       CONTROLLER_IP: $CONTROLLER_IP
             LB_HOST: $LB_HOST
EOF

# Get the controller's SSH fingerprint.
ssh-keygen -R $CONTROLLER_IP > /dev/null 2>&1
ssh-keyscan -H $CONTROLLER_IP >> ~/.ssh/known_hosts 2>/dev/null

mkdir -p ~/.kube
ssh -F ssh-bastion.conf centos@$CONTROLLER_IP "sudo chmod 644 /etc/kubernetes/admin.conf"
scp -F ssh-bastion.conf centos@$CONTROLLER_IP:/etc/kubernetes/admin.conf ~/.kube/config
sed -i "s^server:.*^server: https://$LB_HOST:6443^" ~/.kube/config
kubectl get nodes

Create script allowing SSH to controller, worker, and etcd servers.

cat <<EOF > ssh-to-controller.sh
HOST_NAME=$(cat ./inventory/hosts | grep "\[kube-master\]" -A 1 | tail -n 1)
IP=$(cat ./inventory/hosts | grep $HOST_NAME | grep ansible_host | cut -d'=' -f2)
ssh -F ssh-bastion.conf centos@$IP
EOF

cat <<EOF > ssh-to-worker.sh
HOST_NAME=$(cat ./inventory/hosts | grep "\[kube-node\]" -A 1 | tail -n 1)
IP=$(cat ./inventory/hosts | grep $HOST_NAME | grep ansible_host | cut -d'=' -f2)
ssh -F ssh-bastion.conf centos@$IP
EOF

cat <<EOF > ssh-to-etcd.sh
HOST_NAME=$(cat ./inventory/hosts | grep "\[etcd\]" -A 1 | tail -n 1)
IP=$(cat ./inventory/hosts | grep $HOST_NAME | grep ansible_host | cut -d'=' -f2)
ssh -F ssh-bastion.conf centos@$IP
EOF

Pod Security Policy - If you need this, run the following command which provides a basic set of policies. Learning about pod security policies is a big topic. We won’t cover it here other than to say that before enabling the PodSecurityPolicy admission controller, pod security policies need to be in place so that pods in the kube-system namespace can start. That’s what the following command does, it provides a bare minimum set of policies needed to start the apiserver pod. The restricted clusterrole as zero rules. If you want normal users to perform commands, you’ll need to explicitly create rules. Here is summary of the restricted PSP.
- Enable read-only root filesystem
- Enable security profiles
- Prevent host network access
- Prevent privileged mode
- Prevent root privileges
- Whitelist read-only host path
- Whitelist volume types

Each time a Gist is changed, the URL for it changes as well. It’s important to get the latest version. Visit https://gist.github.com/medined/73cfb72c240a413eaf499392fe4026cf, then click on ‘Raw’. Make sure the URL is the same as the one shown below.

kubectl apply -f https://gist.githubusercontent.com/medined/73cfb72c240a413eaf499392fe4026cf/raw/a24a6c9da7d1b19195a0b0ac777e9032a3bc8ec3/rbac-for-pod-security-policies.yaml

This command creates these resources:

podsecuritypolicy.policy/privileged unchanged
podsecuritypolicy.policy/restricted configured
clusterrole.rbac.authorization.k8s.io/psp:privileged unchanged
clusterrole.rbac.authorization.k8s.io/psp:restricted unchanged
clusterrolebinding.rbac.authorization.k8s.io/default:restricted unchanged
rolebinding.rbac.authorization.k8s.io/default:privileged unchanged

Pod Security Policy - If you need this, find the public IP of the controller node, then SSH to it. sudo to be the root user. Now edit /etc/kubernetes/manifests/kube-apiserver.yaml by adding PodSecurityPolicy to the admission plugin list. As soon as you save the file, the apiserver pod will be restarted. This will cause connection errors because the api server stops responding. This is normal. Wait a few minutes and the pod will restart and start responds to requests. Check the command using octant or another technique. If you don’t see the admission controllers in the command, resave the file to restart the pod.

--enable-admission-plugins=NodeRestriction,PodSecurityPolicy

Encryption at Rest - If you need this, visit https://medined.github.io/kubernetes/kubespray/encryption/ansible/add-aws-encryption-provider-to-kubespray/ to complete the cluster creation process.

The cluster creation is complete.

Ingress Controller

See https://docs.nginx.com/nginx-ingress-controller/overview/ for more information.

By default, pods of Kubernetes services are not accessible from the external network, but only by other pods within the Kubernetes cluster. Kubernetes has a built‑in configuration for HTTP load balancing, called Ingress, that defines rules for external connectivity to Kubernetes services. Users who need to provide external access to their Kubernetes services create an Ingress resource that defines rules, including the URI path, backing service name, and other information. The Ingress controller can then automatically program a front‑end load balancer to enable Ingress configuration. The NGINX Ingress Controller for Kubernetes is what enables Kubernetes to configure NGINX and NGINX Plus for load balancing Kubernetes services.

Download the official manifest file.

curl -o ingress-nginx-controller-0.34.1.yaml https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.34.1/deploy/static/provider/aws/deploy.yaml

Pod Security Policy - If you need this, add the following to all ClusterRole and Role resources in the downloaded yaml file. This change lets the service accounts use the privileged pod security policy.

  - apiGroups:      [policy]
    resources:      [podsecuritypolicies]
    resourceNames:  [privileged]
    verbs:          [use]

Pod Security Policy - If you need this, add the following to the end of the file in the downloaded yaml file. I don’t recall why this was needed.

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    helm.sh/chart: ingress-nginx-2.11.1
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.34.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-nginx
subjects:
  - kind: ServiceAccount
    name: ingress-nginx
    namespace: default

Create the resources.

kubectl apply -f ingress-nginx-controller-0.34.1.yaml

This command creates these resources:

namespace/ingress-nginx created
serviceaccount/ingress-nginx created
configmap/ingress-nginx-controller created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
service/ingress-nginx-controller-admission created
service/ingress-nginx-controller created
deployment.apps/ingress-nginx-controller created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
serviceaccount/ingress-nginx-admission created

Verify installation

kubectl get pods -n ingress-nginx \
  -l app.kubernetes.io/name=ingress-nginx --watch

Deploy HTTP Application

In order to make this section work, you’ll need to mess around with DNS.

Find your Ingress Nginx Controller load balancer domain. The answer will look something like aaXXXXf67c55949d8b622XXXX862dce0-bce30cd38eXXXX95.elb.us-east-1.amazonaws.com.

kubectl -n ingress-nginx get service ingress-nginx-controller

NOTE: If the EXTERNAL-IP stays in the pending state, verify that you set -e cloud_provider=aws when the cluster was created.

Create a vanity domain for the service being created in this section. This domain needs to point to the load balancer found in the previous step. I use Route 53 but you can use any DNS service. Please make sure that your can correctly resolve the domain using dig.

export TEXT_RESPONDER_HOST="text-responder.david.va-oit.cloud"

Create a namespace.

kubectl apply -f - <<EOF
apiVersion: v1
kind: Namespace
metadata:
    name: text-responder
    labels:
        name: text-responder
EOF

Deploy a small web server that returns a text message.

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: text-responder
  namespace: text-responder
spec:
  selector:
    matchLabels:
      app: text-responder
  replicas: 1
  template:
    metadata:
      labels:
        app: text-responder
    spec:
      containers:
      - name: text-responder
        image: hashicorp/http-echo
        args:
        - "-text=silverargint"
        ports:
        - containerPort: 5678
---
apiVersion: v1
kind: Service
metadata:
  name: text-responder
  namespace: text-responder
spec:
  ports:
  - port: 80
    targetPort: 5678
  selector:
    app: text-responder
EOF

Check the service is running. You should see the text-responder service in the list. The external IP should be <none>.

kubectl --namespace text-responder get service

Curl should get the default 404 response. The HTTPS request should fail because the local issuer certificate can’t be found.

curl http://$TEXT_RESPONDER_HOST

curl https://$TEXT_RESPONDER_HOST

Route traffic directed at the text-responder subdomain within the cluster.

kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: text-responder-ingress
  namespace: text-responder
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: $TEXT_RESPONDER_HOST
    http:
      paths:
      - backend:
          serviceName: text-responder
          servicePort: 80
EOF

Call the service. It should return silverargint.

curl http://$TEXT_RESPONDER_HOST

Delete Certificate Manager

kubectl delete namespace cert-manager

kubectl delete crd certificaterequests.cert-manager.io certificates.cert-manager.io  challenges.acme.cert-manager.io clusterissuers.cert-manager.io issuers.cert-manager.io orders.acme.cert-manager.io

kubectl delete clusterrole cert-manager-cainjector cert-manager-controller-certificates cert-manager-controller-challenges cert-manager-controller-clusterissuers cert-manager-controller-ingress-shim cert-manager-controller-issuers cert-manager-controller-orders cert-manager-edit cert-manager-role cert-manager-view

kubectl delete clusterrolebinding cert-manager-cainjector cert-manager-controller-certificates cert-manager-controller-challenges cert-manager-controller-clusterissuers cert-manager-controller-ingress-shim cert-manager-controller-issuers cert-manager-controller-orders

kubectl -n kube-system delete role cert-manager-cainjector:leaderelection cert-manager:leaderelection

kubectl -n kube-system delete rolebinding cert-manager-cainjector:leaderelection cert-manager:leaderelection

kubectl delete MutatingWebhookConfiguration cert-manager-webhook

kubectl delete ValidatingWebhookConfiguration cert-manager-webhook

Deploy Certificate Manager

Install certificate manager. Check the chart versions at https://hub.helm.sh/charts/jetstack/cert-manager to find the latest version number.
Apply the custom resource definitions.

kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v1.0.0-beta.0/cert-manager.crds.yaml

In this installation process, do not use global.podSecurityPolicy.enabled=true because it will set apparmor annotations on three pod security polices which get created. The nodes do not support AppArmor. This will result in blocked pods.
Add the jetstack repository.

helm repo add jetstack https://charts.jetstack.io

Get a local copy of the manifest needed for cert-manager.

helm template cert-manager jetstack/cert-manager --namespace cert-manager > cert-manager.yaml

For PodSecurityPolicy If you need this, insert the following at the top of the cert-manager.yaml file.

---
apiVersion: v1
kind: Namespace
metadata:
  name: cert-manager
  labels:
    name: cert-manager

Pod Security Policy - If you need this, the cert-manager roles have no permission to use Pod Security Policies. Add the following to every role and ClusterRole in the cert-manager.yaml file. This is definitly overkill, but I don’t have time to experiment to get more granular.

  - apiGroups:      [policy]
    resources:      [podsecuritypolicies]
    resourceNames:  [restricted]
    verbs:          [use]

Apply the cert-manager manifest.

kubectl apply -f cert-manager.yaml

Check that the pods started.

kubectl get pods --namespace cert-manager

Create an issuer to test the webhook works okay.

kubectl apply -f - <<EOF
apiVersion: v1
kind: Namespace
metadata:
  name: cert-manager-test
---
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: test-selfsigned
  namespace: cert-manager-test
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: selfsigned-cert
  namespace: cert-manager-test
spec:
  dnsNames:
    - example.com
  secretName: selfsigned-cert-tls
  issuerRef:
    name: test-selfsigned
EOF

Check the new certificate. You should see “Certificate issued successfully”.

kubectl --namespace cert-manager-test describe certificate

Cleanup the test resources.

kubectl delete namespace cert-manager-test

Create Let’s Encrypt ClusterIssuer for staging and production environments. The main difference is the ACME server URL. I use the term staging because that is what Let’s Encrypt uses.

Change the email address.

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    email: david.medinets@gmail.com
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-staging-secret
    solvers:
    - http01:
        ingress:
          class: nginx
---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-production
spec:
  acme:
    email: david.medinets@gmail.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-production-secret
    solvers:
    - http01:
        ingress:
          class: nginx
EOF

Check on the status of the development issuer. The entries should be ready.

kubectl get clusterissuer

Add annotation to text-responder ingress. This uses the staging Let’s Encrypt to avoid being rate limited while testing.

kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: text-responder-ingress
  namespace: text-responder
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt-staging
spec:
  tls:
  - hosts:
    - text-responder.david.va-oit.cloud
    secretName: text-responder-tls
  rules:
  - host: text-responder.david.va-oit.cloud
    http:
      paths:
      - backend:
          serviceName: text-responder
          servicePort: 80
        path: "/"
EOF

Review the certificate that cert-manager has created. You’re looking for The certificate has been successfully issued in the message section. It may take a minute or two. If the certificate hasn’t been issue after five minutes, go looking for problems. Start in the logs of the pods in the nginx-ingress namespace.

kubectl --namespace text-responder describe certificate text-responder-tls

Review the secret that is being created by cert-manager.

kubectl --namespace text-responder describe secret text-responder-tls

Add annotation to text-responder ingress.

kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: text-responder-ingress
  namespace: text-responder
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt-production
spec:
  tls:
  - hosts:
    - $TEXT_RESPONDER_HOST
    secretName: text-responder-tls
  rules:
  - host: $TEXT_RESPONDER_HOST
    http:
      paths:
      - backend:
          serviceName: text-responder
          servicePort: 80
        path: "/"
EOF

Delete secret to get new certificate.

kubectl --namespace text-responder delete secret text-responder-tls

You’ll see the certificate is re-issued.

kubectl --namespace text-responder describe certificate text-responder-tls

Wait a few minutes for the certificate to be issues and the pods to settle.

kubectl --namespace text-responder describe secret text-responder-tls

At this point, an HTTPS request should work.

curl https://$TEXT_RESPONDER_HOST

An HTTP request will work as long as you follow the redirect.

curl -L http://$TEXT_RESPONDER_HOST

Install KeyCloak

Note that this KeyCloak as no backup and uses ephemeral drives. Any users and groups will be lost if the pods is restarted. I think.

Once you have KeyCloak integrated into the cluster, you (as the admin) will need to use --context='admin' and ``–context=’medined’` to select which user to authenticate as.

See https://medined.github.io/kubernetes/keycloak/oidc-connect/oidc/add_oidc_to_kubernetes/

Install Istio

Download Istio.

curl -L https://istio.io/downloadIstio | sh -

Put the download directory in your PATH.

export PATH="$PATH:/data/projects/ic1/kubespray/istio-1.7.1/bin"

Connect to the installation directory.

cd istio-1.7.1

Run the precheck.

istioctl x precheck

Install Istio with the demo configuration profile.

istioctl install --set profile=demo

Create a namespace for testing Istio.

kubectl create namespace playground

Enable Istio in the playground namespace.

kubectl label namespace playground istio-injection=enabled

Deploy the sample application.

kubectl --namespace playground apply -f samples/bookinfo/platform/kube/bookinfo.yaml

Check the pods and services. Keep checking the pods until they are ready.

kubectl --namespace playground get services
kubectl --namespace playground get pods

Verify the application is running and serving HTML pages. If the application is working correctly, the response will be <title>Simple Bookstore App</title>.

kubectl --namespace playground exec "$(kubectl --namespace playground get pod -l app=ratings -o jsonpath='{.items[0].metadata.name}')" -c ratings -- curl -s productpage:9080/productpage | grep -o "<title>.*</title>"

Open the application to outside traffic by associating the application to the istio gateway.

kubectl --namespace playground apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

Check the configuration for errors.

istioctl --namespace playground analyze

Get connection information.

export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].port}')
export SECURE_INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="https")].port}')

Set the gateway URL.

export GATEWAY_URL=$INGRESS_HOST:$INGRESS_PORT

Visit the product page in your browser.

xdg-open http://$GATEWAY_URL/productpage

Install the Kiali dashboard, along with Prometheus, Grafana, and Jaeger.

kubectl apply -f samples/addons
while ! kubectl wait --for=condition=available --timeout=600s deployment/kiali -n istio-system; do sleep 1; done

Visit the Kiali dashboard.

istioctl dashboard kiali

Install Custom Docker Registry

TBD

Install Jenkins

TBD

Install Krew

TBD

Install Octant

TBD

Destroy Cluster

cd contrib/terraform/aws
terraform destroy --var-file=credentials.tfvars --auto-approve

Links

https://www.youtube.com/watch?v=OEEr2EX8WYc

06/25/2020: Running MindPoint Group RHEL7 STIG On Centos

This is going to be a long post. We’ll start from scratch and develop the ability to run the MindPoint Group RHEL7 STIG on Centos 7. If you want to go farther, there is also discussion of Lynis. Just running the STIG playbook results in the server having a Lynis hardening index of 73. It can be better.

This work is being done at the request of the Enterprise Container Working Group (ECWG) of the Office of Information and Technology (OIT - https://www.oit.va.gov/ at the Department of Veteran Affairs.

Overview

Provision a server.
Initial setup so Ansible will run.
Run Ansible playbook to be STIG compliant.

Provision

Terraform is used to provision an EC2 server. All configuration is done in variables.tf.

Create a provision directory.
Create variables.tf. Make sure to update using your own values.

variable "aws_profile" {
  default = "ic1"
}

variable "aws_region" {
  default = "us-east-1"
}

variable "instance_type" {
  default = "t3.medium"
}

variable "pki_private_key" {
  default = "KEYPAIR.pem"
}

variable "pki_public_key" {
  default = "KEYPAIR.pub"
}

variable "ssh_cidr_block" {
  default = "0.0.0.0/0"
}

# The ssh_user variable is used by both Terraform and Ansible.
variable "ssh_user" {
  default = "centos"
}

variable "subnet_id" {
  default = "subnet-02c78f939d58e2320"
}

variable "vpc_id" {
  default = "vpc-04bdc9b68b19472c3"
}

# Ansible Variables

variable "ansible_python_interpreter" {
  default = "/bin/python3"
}

variable "banner_text_file" {
  default = "file-banner-text.txt"
}

variable "password_max_days" {
  default = "90"
}

variable "password_min_days" {
  default = "1"
}

variable "sha_crypt_max_rounds" {
  default = "10000"
}

variable "sha_crypt_min_rounds" {
  default = "5000"
}

Create tr_ansible_vars_file.yml.tpl. This template is used to generate tr_ansible_vars_file.yml which is read by the Ansible playbook.

ansible_python_interpreter: ${ansible_python_interpreter}
centos_user_password: ${centos_user_password}

Create data-sources.tf.

#
# Find the latest CentOS AMI.
#
data "aws_ami" "centos" {
  owners      = ["679593333241"]
  most_recent = true

  filter {
      name   = "name"
      values = ["CentOS Linux 7 x86_64 HVM EBS *"]
  }

  filter {
      name   = "architecture"
      values = ["x86_64"]
  }

  filter {
      name   = "root-device-type"
      values = ["ebs"]
  }
}

#
# The password will be created as a resource in another file. After the
# terraform plan is applied, the clear text password is in
# tf_ansible_vars_file.yml. Be careful not to check that file into
# a code repository.
#
data "template_file" "tf_ansible_vars_file" {
    template = "${file("./tr_ansible_vars_file.yml.tpl")}"
    vars = {
        ansible_python_interpreter = var.ansible_python_interpreter
        centos_user_password = random_password.centos_user_password.result
    }
}

Create security-groups.tf

#
# Ingress
#

resource "aws_security_group" "centos_allow_ssh" {
  name        = "centos_allow_ssh"
  description = "Allow SSH"
  vpc_id      = var.vpc_id

  ingress {
    description = "SSH"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = [ var.ssh_cidr_block ]
  }

  tags = {
    Name = "centos_allow_ssh"
  }
}

#
# Egress
#
resource "aws_security_group" "centos_allow_any_outbound" {
  name        = "centos_allow_any_outbound"
  description = "Centos Allow Any Outbound"
  vpc_id      = var.vpc_id

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "centos_allow_any_outbound"
  }
}

Create infrastructure-as-code.tf. Here is the heart of the Terraform file. This file creates the EC2 instance. Note that an EIP is allocated so that the instance can be stopped and restarted without a chance in IP address. An inventory file is created for Ansible to use. The EC2 name will have a timestamp so that you can create more than one instance. However, if you do provision more than one server you lose the ability to destroy using terraform.

# When a profile is specified, tf will try to use
# ~/.aws/credentials.

provider "aws" {
  region  = var.aws_region
  profile = var.aws_profile
  version = "~> 2.66"
}

resource "random_password" "centos_user_password" {
  length = 16
  special = true
  override_special = "_%@"
}

resource "aws_key_pair" "centos" {
  public_key = file(var.pki_public_key)
}

resource "aws_instance" "centos" {
  ami           = data.aws_ami.centos.id
  associate_public_ip_address = "true"
  instance_type = var.instance_type
  key_name      = aws_key_pair.centos.key_name
  subnet_id     = var.subnet_id
  vpc_security_group_ids = [
    aws_security_group.centos_allow_ssh.id,
    aws_security_group.centos_allow_any_outbound.id
  ]
  tags = { Name = "centos-${formatdate("YYYYMMDDhhmmss", timestamp())}" }
}

resource "local_file" "tf_ansible_vars_file" {
  content = data.template_file.tf_ansible_vars_file.rendered
  filename = "${path.module}/tf_ansible_vars_file.yml"
}

resource "aws_eip" "centos" {
  instance = aws_instance.centos.id
  vpc      = true
  tags = { Name = "centos-${formatdate("YYYYMMDDhhmmss", timestamp())}" }
  connection {
    type        = "ssh"
    user        = var.ssh_user
    private_key = file(var.pki_private_key)
    host        = self.public_ip
  }
  provisioner "remote-exec" {
    inline = [
      "sudo yum install -y python3"
    ]  
  }
  provisioner "local-exec" {
    command = "ansible-playbook -u ${var.ssh_user} -i '${self.public_ip},' --private-key ${var.pki_private_key} playbook.setup.yml"
    environment = {
      ANSIBLE_HOST_KEY_CHECKING = "False"
    }
    #    command = "./run-setup-playbook.sh ${var.ssh_user} ${self.public_ip},' ${var.pki_private_key}"
  }
}

#
# We need to export the EIP ip address, not the instance's.
#
resource "local_file" "inventory" {
  content = "[all]\n${aws_eip.centos.public_ip}\n"
  filename = "${path.module}/inventory"
}

Create the last file needed for provisioning, the playbook.setup.yml for Ansible.

---
- hosts: all
  gather_facts: false
  become: yes

  vars_files:
    - tf_ansible_vars_file.yml

  tasks:

    - name: upgrade all packages
      yum:
        name: '*'
        state: latest
      vars:
        ansible_python_interpreter: /usr/bin/python

    #
    # The python3-dnf package is not being found. So I am using yum
    # instead of dnf.
    #
    - name: install packages with python2
      yum:
        name:
          - epel-release
        state: latest
        update_cache: yes
      vars:
        ansible_python_interpreter: /usr/bin/python

    - name: install python selinux bindings so that copy task will work.
      yum:
        name:
          - libselinux-python3
        state: latest
        update_cache: yes
      vars:
        ansible_python_interpreter: /usr/bin/python

    - name: Create Lynis Yum repository file.
      copy:
        dest: /etc/yum.repos.d/lynis.repo
        content: |
          [lynis]
          name=CISOfy Software - Lynis package
          baseurl=https://packages.cisofy.com/community/lynis/rpm/
          enabled=1
          gpgkey=https://packages.cisofy.com/keys/cisofy-software-rpms-public.key
          gpgcheck=1
          priority=2
        mode: "644"

    - name: install packages with python2
      yum:
        name:
          - aide
          - ca-certificates
          - curl
          - fail2ban
          - lynis
          - nss
          - openssl
          - python2-jmespath
          - usbguard
        state: latest
        update_cache: yes
      vars:
        ansible_python_interpreter: /usr/bin/python

    - name: set password for 'centos' user.
      user:
        name: centos
        password: ""

    - name: delete no password sudo configuration
      file:
        path: /etc/sudoers.d/90-cloud-init-users
        state: absent

Let Terraform provision the server.

teraform init
terraform plan
terraform apply --auto-approve

When the apply is done, you’ll be able to SSH into the server. You don’t need to perform this step. It’s just for general knowlege in case you do.

ssh-add KEYPAIR.pem
IP_ADDRESS=$(cat inventory | tail -n 1)
ssh centos@$IP_ADDRESS

Make the current directory available to future steps.

export PROVISION_DIR=$(pwd)

Run RHEL7 STIG

Make sure to have jmespath installed on your local workstation.

pip install jmespath

Download the STIG project.

cd /data/projects
git clone https://github.com/MindPointGroup/RHEL7-STIG.git

Connect to the STIG directory.

cd /data/projects/RHEL7-STIG

Create a playbook to run the STIG.

cat <<EOF > playbook.stig.yml
---
- name: Apply STIG
  hosts: all
  become: yes
  roles:
    - role: ""
EOF

Create a script to run the STIG playbook.

cat <<EOF > run-stig-playbook.sh
#!/bin/bash

# Copy the inventory file generated by Terraform.
cp $PROVISION_DIR/inventory inventory

# Get the SUDO password from the generated Ansible variable file.
ANSIBLE_SUDO_PASS=\$(cat \$PROVISION_DIR/tf_ansible_vars_file.yml | grep centos_user_password | awk '{print \$2}')

python3 \
  \$(which ansible-playbook) \
	--extra-vars "ansible_sudo_pass=\$ANSIBLE_SUDO_PASS" \
  -i inventory \
  -u centos \
  playbook.stig.yml
EOF

chmod +x run-stig-playbook.sh

Run the STIG playbook.

./run-stig-playbook.sh | tee stig.out

STIG Results

The results are too long to include directly. See https://gist.github.com/medined/52d814466fa11d4a633561011c29ccf1.

Lynis Results

The results are too long to include directly. See https://gist.github.com/medined/1d15a0c5b599fed8fc2515bcd0c212ad.

Improving Lynis Hardening Index

Create a script to run the Lynis playbook.

cat <<EOF > run-lynis-playbook.sh
#!/bin/bash

# Copy the inventory file generated by Terraform.
cp $PROVISION_DIR/inventory inventory

# Get the SUDO password from the generated Ansible variable file.
ANSIBLE_SUDO_PASS=\$(cat \$PROVISION_DIR/tf_ansible_vars_file.yml | grep centos_user_password | awk '{print \$2}')

python3 \
  \$(which ansible-playbook) \
	--extra-vars "ansible_sudo_pass=\$ANSIBLE_SUDO_PASS" \
	--extra-vars "ssh_user=centos" \
  -i inventory \
  -u centos \
  playbook.lynis.yml
EOF

chmod +x run-lynis-playbook.sh

Create playbook.lynis.yml

---
- hosts: all
  gather_facts: false
  become: yes

  handlers:

    - name: restart sshd
      service: name=sshd state=restarted
      listen: restart sshd

    - name: Unconditionally reboot the machine
      reboot:
      listen: reboot system

  tasks:

    # ....###....##.....##.########.##.....##
    # ...##.##...##.....##....##....##.....##
    # ..##...##..##.....##....##....##.....##
    # .##.....##.##.....##....##....#########
    # .#########.##.....##....##....##.....##
    # .##.....##.##.....##....##....##.....##
    # .##.....##..#######.....##....##.....##


    - name: AUTH-9230 password hashing rounds - min
      lineinfile:
        path: /etc/login.defs
        state: present
        regexp: "^SHA_CRYPT_MIN_ROUNDS"
        line: "SHA_CRYPT_MIN_ROUNDS 6000"
      tags:
        - AUTH-9230

    - name: AUTH-9230 password hashing rounds - max
      lineinfile:
        path: /etc/login.defs
        state: present
        regexp: "^SHA_CRYPT_MAX_ROUNDS"
        line: "SHA_CRYPT_MAX_ROUNDS 10000"
      tags:
        - AUTH-9230

    - name: AUTH-9328 - Default umask values
      lineinfile:
        path: /etc/login.defs
        state: present
        regexp: "^UMASK"
        line: "UMASK 027"
      tags:
        - AUTH-9328

    - name: AUTH-9328 - Default umask values in /etc/login.defs
      copy:
        dest: /etc/profile.d/umask.sh
        content: |
          # By default, we want umask to get set. This sets it for login shell
          # Current threshold for system reserved uid/gids is 200
          # You could check uidgid reservation validity in
          # /usr/share/doc/setup-*/uidgid file
          if [ $UID -gt 199 ] && [ "`id -gn`" = "`id -un`" ]; then
              umask 007
          else
              umask 027
          fi
        mode: "644"
      tags:
        - AUTH-9328

    # NIST recommends setting the daemon umask to 027
    # (REHL5: http://nvd.nist.gov/scap/content/stylesheet/scap-rhel5-document.htm).
    #
    - name: AUTH-9328 - does /etc/init.d/functions exist?
      stat:
        path: /etc/init.d/functions
      register: auth9328

    - name: AUTH-9328 - Default umask values in /etc/init.d/functions
      lineinfile:
        path: /etc/init.d/functions
        state: present
        regexp: "^umask 022"
        line: "umask 027"
      when: auth9328.stat.exists
      tags:
        - AUTH-9328

    - name: AUTH-9408 (Logging of failed login attempts)
      lineinfile:
        path: /etc/login.defs
        state: present
        regexp: "^FAILLOG_ENAB"
        line: "FAILLOG_ENAB yes"
      tags:
        - AUTH-9328

    - name: Ensure delay after failed login
      lineinfile:
        path: /etc/login.defs
        state: present
        regexp: "^FAIL_DELAY"
        line: "FAIL_DELAY 4"
      tags:
        - "https://www.lisenet.com/2017/centos-7-server-hardening-guide/"


    # .########.####.##.......########
    # .##........##..##.......##......
    # .##........##..##.......##......
    # .######....##..##.......######..
    # .##........##..##.......##......
    # .##........##..##.......##......
    # .##.......####.########.########

    - name: FILE-6344 proc mount - hidepid
      block:
        - name: FILE-6344 proc mount - hidepid
          lineinfile:
            path: /etc/fstab
            state: present
            regexp: "^#?proc /proc"
            line: proc /proc proc rw,nosuid,nodev,noexec,relatime,hidepid=2 0 0
          tags:
            - FILE-6344

        #
        # Since /proc is using hidepid, the polkitd can not see
        # /proc unless we fix its access.
        #
        # The next three steps fixes "GDBus.Error:org.freedesktop.PolicyKit1.Error.Failed  - Cannot determine user of subject"
        #
        - name: FILE-6344 proc mount - create group
          group:
            name: monitor
            state: present

        - name: FILE-6344 proc mount - add monitor to group polkitd
          user:
            name: polkitd
            groups: monitor
            append: yes

        - name: FILE-6344 proc mount - get group id
          shell: getent group monitor | cut -d':' -f3
          register: monitor_group_register

        - debug:
            var: monitor_group_register.stdout

        - name: FILE-6344 proc mount - hidepid
          lineinfile:
            path: /etc/fstab
            state: present
            regexp: "^#?proc /proc"
            line: "proc /proc proc rw,nosuid,nodev,noexec,relatime,hidepid=2,gid= 0 0"
          tags:
            - FILE-6344

    - name: FILE-6374 mount /dev/shm noexec
      lineinfile:
        path: /etc/fstab
        state: present
        regexp: "^#?tmpfs /dev/shm"
        line: tmpfs /dev/shm /tmpfs rw,seclabel,nosuid,noexec,nodev,size=2G 0 0
      tags:
        - FILE-6374

    - name: FILE-6374 mount /dev noexec
      lineinfile:
        path: /etc/fstab
        state: present
        regexp: "^#?devtmpfs /dev"
        line: devtmpfs /dev devtmpfs rw,seclabel,nosuid,noexec,size=2G,nr_inodes=471366,mode=755 0 0
      tags:
        - FILE-6374

    - name: FILE-6374 mount /tmp noexec
      lineinfile:
        path: /etc/fstab
        state: present
        regexp: "^#?tmpfs /tmp"
        line: tmpfs /tmp tmpfs rw,seclabel,nosuid,noexec,nodev,size=2G 0 0
      tags:
        - FILE-6374

    #
    # Some pages on the Internet suggested to use "blacklist <filesystem>"
    # instead of the "/bin/true" approach. Empirical testing shows that
    # the approach below works. At least as far as Lynis is concerned.
    #
    - name: FILE-6430 (Disable mounting of some filesystems)
      copy:
        dest: /etc/modprobe.d/lynis-filesystem-blacklist.conf
        content: |
          install cramfs /bin/true
          install squashfs /bin/true
          install udf /bin/true
      tags:
        - FILE-6430
        - CCE-80137-3


    # .##.....##....###....########..########..########.##....##
    # .##.....##...##.##...##.....##.##.....##.##.......###...##
    # .##.....##..##...##..##.....##.##.....##.##.......####..##
    # .#########.##.....##.########..##.....##.######...##.##.##
    # .##.....##.#########.##...##...##.....##.##.......##..####
    # .##.....##.##.....##.##....##..##.....##.##.......##...###
    # .##.....##.##.....##.##.....##.########..########.##....##

    - name: HRDN-7220 (Check if one or more compilers are installed)
      file:
        path: /usr/bin/as
        state: absent
      tags:
        - HRDN-7220


    # .##....##.########.########..##....##.########.##......
    # .##...##..##.......##.....##.###...##.##.......##......
    # .##..##...##.......##.....##.####..##.##.......##......
    # .#####....######...########..##.##.##.######...##......
    # .##..##...##.......##...##...##..####.##.......##......
    # .##...##..##.......##....##..##...###.##.......##......
    # .##....##.########.##.....##.##....##.########.########

    - name: KRNL-5820 - Core dump - ProcessSizeMax
      lineinfile:
        path: /etc/systemd/coredump.conf
        state: present
        regexp: "^#?ProcessSizeMax"
        line: "ProcessSizeMax=0"
      notify: restart sshd
      tags:
        - KRNL-5820

    - name: KRNL-5820 - Core dump - storage
      lineinfile:
        path: /etc/systemd/coredump.conf
        state: present
        regexp: "^#?Storage"
        line: "Storage=none"
      notify: restart sshd
      tags:
        - KRNL-5820

    - name: KRNL-5820 - Core dump - profile
      copy:
        dest: /etc/profile.d/KRNL-5820.sh
        content: |
          ulimit -c 0
        mode: 644
      notify: restart sshd
      tags:
        - KRNL-5820

    - name: KRNL-5820 - Core dump - limits
      copy:
        dest: /etc/security/limits.d/KRNL-5820.conf
        content: |
          #<domain> <type> <item> <value>
          *         hard   core   0
        mode: 644
      notify: restart sshd
      tags:
        - KRNL-5820

    #
    # net.ipv6.conf.default.accept_redirects and net.ipv4.conf.all.forwarding are not being set.
    #
    - name: KRNL-6000 (Check sysctl key pairs in scan profile)
      copy:
        dest: /etc/sysctl.d/90-lynis.conf
        content: |
          kernel.dmesg_restrict=1
          kernel.kptr_restrict=2
          kernel.sysrq=0
          kernel.yama.ptrace_scope=1
          net.ipv4.conf.all.accept_redirects=0
          net.ipv4.conf.all.forwarding=0
          net.ipv4.conf.all.log_martians=1
          net.ipv4.conf.all.rp_filter=1
          net.ipv4.conf.all.send_redirects=0
          net.ipv4.conf.default.accept_redirects=0
          net.ipv4.conf.default.log_martians=1
          net.ipv6.conf.all.accept_redirects=0
          net.ipv6.conf.default.accept_redirects=0
      notify: reboot system
      tags:
        KRNL-6000

    # ..######..##.....##.########.##.......##......
    # .##....##.##.....##.##.......##.......##......
    # .##.......##.....##.##.......##.......##......
    # ..######..#########.######...##.......##......
    # .......##.##.....##.##.......##.......##......
    # .##....##.##.....##.##.......##.......##......
    # ..######..##.....##.########.########.########

    - name: SHLL-6230 umask check - /etc/bashrc 002
      lineinfile:
        path: /etc/bashrc
        state: present
        regexp: "^       umask 002"
        line: "       umask 027"
      tags:
        - SHLL-6230

    - name: SHLL-6230 umask check - /etc/bashrc 022
      lineinfile:
        path: /etc/bashrc
        state: present
        regexp: "^       umask 022"
        line: "       umask 027"
      tags:
        - SHLL-6230

    - name: SHLL-6230 umask check - /etc/csh.cshrc 002
      lineinfile:
        path: /etc/csh.cshrc
        state: present
        regexp: "^    umask 002"
        line: "    umask 027"
      tags:
        - SHLL-6230

    - name: SHLL-6230 umask check - /etc/csh.cshrc 022
      lineinfile:
        path: /etc/csh.cshrc
        state: present
        regexp: "^    umask 022"
        line: "    umask 027"
      tags:
        - SHLL-6230

    - name: SHLL-6230 umask check - /etc/profile 002
      lineinfile:
        path: /etc/profile
        state: present
        regexp: "^    umask 002"
        line: "    umask 027"
      tags:
        - SHLL-6230

    - name: SHLL-6230 umask check - /etc/profile 022
      lineinfile:
        path: /etc/profile
        state: present
        regexp: "^    umask 022"
        line: "    umask 027"
      tags:
        - SHLL-6230

    # ..######..##....##.####.########.....########.########..######..########..######.
    # .##....##.##...##...##..##.....##.......##....##.......##....##....##....##....##
    # .##.......##..##....##..##.....##.......##....##.......##..........##....##......
    # ..######..#####.....##..########........##....######....######.....##.....######.
    # .......##.##..##....##..##..............##....##.............##....##..........##
    # .##....##.##...##...##..##..............##....##.......##....##....##....##....##
    # ..######..##....##.####.##..............##....########..######.....##.....######.

    - name: Copy default lynis profile
      copy:
        src: /etc/lynis/default.prf
        dest: /etc/lynis/custom.prf
        remote_src: true

    #
    # Centos does not have a /var/account directory. However,
    # we do load the audit package which tracks user actions.
    #
    - name: Skip ACCT-9622 (Check for available Linux accounting information)
      lineinfile:
        path: /etc/lynis/custom.prf
        state: present
        regexp: "^skip-test=ACCT-9622"
        line: "skip-test=ACCT-9622"
      tags:
        ACCT-9622

    - name: Skip AUTH-9229 (Check password hashing methods)
      lineinfile:
        path: /etc/lynis/custom.prf
        state: present
        regexp: "^skip-test=AUTH-9229"
        line: "skip-test=AUTH-9229"
      tags:
        AUTH-9229

    #
    # Changing how and where directories are mounted is beyond the scope of this
    # project. Ideally /tmp, /home, and /var should be on separate drives.
    #
    - name: Skip FILE-6310 (Checking /tmp, /home and /var directory)
      lineinfile:
        path: /etc/lynis/custom.prf
        state: present
        regexp: "^skip-test=FILE-6310"
        line: "skip-test=FILE-6310"
      tags:
        FILE-6310

    # IPTABLES are beyond the scope of this project. I believe include
    # defense in depth. However,
    #
    # 1. Firewall rules are very application-specific.
    # 2. EC2 instances use security groups.
    #
    - name: Skip FIRE-4508 (Check used policies of iptables chains)
      lineinfile:
        path: /etc/lynis/custom.prf
        state: present
        regexp: "^skip-test=FIRE-4508"
        line: "skip-test=FIRE-4508"
      tags:
        FIRE-4508

    #
    # malware scans are too environment specific for a generic
    # project like this to resolve.
    #
    - name: Skip HRDN-7230 (Check for malware scanner)
      lineinfile:
        path: /etc/lynis/custom.prf
        state: present
        regexp: "^skip-test=HRDN-7230"
        line: "skip-test=HRDN-7230"
      tags:
        HRDN-7230

    # Checking for external logging is beyond the scope of this
    # project. There are simply too many ways to enable this
    # feature.
    #
    - name: Skip LOGG-2154 (Checking syslog configuration file)
      lineinfile:
        path: /etc/lynis/custom.prf
        state: present
        regexp: "^skip-test=LOGG-2154"
        line: "skip-test=LOGG-2154"
      tags:
        LOGG-2154

    # Checking for anti-virus software is beyond the scope of this
    # project.
    #
    - name: Skip MALW-3280 (Check if anti-virus tool is installed)
      lineinfile:
        path: /etc/lynis/custom.prf
        state: present
        regexp: "^skip-test=MALW-3280"
        line: "skip-test=MALW-3280"
      tags:
        MALW-3280

    - name: Skip PKGS-7420 because servers will be terminated, not updated.
      lineinfile:
        path: /etc/lynis/custom.prf
        state: present
        regexp: "^skip-test=PKGS-7420"
        line: "skip-test=PKGS-7420"
      tags:
        - PKGS-7420

    #
    # SSH-7408 checks to see if the server runs SSH on something other
    # than 22 (the default port).
    #
    # Changing the port is a bit complex in an automated provision.
    #  - switch to terraform to generate custom security group.
    #  - connect via 22:
    #      - change the port number in /etc/ssh/sshd_config.
    #      - semanage port -a -t ssh_port_t -p tcp 15762
    #      - sudo systemctl restart sshd
    #  - change ansible and other scripts to use the new port number.
    #
    # All of that work is possible but should not be done on a whim.
    #
    - name: Skip SSH-7408 SSH non-default port
      lineinfile:
        path: /etc/lynis/custom.prf
        state: present
        regexp: "^skip-test=SSH-7408:Port"
        line: "skip-test=SSH-7408:Port"
      tags:
        - SSH-7408


    # ..######...######..##.....##
    # .##....##.##....##.##.....##
    # .##.......##.......##.....##
    # ..######...######..#########
    # .......##.......##.##.....##
    # .##....##.##....##.##.....##
    # ..######...######..##.....##

    - name: SSH-7408 - hardening SSH configuration - AllowAgentForwarding
      lineinfile:
        path: /etc/ssh/sshd_config
        state: present
        regexp: "^#?AllowAgentForwarding"
        line: "AllowAgentForwarding no"
        validate: sshd -tf %s
      notify: restart sshd
      tags:
        - SSH-7408

    - name: SSH-7408 - hardening SSH configuration - AllowTcpForwarding
      lineinfile:
        path: /etc/ssh/sshd_config
        state: present
        regexp: "^#?AllowTcpForwarding"
        line: "AllowTcpForwarding no"
        validate: sshd -tf %s
      notify: restart sshd
      tags:
        - SSH-7408

    - name: SSH-7408 - hardening SSH configuration - ClientAliveCountMax
      lineinfile:
        path: /etc/ssh/sshd_config
        state: present
        regexp: "^#?ClientAliveCountMax"
        line: "ClientAliveCountMax 2"
        validate: sshd -tf %s
      notify: restart sshd
      tags:
        - SSH-7408

    - name: SSH-7408 - hardening SSH configuration - ClientAliveInterval
      lineinfile:
        path: /etc/ssh/sshd_config
        state: present
        regexp: "^#?ClientAliveInterval"
        line: "ClientAliveInterval 300"
        validate: sshd -tf %s
      notify: restart sshd
      tags:
        - SSH-7408

    - name: SSH-7408 - hardening SSH configuration - Compression
      lineinfile:
        path: /etc/ssh/sshd_config
        state: present
        regexp: "^#?Compression"
        line: "Compression no"
        validate: sshd -tf %s
      notify: restart sshd
      tags:
        - SSH-7408

    - name: SSH-7408 - hardening SSH configuration - INFO
      lineinfile:
        path: /etc/ssh/sshd_config
        state: present
        regexp: "^#?LogLevel"
        line: "LogLevel VERBOSE"
        validate: sshd -tf %s
      notify: restart sshd
      tags:
        - SSH-7408

    - name: SSH-7408 - hardening SSH configuration - MaxAuthTries
      lineinfile:
        path: /etc/ssh/sshd_config
        state: present
        regexp: "^#?MaxAuthTries"
        line: "MaxAuthTries 3"
        validate: sshd -tf %s
      notify: restart sshd
      tags:
        - SSH-7408

    - name: SSH-7408 - hardening SSH configuration - MaxSessions
      lineinfile:
        path: /etc/ssh/sshd_config
        state: present
        regexp: "^#?MaxSessions"
        line: "MaxSessions 2"
        validate: sshd -tf %s
      notify: restart sshd
      tags:
        - SSH-7408

    - name: SSH-7408 - hardening SSH configuration - PermitRootLogin
      lineinfile:
        path: /etc/ssh/sshd_config
        state: present
        regexp: "^#?PermitRootLogin"
        line: "PermitRootLogin no"
        validate: sshd -tf %s
      notify: restart sshd
      tags:
        - SSH-7408

    # - name: SSH-7408 - hardening SSH configuration - Port
    #   lineinfile:
    #     path: /etc/ssh/sshd_config
    #     state: present
    #     regexp: "^#?Port"
    #     line: "Port "
    #     validate: sshd -tf %s
    #   notify: restart sshd
    #   tags:
    #     - SSH-7408

    - name: SSH-7408 - hardening SSH configuration - TCPKeepAlive
      lineinfile:
        path: /etc/ssh/sshd_config
        state: present
        regexp: "^#?TCPKeepAlive"
        line: "TCPKeepAlive no"
        validate: sshd -tf %s
      notify: restart sshd
      tags:
        - SSH-7408

    - name: SSH-7408 - hardening SSH configuration - UseDNS
      lineinfile:
        path: /etc/ssh/sshd_config
        state: present
        regexp: "^#?UseDNS"
        line: "UseDNS no"
        validate: sshd -tf %s
      notify: restart sshd
      tags:
        - SSH-7408

    - name: SSH-7408 - hardening SSH configuration - X11Forwarding
      lineinfile:
        path: /etc/ssh/sshd_config
        state: present
        regexp: "^#?X11Forwarding"
        line: "X11Forwarding no"
        validate: sshd -tf %s
      notify: restart sshd
      tags:
        - SSH-7408

    - name: SSH-7440 (Check OpenSSH option AllowUsers and AllowGroups)
      lineinfile:
        path: /etc/ssh/sshd_config
        state: present
        regexp: "^#?AllowUsers"
        line: "AllowUsers "
        validate: sshd -tf %s
      notify: restart sshd
      tags:
        - SSH-7440


    # ..######..########..#######..########.....###.....######...########
    # .##....##....##....##.....##.##.....##...##.##...##....##..##......
    # .##..........##....##.....##.##.....##..##...##..##........##......
    # ..######.....##....##.....##.########..##.....##.##...####.######..
    # .......##....##....##.....##.##...##...#########.##....##..##......
    # .##....##....##....##.....##.##....##..##.....##.##....##..##......
    # ..######.....##.....#######..##.....##.##.....##..######...########

    - name: STRG-1846 - Check if firewire storage is disabled
      copy:
        dest: /etc/modprobe.d/firewire.conf
        content: |
          blacklist firewire-core
      tags:
        - STRG-1846


    # .########..#######...#######..##......
    # ....##....##.....##.##.....##.##......
    # ....##....##.....##.##.....##.##......
    # ....##....##.....##.##.....##.##......
    # ....##....##.....##.##.....##.##......
    # ....##....##.....##.##.....##.##......
    # ....##.....#######...#######..########

    - name: TOOL-5104 Fail2ban - create jail
      copy:
        dest: /etc/fail2ban/jail.local
        content: |
          [DEFAULT]
          bantime  = 1800
          findtime  = 300
          maxretry = 3
          banaction = iptables-multiport
          backend = systemd

          [sshd]
          enabled = true
      tags:
        - TOOL-5104

    - name: TOOL-5104 Fail2ban - start and enable
      systemd:
        daemon_reload: yes
        enabled: yes
        masked: no
        name: fail2ban
        state: started


    # .##.....##..######..########.
    # .##.....##.##....##.##.....##
    # .##.....##.##.......##.....##
    # .##.....##..######..########.
    # .##.....##.......##.##.....##
    # .##.....##.##....##.##.....##
    # ..#######...######..########.

    - name: USB-1000 (Check if USB storage is disabled)
      copy:
        dest: /etc/modprobe.d/lynis-usb-storage-blacklist.conf
        content: |
          install usb-storage /bin/true
      tags:
        - USB-1000

    - name: USB-3000 (Check for presence of USBGuard)
      lineinfile:
        path: /etc/usbguard/usbguard-daemon.conf
        state: present
        regexp: "^PresentControllerPolicy="
        line: "PresentControllerPolicy=apply-policy"
      tags:
        - USB-3000


    # .########..########.########...#######...#######..########
    # .##.....##.##.......##.....##.##.....##.##.....##....##...
    # .##.....##.##.......##.....##.##.....##.##.....##....##...
    # .########..######...########..##.....##.##.....##....##...
    # .##...##...##.......##.....##.##.....##.##.....##....##...
    # .##....##..##.......##.....##.##.....##.##.....##....##...
    # .##.....##.########.########...#######...#######.....##...

    #
    # Lots of changes were made. Let's reboot to make sure
    # everything takes effect.
    #

    - name: Unconditionally reboot the machine
      reboot:

Run the Ansible playbook.

./run-stig-playbook.sh

The log file is /var/log/lynis.log. It should show a hardening index of 100. Please review the skipped tests. And the suggestions.

Previous Page: 7 of 154 Next