2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2020

06/12/2020: Add AWS SSM Agent on Fedora CoreOS

Acknowledgements

This work is being done at the request of the Enterprise Container Working Group (ECWG) of the Office of Information and Technology (OIT - https://www.oit.va.gov/) at the Department of Veteran Affairs.

Article

This article shows how to install the AWS SSM Agent on Fedora CoreOS. My goal was to alert when ‘denied’ messages appear in audit logs. My first step towards this goal was to install the SSM agent to allow the server’s /var/log/audit/audit.log file to appear in the CloudWatch Logs console.

After I did this work, I realized that I should have started with the CloudWatch agent. Ah well.

The first step is entirely in your hands. Clone https://github.com/aws/amazon-ssm-agent.git and build the binary files. Whatever directory you use will become the ssm_binary_dir in the ansible command.

The ansible-playbook command looks like this. core is the ssh user for Fedora CoreOS.

python3 ansible-playbook \
  --extra-vars ssm_binary_dir=/data/projects/amazon-ssm-agent/bin \
  -i inventory \
  -u core \
  playbook.harden.yml

Define an inventory file with the IP address of the remote server.

cat <<EOF > inventory
[fcos]
5.235.160.104
EOF

I add the PEM file to SSH to simplify my life. I learned that having PEM files in ~/.ssh slows the connection process and can cause trouble if there are too many. That’s I am using my Downloads directory.

ssh-add /home/medined/Downloads/ec2-key-pair.pem

Now here is the playbook.

cat <<EOF > playbook.aws-ssm-agent.yml
---
- hosts: fcos
  gather_facts: false

  vars:
    ansible_python_interpreter: '/usr/bin/python3'

  tasks:

  # ##########
  # # Amazon SSM Agent
  # ##########

  - name: Copy Amazon SSM Agent
    become: yes
    copy:
      src: ""
      dest: /usr/local/bin
      mode: 755
      force: no
    with_fileglob:
      - "/linux_amd64/*"

  - name: Make logging directory
    become: yes
    file:
      path: /var/log/amazon/ssm
      state: directory

  - name: Make config directory
    become: yes
    file:
      path: /etc/amazon/ssm
      state: directory

  - name: Copy Amazon SSM Agent JSON
    become: yes
    copy:
      src: "/amazon-ssm-agent.json.template"
      dest: /etc/amazon/ssm/amazon-ssm-agent.json

  - name: Copy Amazon SSM Agent JSON
    become: yes
    copy:
      src: "/seelog_unix.xml"
      dest: /etc/amazon/ssm

  - name: Create SSM service file.
    become: yes
    copy:
      dest: /etc/systemd/system/amazon-ssm-agent.service
      content: |
        [Unit]
        Description=amazon-ssm-agent
        [Service]
        Type=simple
        WorkingDirectory=/usr/local/bin
        ExecStart=/usr/local/bin/amazon-ssm-agent
        KillMode=process
        Restart=on-failure
        RestartSec=15min
        [Install]
        WantedBy=network-online.target

  - name: Enable SSM service
    become: yes
    service:
      name: amazon-ssm-agent
      enabled: yes
      state: started
EOF

You now have all of the pieces to deploy the AWS SSM Agent on Fedora CoreOS.

06/11/2020: Using Eureka REST API For Python Service

Acknowledgements

Article

This article shows how to interact with Eureka using its REST API. I provide fully working examples.

Eureka has a REST API that allows non-java technologies (like Python) to interact with it. We’ll use curl to make the REST calls. Of course, the calls can be replicated in any language.

Eureka doesn’t care if a service is actually running. It only cares about the information being sent to it. Therefore, this directory will use FAKE_SERVICE as its service name.

Security Alert

From what I can tell, Eureka provides no security around its REST service. Any process that can reach the Eureka service can change a service’s status or de-register a service. Please make sure your security team understands this issue.

Starting Eureka

The first step is to start a Eureka server. You can do this any way you’d like. The easiest way, if you have docker installed is to run the following command. See https://hub.docker.com/repository/docker/medined/eureka-server if you want more information about the image.

docker run \
  --name eureka-server \
  --rm=true \
  --detach \
  --publish 8761:8761 \
  medined/eureka-server:2.3.0.RELEASE

Available Endpoints

What can you do with the REST API? The list is below. This list is pulled directly from the official documentation so you don’t need to get reference it.

Register application
De-register application
Send heartbeat
Query for all instances
Query for all appID instances
Query for a specific appID/instanceID
Query for a specific instanceID
Take instance out of service
Move instance back into service (remove override)
Update metadata
Query for all instances under a particular vip address
Query for all instances under a particular secure vip address

Examples about how to perform each of these functions are below.

Service Registration

Save the script below as service-register.sh script. Then update the configuration information if needed.

#!/bin/bash

# $1 is used as the host name.

EUREKA_HOST="localhost"
EUREKA_PORT="8761"
EUREKA_URI="http://$EUREKA_HOST:$EUREKA_PORT"

SERVICE_NAME="FAKE-SERVICE"
SERVICE_PROTOCOL="http"
SERVICE_HOST="fake.com"
SERVICE_PORT="5000"

SERVICE_URI="$SERVICE_PROTOCOL://$SERVICE_HOST:$SERVICE_PORT"
HOME_URI="$SERVICE_URI/home"
HEALTH_URI="$SERVICE_URI/health"

# This is the URL shown in the "status" field in the
# instances section of the eureka dashboard.
#
# It's up to you to decide what the URL points to. Some
# information or status endpoint might be good.
STATUS_URI="$SERVICE_URI/health"

# This is the name displayed to the right of the status
# on the eureka dashbard. If the app (FAKE_SERVICE) is
# registered with more than one hostname, they will be
# displayed as a comma-separated list. This hostname
# is part of the heartbeat message.
#
# If you'll have more than one host per service,
# make sure they have different host names.
HOST_NAME="${1:-fake01}"

# Everyone of these parameters seem to be required. I don't know
# anything about secureVipAddress and vipAddress.
#
# dataCenterInfo must have a name of "MyOwn" or "Amazon".
#
# status can be UP, DOWN, STARTING, OUT_OF_SERVICE, UNKNOWN.
#   if the registration status is STARTING, then the service
#   will never be evicted. Also, simply sending a Heartbeat
#   does not change the status.
#
# The metadata fields can be any information you want associated
# with a service. I recommend keeping it short.
#

cat <<EOF > /tmp/json.json
{
  "instance": {
    "app": "$SERVICE_NAME",
    "dataCenterInfo": {
      "@class": "com.netflix.appinfo.MyDataCenterInfo",
      "name": "MyOwn"
    },
    "healthCheckUrl": "$HEALTH_URI",
    "homePageUrl": "$HOME_URI",
    "hostName": "$HOST_NAME",
    "ipAddr": "$SERVICE_HOST",
    "leaseInfo": {
      "evictionDurationInSecs": 90
    },
    "metadata": {
      "owner": "George Harris",
      "cost-code": "1D234R"
    },
    "port": {
      "\$": 5000,
      "@enabled": "true"
    },
    "securePort": {
      "\$": 443,
      "@enabled": "false"
    },
    "secureVipAddress": null,
    "status": "UP",
    "statusPageUrl": "$STATUS_URI",
    "vipAddress": null
  }
}
EOF


curl \
  --header "content-type: application/json" \
  --data-binary @/tmp/json.json \
  $EUREKA_URI/eureka/apps/$SERVICE_NAME

When registering a service, you can specify a status (UP, DOWN, STARTING, OUT_OF_SERVICE, and UNKNOWN). If you specify, STARTING the service will not be evicted if no heartbeat is received. Also, sending a heartbeat does not change the status to UP.

hostname and instance can be thought of as synonmous as far as Eureka is concerned.

Service Information

You can request information about registered services using the following URL:

curl http://localhost:9000/eureka/apps

Then you can narrow down to a specific service:

curl http://localhost:9000/eureka/apps/FAKE-SERVICE

And finally, you can filter down to a specific instance or hostname. In the URL below, fake01 is the host name (AKA the instance).

curl http://localhost:9000/eureka/apps/FAKE-SERVICE/fake01
<instance>
  <hostName>fake01</hostName>
  <app>FAKE-SERVICE</app>
  <ipAddr>fake.com</ipAddr>
  <status>UP</status>
  <overriddenstatus>UNKNOWN</overriddenstatus>
  <port enabled="true">5000</port>
  <securePort enabled="false">443</securePort>
  <countryId>1</countryId>
  <dataCenterInfo class="com.netflix.appinfo.InstanceInfo$DefaultDataCenterInfo">
    <name>MyOwn</name>
  </dataCenterInfo>
  <leaseInfo>
    <renewalIntervalInSecs>30</renewalIntervalInSecs>
    <durationInSecs>90</durationInSecs>
    <registrationTimestamp>1591812947850</registrationTimestamp>
    <lastRenewalTimestamp>1591813378305</lastRenewalTimestamp>
    <evictionTimestamp>0</evictionTimestamp>
    <serviceUpTimestamp>1591812947341</serviceUpTimestamp>
  </leaseInfo>
  <metadata>
    <owner>George Harris</owner>
    <cost-code>1D234R</cost-code>
  </metadata>
  <homePageUrl>http://fake.com:5000/home</homePageUrl>
  <statusPageUrl>http://fake.com:5000/health</statusPageUrl>
  <healthCheckUrl>http://fake.com:5000/health</healthCheckUrl>
  <isCoordinatingDiscoveryServer>false</isCoordinatingDiscoveryServer>
  <lastUpdatedTimestamp>1591812947850</lastUpdatedTimestamp>
  <lastDirtyTimestamp>1591812947341</lastDirtyTimestamp>
  <actionType>ADDED</actionType>
</instance>

Service Heartbeat

NOTE: Sending a heartbeat simply means that a service is not evicted. It does not change a service’s status.

The service-heartbeat.sh script can be used to send a heartbeat for our fake service. It’s function is quite simpe. It PUTs a request to the following URL.

http://localhost:9000/eureka/apps/FAKE-SERVICE/fake01

Notice that no parameters are needed. This contradicts the examples on the web but not the official documentation. I’ve seen examples showing that status or lastDirtyTimestamp should be specified. They don’t.

Heartbeat messages should be sent every 30 seconds.

When a service is evicted, you might see a message like the following in the logs.

Evicting 1 items (expired=1, evictionLimit=1)
DS: Registry: expired lease for FAKE-SERVICE/fake01

Service Deregistration

Deregistering a service is quite simple. Make a request like the following:

curl -X DELETE http://localhost:9000/eureka/apps/FAKE-SERVICE/fake01

Query for a specific instanceID

Although we’re not using Instance IDs, the URL will look like this:

curl http://localhost:9000/eureka/instances/fake01

Service Status Change (overriddenstatus)

Using a curl command like that below, you can change the status of a service. As far as I know, there is no security mechanism stopping a malicious actor from arbitrarily changing status on a production system. Therefore, network controls (like security groups) must be used to secure access.

curl -X PUT http://localhost:9000/eureka/apps/FAKE-SERVICE/fake01/status?value=DOWN

The REST request above sets the overriddenstatus field of a service. This status must be cleared using this kind of request. Make sure to provide the optional status. Otherwise, the status becomes UNKNOWN and the service will be shortly evicted.

curl -X DELETE http://localhost:9000/eureka/apps/FAKE-SERVICE/fake01/status?value=UP

Service Metadata Update

The request below will add or change metadata associated with a service. There is no security stopping a bad actor from changing the wrong key, value or even changing the wrong service.

curl -X PUT http://localhost:9000/eureka/apps/FAKE-SERVICE/fake01/metadata?color=BLUE

Research Links

https://github.com/Netflix/eureka/wiki/Eureka-REST-operations
https://dzone.com/articles/the-mystery-of-eureka-health-monitoring
https://blog.asarkar.org/technical/netflix-eureka/

Previous Page: 9 of 154 Next