Building a Simple CI/CD Pipeline with Docker, EC2, GitHub Actions, and ECR 🚀🐳

Mladen Radivojevic

Mladen Radivojevic

— 6 min read
Building a Simple CI/CD Pipeline with Docker, EC2, GitHub Actions, and ECR 🚀🐳
Pipelines

In this guide, we’ll create a fully automated CI/CD pipeline to deploy a Dockerized Hello World app. We’ll use GitHub Actions to build and push Docker images to AWS Elastic Container Registry (ECR) and deploy updates on an AWS EC2 instance through a Python webhook listener. Additionally, we’ll implement resource cleanup on both EC2 and ECR to maintain an efficient deployment system.

Whether you’re a DevOps enthusiast or just starting, this walkthrough offers valuable insights into building an automated deployment pipeline. Let’s dive in! 🛠️

Overview of the Workflow

  1. Write and containerize a Hello World app using Flask and Docker.
  2. Configure AWS Elastic Container Registry (ECR) to store Docker images.
  3. Set up GitHub Actions for CI/CD, including permissions and secrets.
  4. Deploy the Docker container on an EC2 instance using a Python webhook listener.
  5. Automate cleanup of unused Docker images on EC2 and outdated images in ECR.

Step 1: Create the Hello World App

We’ll start by building a simple Python Flask app.

Project Structure

hello-world/  
├── app.py  
├── Dockerfile  
├── requirements.txt  
└── .github/workflows/deploy.yaml

Write the Flask App (app.py)

from flask import Flask  

app = Flask(__name__)  

@app.route("/")  
def hello_world():  
    return "Hello, World! Your CI/CD pipeline is live!"  

if __name__ == "__main__":  
    app.run(host="0.0.0.0", port=5000)

Define Dependencies (requirements.txt)

flask==2.2.2

Create a Dockerfile

FROM python:3.9-slim  

WORKDIR /app  

COPY . .  

RUN pip install -r requirements.txt  

EXPOSE 5000  

CMD ["python", "app.py"]

Step 2: Configure AWS Elastic Container Registry (ECR)

Create an ECR Repository

Via AWS Console:

  1. Go to the ECR section in the AWS Management Console.
  2. Click Create Repository and name it hello-world.

Via AWS CLI:

aws ecr create-repository --repository-name hello-world

Add an ECR Lifecycle Policy (Optional)

To automatically clean up old images:

Via AWS Console:

  1. Navigate to the repository.
  2. Go to the Lifecycle Policy tab and create a rule to retain the last 2 images.

Via AWS CLI:

  1. Save the following JSON as lifecycle-policy.json:
{  
    "rules": [  
        {  
            "rulePriority": 1,  
            "description": "Retain only last 2 images",  
            "selection": {  
                "tagStatus": "any",  
                "countType": "imageCountMoreThan",  
                "countNumber": 2  
            },  
            "action": {  
                "type": "expire"  
            }  
        }  
    ]  
}
  1. Apply the policy:
aws ecr put-lifecycle-policy --repository-name hello-world --lifecycle-policy-text file://lifecycle-policy.json

Step 3: Set Up the EC2 Instance

Launch an EC2 Instance

  1. Log in to the AWS Management Console.
  2. Navigate to EC2 > Instances > Launch Instance.
  3. Choose an AMI, such as Amazon Linux 2 or Ubuntu.
  4. Select an instance type, such as t2.micro (free tier eligible).
  5. In the Configure Security Group section:
    • Create or select a security group.
    • Add inbound rules for ports 80, 5000 and 6000:
      • HTTP (port 80): Source 0.0.0.0/0.
      • Custom TCP (port 5000): Source 0.0.0.0/0.
      • Custom TCP (port 6000): Source 0.0.0.0/0.
  6. Complete the remaining setup (key pair, storage, etc.) and launch the instance.

Install Docker on the Instance

After launching the instance:

  1. SSH into the instance:
ssh -i <your-key-file.pem> ec2-user@<your-ec2-public-ip>
  1. Install Docker:
sudo apt update  
sudo apt install -y docker.io  
sudo systemctl start docker  
sudo usermod -aG docker $USER

Assign IAM Role for ECR Access

Create a New IAM Role

  1. Navigate to IAM > Roles.
  2. Click Create Role and choose AWS Service > EC2.
  3. Attach the AmazonEC2ContainerRegistryReadOnly policy.
  4. Name the role (e.g., ECRAccessRole) and create it.

Assign the Role to the EC2 Instance

  1. Navigate to EC2 > Instances.
  2. Select the instance, then go to Actions > Security > Modify IAM Role.
  3. Select the ECRAccessRole and save.

Step 4: Configure GitHub Actions

Create an IAM Role for GitHub Actions

This role will be assumed by GitHub actions, we need to create custom trust policy so this role can be assumed from our repository.

  1. Navigate to IAM > Roles and click Create Role.
  2. Choose Custom trust policy and add this JSON:
{  
  "Version": "2012-10-17",  
  "Statement": [  
    {  
      "Effect": "Allow",  
      "Principal": {  
        "Federated": "arn:aws:iam::<YOUR_ACCOUNT_ID>:oidc-provider/token.actions.githubusercontent.com"  
      },  
      "Action": "sts:AssumeRoleWithWebIdentity",  
      "Condition": {  
        "StringEquals": {  
          "token.actions.githubusercontent.com:sub": "repo:<your-github-account>/<your-repo>:ref:refs/heads/main",  
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"  
        }  
      }  
    }  
  ]  
}
  1. Replace <YOUR_ACCOUNT_ID>, <your-github-account>, and <your-repo> with your actual values.
  2. Attach the AmazonEC2ContainerRegistryFullAccess policy.
  3. Name the role (e.g., GitHubActionsRole) and save.

Generate the Webhook Secret

A webhook secret ensures secure communication between GitHub Actions and the webhook listener. To generate a secret:

Run the following command on your local machine or any secure environment:

openssl rand -hex 32

Example output:

4b0c57c96d9c4be7d94c403f5474b8e27b26f6b582f49676937889c1464b59ea

Copy the generated secret for the next steps.

Update/Create the GitHub Actions Workflow

.github/workflows/deploy.yaml

Before running workflow and accessing secrets, we need to add them to our repository. To do that, we need to:

  • Navigate to our repository
  • Navigate to Settings > Secrets and Variables > Actions
  • Add following secrets and their values:
    • AWS_ROLE_ARN (Role that we created)
    • AWS_REGION
    • AWS_ACCOUNT_ID (AWS account ID)
    • WEBHOOK_SECRET (Secret that we generated in previous step)
name: Build and Deploy Docker Image  

on:  
  push:  
    branches:  
      - main  

jobs:  
  build-and-push:  
    runs-on: ubuntu-latest  

    steps:  
    - name: Checkout repository  
      uses: actions/checkout@v3  

    - name: Configure AWS Credentials  
      uses: aws-actions/configure-aws-credentials@v4  
      with:  
        role-to-assume: ${{ secrets.AWS_ROLE_ARN }}  
        aws-region: ${{ secrets.AWS_REGION }}  

    - name: Log in to Amazon ECR  
      run: |  
        aws ecr get-login-password --region ${{ secrets.AWS_REGION }} | docker login --username AWS --password-stdin ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com  

    - name: Build and push Docker image  
      run: |  
        docker build -t hello-world .  
        docker tag hello-world:latest ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com/hello-world:latest  
        docker push ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com/hello-world:latest  

    - name: Trigger deployment  
      run: |  
        curl -X POST http://<your-ec2-public-ip>:6000/webhook \  
          -d '{"secret": "${{ secrets.WEBHOOK_SECRET }}"}' \  
          -H "Content-Type: application/json"

Step 5: Write and Start the Webhook Listener

Add the webhook secret as an environment variable:

We need to add webhook secret we generated to as an enivornment variable, so it can be accessed by webhook listener.

Use the echo command to append the secret to the shell configuration file (e.g., .bashrc for Amazon Linux or Ubuntu).

echo "export WEBHOOK_SECRET=4b0c57c96d9c4be7d94c403f5474b8e27b26f6b582f49676937889c1464b59ea" >> ~/.bashrc

Load the environment variable into the current session:

source ~/.bashrc

Verify that the secret is accessible:

echo $WEBHOOK_SECRET

You should see the secret printed in the terminal.

Create the Listener

On the EC2 instance, create webhook_listener.py:

from flask import Flask, request, jsonify  
import subprocess  
import os  

app = Flask(__name__)  
SECRET = os.getenv("WEBHOOK_SECRET")  

@app.route("/webhook", methods=["POST"])  
def webhook():  
    data = request.get_json()  
    if not data or data.get("secret") != SECRET:  
        return jsonify({"error": "Unauthorized"}), 403  

    subprocess.run(["aws", "ecr", "get-login-password", "--region", "your-region"], check=True)  
    subprocess.run(["docker", "pull", "<your-account-id>.dkr.ecr.<region>.amazonaws.com/hello-world:latest"])  
    subprocess.run(["docker", "rm", "-f", "hello-world"], check=False)  
    subprocess.run(["docker", "run", "-d", "--name", "hello-world", "-p", "80:5000",  
                    "<your-account-id>.dkr.ecr.<region>.amazonaws.com/hello-world:latest"])  

    return jsonify({"status": "Deployment successful!"})  

if __name__ == "__main__":  
    app.run(host="0.0.0.0", port=6000)

Start the Webhook Listener

  1. Install Flask:
pip3 install flask
  1. Run the listener in the background:
nohup python3 webhook_listener.py &

Step 6: Automate Cleanup

Cleanup on EC2

Add a cron job to clean up unused Docker images:

crontab -e  
# Add the following line  
0 2 * * * docker image prune -f

Wrapping Up

🎉 Congratulations! You’ve successfully built an automated CI/CD pipeline to deploy Dockerized applications using GitHub Actions, AWS ECR, and EC2.

Test the Pipeline

Now that everything is set up, you can test the pipeline:

  1. Make a change in your codebase, such as updating the message in app.py:
@app.route("/")  
def hello_world():  
    return "Hello, World! The CI/CD pipeline is working!"
  1. Commit and push the changes to the main branch:
git add app.py  
git commit -m "Update message for pipeline test"  
git push origin main
  1. Watch the pipeline in action:
    • Go to your GitHub repository and navigate to the Actions tab.
    • You should see the workflow triggered for the main branch.
    • The workflow will:
      • Build and push the updated Docker image to AWS ECR.
      • Trigger the webhook listener on the EC2 instance.
  2. Verify the Deployment:
    • Open your web browser and navigate to the public IP address of your EC2 instance.
    • You should see the updated message from the Flask application.

With this setup, your pipeline is ready to handle automated deployments efficiently. Push changes confidently, and let automation do the heavy lifting! 🚀

Read more