GitLab CI Pipeline - Build and Push Image to AWS Elastic Container Registry, Deploy Container via AWS Elastic Container Service

Table of Contents

Overview
#

This GitLab CI pipeline updates the task definition ARN of an simple ECS service without loadbalancer.

For a more production-ready ECS deployment, take a look at my previous post:

AWS Elastic Container Service (ECS) - Create a Service and ALB LoadBalancer with HTTPS Listener and TLS Certificate using the AWS CLI, Mermaid Flowchart

2327 words

AWS AWS CLI Elastic Container Service (ECS) Application Load Balancer (ALB) Elastic Container Registry CloudWatch Docker Mermaid Flowchart

AWS Prerequisites
#

Create Elastic Container Registry Repository
#

# Create a repository
aws ecr create-repository --repository-name example-application --region eu-central-1

# Shell output:
{
    "repository": {
        "repositoryArn": "arn:aws:ecr:eu-central-1:012345678912:repository/example-application",
        "registryId": "012345678912",
        "repositoryName": "example-application",
        "repositoryUri": "012345678912.dkr.ecr.eu-central-1.amazonaws.com/example-application",
        "createdAt": "2024-12-14T20:57:34.137000+00:00",
        "imageTagMutability": "MUTABLE",
        "imageScanningConfiguration": {
            "scanOnPush": false
        },
        "encryptionConfiguration": {
            "encryptionType": "AES256"
        }
    }
}

Create ECS Cluster
#

# Create a AWS ECS cluster
aws ecs create-cluster --cluster-name jkw-cluster \
    --region eu-central-1 \
    --no-cli-pager

# Shell output:
{
    "cluster": {
        "clusterArn": "arn:aws:ecs:eu-central-1:012345678912:cluster/jkw-cluster",
        "clusterName": "jkw-cluster",
        "status": "ACTIVE",
        "registeredContainerInstancesCount": 0,
        "runningTasksCount": 0,
        "pendingTasksCount": 0,
        "activeServicesCount": 0,
        "statistics": [],
        "tags": [],
        "settings": [
            {
                "name": "containerInsights",
                "value": "disabled"
            }
        ],
        "capacityProviders": [],
        "defaultCapacityProviderStrategy": []
    }
}

IAM User & Permissions
#

Create IAM User
#

Create a new IAM user, the access keys will be added as GitLab CI/CD variable.

# Create a new IAM user with the name "ecr-user"
aws iam create-user --user-name ecr-user

# Shell output:
{
    "User": {
        "Path": "/",
        "UserName": "ecr-user",
        "UserId": "AIDARCHUALINQ4LNLEYU6",
        "Arn": "arn:aws:iam::012345678912:user/ecr-user",
        "CreateDate": "2024-12-14T16:07:17+00:00"
    }
}

Create Access Keys for the User
#

# Create Access Keys for the user
aws iam create-access-key --user-name ecr-user

# Shell output:
{
    "AccessKey": {
        "UserName": "ecr-user",
        "AccessKeyId": "AKIADCHUALINW6GUYMHF",
        "Status": "Active",
        "SecretAccessKey": "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",
        "CreateDate": "2024-12-14T16:07:28+00:00"
    }
}

Copy the Access Key and Secret Access Key:

“AccessKeyId”: AKIADCHUALINW6GUYMHF
“SecretAccessKey”: olBHUmyygalSG0hB3Rbg3t2CCiLzGEBppu1NPfzD

Attach Policy to User
#

Attach the “AmazonEC2ContainerRegistryFullAccess” and “AmazonECS_FullAccess” to the user:

# Attach the policies to the user
aws iam attach-user-policy --user-name ecr-user --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryFullAccess
aws iam attach-user-policy --user-name ecr-user --policy-arn arn:aws:iam::aws:policy/AmazonECS_FullAccess

Verify the User Policy
#

# Confirm the policy is attached to the user
aws iam list-attached-user-policies --user-name ecr-user

# Shell output:
{
    "AttachedPolicies": [
        {
            "PolicyName": "AmazonEC2ContainerRegistryFullAccess",
            "PolicyArn": "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryFullAccess"
        },
        {
            "PolicyName": "AmazonECS_FullAccess",
            "PolicyArn": "arn:aws:iam::aws:policy/AmazonECS_FullAccess"
        }
    ]
}

IAM Role & Permissions
#

ECS Task Execution Role
#

The ecsTaskExecutionRole is required by ECS to pull the container images from the ECR

# Create the role "ecsTaskExecutionRole"
aws iam create-role \
    --role-name ecsTaskExecutionRole \
    --assume-role-policy-document '{
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Service": "ecs-tasks.amazonaws.com"
          },
          "Action": "sts:AssumeRole"
        }
      ]
    }'

Attach Policy
#

Attach the managed AmazonECSTaskExecutionRolePolicy policy to the role:

# Attach policy to role
aws iam attach-role-policy \
    --role-name ecsTaskExecutionRole \
    --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy

Verify the Role
#

# List role details
aws iam get-role --role-name ecsTaskExecutionRole

# Shell output:
{
    "Role": {
        "Path": "/",
        "RoleName": "ecsTaskExecutionRole",
        "RoleId": "AROARCHUALIN46RDYSU3J",
        "Arn": "arn:aws:iam::012345678912:role/ecsTaskExecutionRole",
        "CreateDate": "2024-12-12T17:27:12+00:00",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "Service": "ecs-tasks.amazonaws.com"
                    },
                    "Action": "sts:AssumeRole"
                }
            ]
        },
        "MaxSessionDuration": 3600,
        "RoleLastUsed": {}
    }
}

Create CloudWatch Log Group
#

# Create a Cloudwatch log group
aws logs create-log-group \
    --log-group-name /ecs/example-task \
    --region eu-central-1

List Subnets and SecurityGroup
#

Find Default VPC
#

# Find default Subnet
aws ec2 describe-vpcs \
    --filters Name=isDefault,Values=true \
    --region eu-central-1 \
    --query "Vpcs[0].VpcId" --output text

# Shell output:
vpc-026ffff23b41d0806

List Security Groups
#

# List security groups
aws ec2 describe-security-groups \
    --filters Name=vpc-id,Values=vpc-026ffff23b41d0806 \
    --region eu-central-1 \
    --query "SecurityGroups[?GroupName=='default'].[GroupId,GroupName]" --output table

# Shell output:
-------------------------------------
|      DescribeSecurityGroups       |
+-----------------------+-----------+
|  sg-07ce383e71b926210 |  default  |
+-----------------------+-----------+

Describe Default Security Group
#

# List security group details
aws ec2 describe-security-groups \
    --group-ids sg-07ce383e71b926210 \
    --region eu-central-1

# Shell output:
{
    "SecurityGroups": [
        {
            "GroupId": "sg-07ce383e71b926210",
            "IpPermissionsEgress": [
                {
                    "IpProtocol": "-1",
                    "UserIdGroupPairs": [],
                    "IpRanges": [
                        {
                            "CidrIp": "0.0.0.0/0"
                        }
                    ],
                    "Ipv6Ranges": [],
                    "PrefixListIds": []
                }
            ],
            "VpcId": "vpc-026ffff23b41d0806",
            "SecurityGroupArn": "arn:aws:ec2:eu-central-1:012345678912:security-group/sg-07ce383e71b926210",
            "OwnerId": "012345678912",
            "GroupName": "default",
            "Description": "default VPC security group",
            "IpPermissions": [
                {
                    "IpProtocol": "-1",
                    "UserIdGroupPairs": [
                        {
                            "UserId": "012345678912",
                            "GroupId": "sg-07ce383e71b926210"
                        }
                    ],
                    "IpRanges": [],
                    "Ipv6Ranges": [],
                    "PrefixListIds": []
                }
            ]
        }
    ]
}

Add Inbound Rule
#

# Allow inbound traffic on port "80" / all sources
aws ec2 authorize-security-group-ingress \
    --group-id sg-07ce383e71b926210 \
    --protocol tcp \
    --port 80 \
    --cidr 0.0.0.0/0 \
    --region eu-central-1

Verify the Inbound Rule
#

# Verify the inbound rule
aws ec2 describe-security-groups \
   --group-ids sg-07ce383e71b926210 \
   --region eu-central-1 \
   --query "SecurityGroups[].IpPermissions[*].{Protocol:IpProtocol,From:FromPort,To:ToPort,Source:IpRanges[].CidrIp}" \
   --output table

# Shell output:
------------------------------
|   DescribeSecurityGroups   |
+-------+-------------+------+
| From  |  Protocol   | To   |
+-------+-------------+------+
|  80   |  tcp        |  80  |
+-------+-------------+------+
||          Source          ||
|+--------------------------+|
||  0.0.0.0/0               ||
|+--------------------------+|
+------+------------+--------+
| From | Protocol   |  To    |
+------+------------+--------+
|  None|  -1        |  None  |
+------+------------+--------+

List Default Subnets
#

List the default subnets:

# List the default VPC in the "eu-central-1" region
aws ec2 describe-vpcs --query "Vpcs[?IsDefault==\`true\`].VpcId" --output text --region eu-central-1

# Shell output:
vpc-026ffff23b41d0806

# List default subnets
aws ec2 describe-subnets --filters Name=vpc-id,Values=vpc-026ffff23b41d0806 --query "Subnets[].SubnetId" --output text --region eu-central-1

# Shell output:
subnet-0df2b184637047097        subnet-0c19f580c157ebabb        subnet-0fae44838d92a6612

Initial Task Definition
#

# Create a configuration file for the task definition
vi task-definition.json

{
  "family": "example-app",
  "executionRoleArn": "arn:aws:iam::012345678912:role/ecsTaskExecutionRole",
  "networkMode": "awsvpc",
  "containerDefinitions": [
    {
      "name": "example-app",
      "image": "012345678912.dkr.ecr.eu-central-1.amazonaws.com/example-application",
      "memory": 512,
      "cpu": 256,
      "portMappings": [
        {
          "containerPort": 80,
          "hostPort": 80,
          "protocol": "tcp"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/example-task",
          "awslogs-region": "eu-central-1",
          "awslogs-stream-prefix": "example"
        }
      }
    }
  ],
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "256",
  "memory": "512"
}

# Register the task definition
aws ecs register-task-definition --cli-input-json file://task-definition.json --region eu-central-1 --no-cli-pager

Create ECS Service
#

Use the default SecurityGroup ID, define one or more of the subnets and create the service:

# Create ECS service
aws ecs create-service \
    --cluster jkw-cluster \
    --service-name example-service \
    --task-definition arn:aws:ecs:eu-central-1:012345678912:task-definition/example-app:1 \
    --desired-count 1 \
    --launch-type FARGATE \
    --network-configuration "awsvpcConfiguration={subnets=[subnet-0df2b184637047097],securityGroups=[sg-07ce383e71b926210],assignPublicIp=ENABLED}" \
    --region eu-central-1 \
    --no-cli-pager

Note: Since the GitLab CI pipeline has not yet pushed a container image into the ECR, the service will fail, until the first time the CI pipeline run through.

GitLab Repository
#

File and Folder Structure
#

The file and folder structure of the GitLab repository looks like this:

├── Dockerfile  # Example Dockerfile
├── .gitlab-ci.yml  # GitLab CI pipeline manifest
├── index.html  # Example HTML file
└── task-definition.json  # Task definition

CI/CD Variables
#

Go to: (Project) “Settings” > “CI/CD”
Expand the “Variables” section
Click “Add variable”
Select type “Variable (default)”
Unflag the “Protect variable” option

Add the following variables:

Key: AWS_ACCOUNT_ID Value: 012345678912
Key: AWS_REGION Value: eu-central-1
Key: AWS_ACCESS_KEY_ID Value: AKIADCHUALINW6GUYMHF
key: AWS_SECRET_ACCESS_KEY Value: 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

CI Pipeline Manifest
#

.gitlab-ci.yml

variables:
  AWS_ACCOUNT_ID: 012345678912  # AWS account ID
  AWS_REGION: eu-central-1 # AWS region
  IMAGE_NAME: example-application # Name of the ECR repository
  CLUSTER_NAME: jkw-cluster # Name of the ECS cluster
  SERVICE_NAME: example-service # Name of the ECS service
  # Images names
  TAG_LATEST: $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$IMAGE_NAME:latest
  TAG_COMMIT: $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$IMAGE_NAME:$CI_COMMIT_SHORT_SHA


### Stages
stages:
  - build
  - deploy


### Build image and push it to AWS ECR repository
build_image:
  image: docker:stable
  stage: build
  services:
    - name: docker:23.0.6-dind
      command: ["--tls=false"]
  variables:
    DOCKER_TLS_CERTDIR: ""
  before_script:
    # Install AWS CLI
    - apk add --no-cache python3 py3-pip
    - pip3 install --no-cache-dir awscli
  script:
    # Login the AWS ECR
    - aws ecr get-login-password --region $AWS_REGION |
      docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com
    # Try pulling the latest image, keep going if it does not exist
    - docker pull $TAG_LATEST || true
    # Build image
    - docker build --cache-from $TAG_LATEST -t $TAG_COMMIT -t $TAG_LATEST -f Dockerfile .
    # Push image to AWS ECR repository
    - docker push $TAG_COMMIT
    - docker push $TAG_LATEST
  rules: 
    # Rule: Run this job only for the main branch and if the specified Dockerfile exists
    - if: $CI_COMMIT_BRANCH == "main"
      exists:
        - Dockerfile


### Deploy the container to ECS
deploy_to_ecs:
  image: docker:stable
  stage: deploy
  services:
    - name: docker:23.0.6-dind
      command: ["--tls=false"]
  variables:
    DOCKER_TLS_CERTDIR: ""
  before_script:
    - apk add --no-cache python3 py3-pip jq
    - pip3 install --no-cache-dir awscli
  script:
    # Configure AWS CLI
    - aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID
    - aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY
    - aws configure set region $AWS_REGION

    # Update the task definition with the new image tag
    - jq --arg IMAGE "$TAG_COMMIT" '.containerDefinitions[0].image = $IMAGE' task-definition.json > updated-task-definition.json

    # Register the updated task definition with ECS
    - TASK_DEF_ARN=$(aws ecs register-task-definition --cli-input-json file://updated-task-definition.json | jq -r '.taskDefinition.taskDefinitionArn')

    # Update ECS service with the new task definition
    - aws ecs update-service --cluster $CLUSTER_NAME --service $SERVICE_NAME --task-definition $TASK_DEF_ARN --region $AWS_REGION
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

Task Definition
#

task-definition.json

{
  "family": "example-app",
  "executionRoleArn": "arn:aws:iam::012345678912:role/ecsTaskExecutionRole",
  "networkMode": "awsvpc",
  "containerDefinitions": [
    {
      "name": "example-app",
      "image": "012345678912.dkr.ecr.eu-central-1.amazonaws.com/example-application",
      "memory": 512,
      "cpu": 256,
      "portMappings": [
        {
          "containerPort": 80,
          "hostPort": 80,
          "protocol": "tcp"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/example-task",
          "awslogs-region": "eu-central-1",
          "awslogs-stream-prefix": "example"
        }
      }
    }
  ],
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "256",
  "memory": "512"
}

Example Application
#

Dockerfile
#

Dockerfile

# Use the official Caddy image as the base
FROM caddy:alpine

# Create a non-root user "caddy"
RUN addgroup -S caddy && adduser -S -G caddy caddy

# Adjust permissions
RUN mkdir -p /usr/share/caddy && \
    chown -R caddy:caddy /usr/share/caddy /config /data

# Copy website files into the container
ADD index.html /usr/share/caddy/

# Switch to the non-root user
USER caddy

# Expose the default Caddy port
EXPOSE 80

HTML File
#

index.html

<!DOCTYPE html>
<html>

<head>
    <title>jklug.work</title>
</head>

<body>
    <h1>Some HTML</h1>
    <p>Example Application<br></p>

</body>

</html>

Verify the Deployment
#

List Running Tasks
#

# List running tasks from the "example-service" service
aws ecs list-tasks \
  --cluster jkw-cluster \
  --service-name example-service \
  --desired-status RUNNING \
  --region eu-central-1

# Shell output:
{
    "taskArns": [
        "arn:aws:ecs:eu-central-1:012345678912:task/jkw-cluster/2ec54510cb684352bf881631add5c4d3"
    ]
}

List Task Details / Find ENI
#

# List task details
aws ecs describe-tasks \
  --cluster jkw-cluster \
  --tasks arn:aws:ecs:eu-central-1:012345678912:task/jkw-cluster/2ec54510cb684352bf881631add5c4d3 \
  --region eu-central-1 \
  --no-cli-pager

Copy the ENI from the “networkInterfaceId”:

"name": "networkInterfaceId",
"value": "eni-086521e98cf530468"

Get the Public IP from the ENI
#

# Retrieve the public IP from the ENI
aws ec2 describe-network-interfaces \
  --network-interface-ids eni-086521e98cf530468 \
  --query 'NetworkInterfaces[0].Association.PublicIp' \
  --output text \
  --region eu-central-1

# Shell output:
63.177.64.112

Verify the Task / Container
#

# Curl the container
curl 63.177.64.112:80

# Shell output:
<!DOCTYPE html>
<html>

<head>
    <title>jklug.work</title>
</head>

<body>
    <h1>Some HTML</h1>
    <p>Example Application for AWS ECS<br></p>

</body>

</html>

Cleanup
#

# Delete the Service
aws ecs delete-service \
    --cluster jkw-cluster \
    --service example-service \
    --force \
    --region eu-central-1 \
    --no-cli-pager

Delete ECS Cluster
#

# Delete the ECS cluster
aws ecs delete-cluster \
    --cluster jkw-cluster \
    --region eu-central-1

Delete ECR Repository
#

# Delete the ECR repository
aws ecr delete-repository --repository-name example-application --region eu-central-1 --force

Delete the CloudWatch Log Group
#

# Delete the log group
aws logs delete-log-group \
    --log-group-name /ecs/example-task \
    --region eu-central-1

Delete IAM User
#

# Detach the policies attached to the user
aws iam detach-user-policy --user-name ecr-user --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryFullAccess
aws iam detach-user-policy --user-name ecr-user --policy-arn arn:aws:iam::aws:policy/AmazonECS_FullAccess

# List the access keys for the user
aws iam list-access-keys --user-name ecr-user

# Shell output:
{
    "AccessKeyMetadata": [
        {
            "UserName": "ecr-user",
            "AccessKeyId": "AKIADCHUALINW6GUYMHF",
            "Status": "Active",
            "CreateDate": "2024-12-14T16:07:28+00:00"
        }
    ]
}

# Delete the access keys
aws iam delete-access-key --user-name ecr-user --access-key-id AKIADCHUALINW6GUYMHF

# Delete the "ecr-user" IAM user
aws iam delete-user --user-name ecr-user

Delete Task Definition
#

List Task Definitions
#

# List ECS task definitions
aws ecs list-task-definitions --region eu-central-1

# Shell output:
{
    "taskDefinitionArns": [
        "arn:aws:ecs:eu-central-1:012345678912:task-definition/example-app:1",
        "arn:aws:ecs:eu-central-1:012345678912:task-definition/example-app:2"
    ]
}

Deregister Task Definition
#

# Deregister the task definition "example-app:1"
aws ecs deregister-task-definition \
    --task-definition example-app:1 \
    --region eu-central-1 \
    --no-cli-pager

# Deregister the task definition "example-app:2"
aws ecs deregister-task-definition \
    --task-definition example-app:2 \
    --region eu-central-1 \
    --no-cli-pager

Overview #

AWS Prerequisites #

Create Elastic Container Registry Repository #

Create ECS Cluster #

IAM User & Permissions #

Create IAM User #

Create Access Keys for the User #

Attach Policy to User #

Verify the User Policy #

IAM Role & Permissions #

ECS Task Execution Role #

Attach Policy #

Verify the Role #

Create CloudWatch Log Group #

List Subnets and SecurityGroup #

Find Default VPC #

List Security Groups #

Describe Default Security Group #

Add Inbound Rule #

Verify the Inbound Rule #

List Default Subnets #

Initial Task Definition #

Create ECS Service #

GitLab Repository #

File and Folder Structure #

CI/CD Variables #

CI Pipeline Manifest #

Task Definition #

Example Application #

Dockerfile #

HTML File #

Verify the Deployment #

List Running Tasks #

List Task Details / Find ENI #

Get the Public IP from the ENI #

Verify the Task / Container #

Cleanup #

Delete ECS Cluster #

Delete ECR Repository #

Delete the CloudWatch Log Group #

Delete IAM User #

Delete Task Definition #

List Task Definitions #

Deregister Task Definition #