↓Skip to main content

GitLab CI Pipeline - Push data into AWS S3 Bucket, PowerShell Git Commands

500 words·
GitLab GitLab CI CI Pipeline AWS S3 Git PowerShell
Table of Contents

AWS Prerequisites
#

  • Create a S3 bucket, in this tutorial it’s named jkw-gitlab

  • Create an IAM user

  • Create Access keys for the user

# Access key
AKIAIOSFODNN7EXAMPLE

# Secret access key
wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
  • Create IAM Policy and attach it to the IAM user
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::jkw-gitlab/*", # Define S3 bucket name
                "arn:aws:s3:::jkw-gitlab" # Define S3 bucket name
            ]
        }
    ]
}

GitLab
#

GitLab Project
#

Create a folder s3-data for the files that should be pushed into the S3 bucket.

# Repository folder structure
deploy-into_s3
├── .gitlab-ci.yml
└── s3-data
    ├── File1.txt
    └── File2.txt

GitLab Variables
#

Add the Access keys of your IAM user as variables to the GitLab project.

  • Go to: Settings > CI/CD

  • Expand the Variables section and add the IAM credentials:

# Variable 1
AWS_ACCESS_KEY_ID

# Variable 1 Value
AKIAIOSFODNN7EXAMPLE

# Variable 2
AWS_SECRET_ACCESS_KEY

# Variable 2 Value
wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

GitLab CI Pipeline
#

Add the GitLab CI pipeline to the repository:

# .gitlab-ci.yml
stages:
  - deploy

deploy-into_s3:
  stage: deploy
  image: python:latest
  script:
    - pip install awscli
    - aws s3 cp ./s3-data s3://jkw-gitlab/ --recursive
  only:
    - main

The pipeline job will start automatically and every time new data is pushed to the GitLab repository.


Wait till pipeline completes
#

  • Go to Build > Jobs and click on the job for the logs.
# Job logs
...
$ aws s3 cp ./s3-data s3://jkw-gitlab/ --recursive
upload: s3-data/File1.txt to s3://jkw-gitlab/File1.txt           
upload: s3-data/File2.txt to s3://jkw-gitlab/File2.txt            
Cleaning up project directory and file based variables 00:00
Job succeeded
  • Refresh the browser till the status jumps to Passed

AWS S3 Bucket
#

The files from the GitLab folder s3-data should now be available in the S3 bucket:


Git PowerShell Commands
#

Install Git with Winget
#

  • Open PowerShell as Administrator
# Install Git
winget install Git.Git
  • Close and open a new PowerShell
# Check Git version
git --version

Prerequisites
#

Set the username and email address that will be associated with any commits you create across all Git repositories on your computer.

# Define your name
git config --global user.name "Your Name"

# Define youremail
git config --global user.email "you@example.com"

Repository
#

  • Clone the repository
# Clone the repository
git clone git@gitlab.com:jueklug/deploy-into_s3.git

# CD into the repository
cd .\deploy-into_s3\

# Create main branch & switch into main branch
git switch --create main
  • Commit the “s3-data” folder
# Stages the "s3-data" folder and its contents for the next commit
git add .\s3-data\

# Commit the changes
git commit -m "Added folder"

# Push the commit to the main branch of the remote repository on GitLab
git push origin main
  • Commit the CI pipeline
# Stages the "s3-data" folder and its contents for the next commit
git add .\.gitlab-ci.yml

# Commit the changes
git commit -m "Added CI pipeline"

# Push the commit to the main branch of the remote repository on GitLab
git push origin main
  • Other Commands
# Stage all new and modified files in current folder
git add .