diff --git a/.gitignore b/.gitignore index 8d1ef24091c15c17031065720a92c5c4be7f9a35..e1790ba75e946bcdf7243a49dec343c7a5045782 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,2 @@ priv +README.pdf diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 520208c4abb56648b1928f091c7ed8c127917af8..39d6c2e94f5dc5212816fff87f969db504492840 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -14,7 +14,6 @@ test: stage: test except: - tags - - pdf@idevelop/howtogitlabrunner script: - whoami - pwd @@ -26,12 +25,10 @@ test: - echo $MY_TOKEN - echo $(date '+%Y%m%d') - build: stage: build except: - tags - - pdf@idevelop/howtogitlabrunner script: - echo "pandocify README" - pandoc README.md -o README.pdf @@ -44,7 +41,6 @@ create_release: stage: deploy except: - tags - - pdf@idevelop/howtogitlabrunner script: - ls -al * - sh deploy.sh @@ -53,7 +49,6 @@ deploy_to_production: stage: deploy except: - tags - - pdf@idevelop/howtogitlabrunner when: manual allow_failure: false script: echo "Manual input needed to deploy to production..." @@ -62,5 +57,4 @@ control: stage: control except: - tags - - pdf@idevelop/howtogitlabrunner script: echo "control, cleanup and finish the pipeline" diff --git a/README.md b/README.md index 6166352a590c9661de078cfe7ea5f0ecf7aade7b..34b20a4b1947fcf3fea207280c66d0f574114767 100644 --- a/README.md +++ b/README.md @@ -4,19 +4,41 @@ The goal of this project is to demonstrate how to use [GitLab-Runner](https://docs.gitlab.com/runner) with a very simple use case: transform this README.md into README.pdf with [pandoc](https://pandoc.org/). -## Setup the runner + +## Preamble + +Doing this demo taught us some important lesson: we first tried to build the +PDF and push it into this repo. While it seems feasible, it not a good idea. +First of all, you want to keep your commits atomic, meaning that you don't want +a commit creates a new commit automatically. Secondly, the CI/CD mechanisms is +here to build and/or deploy a specific commit to other systems/environment, not +modifying the current one. Finally, we choose to do it the following way: + + 1. One modify the README.md file; + 1. It trigger the build to create the README.pdf file; + 1. Then, a release is created, association the README.pdf file to it. + +That way, the things are kept as close as possible to a standard dev/stage/prod +environment. + +Before diving into it, you may want to check the +[documentation](https://docs.gitlab.com/ce/ci/README.html) and some [real world +examples](https://docs.gitlab.com/ce/ci/examples/README.html). + + +## Setup the "shell" runner First of all, you need a runner on a machine. The installation process is described here: https://docs.gitlab.com/runner/install/ -It create the gitlab-runner user on your system. On some Linux, you may want to +It creates the gitlab-runner user on your system. On some Linux, you may want to remove the `~/.bash_logout` file to avoid the issue [4092](https://gitlab.com/gitlab-org/gitlab-runner/issues/4092) — thanks me later. -The last step explain how to register the runner with `sudo gitlab-runner -register`. Answer the questions (the runner key is found on your project > -settings > CI/CD > runner). You may want to check the runner options (for +The last step explains how to register the runner with `sudo gitlab-runner +register`. Answer the questions (the runner key is found on your Project > +Settings > CI/CD > Runner). You may want to check the runner options (for example "Run untagged jobs"). It will create the `/etc/gitlab-runner/config.toml` file which looks like that: @@ -38,4 +60,49 @@ check_interval = 0 [runners.cache.gcs] ``` -## next +## Check and edit the `.gitlab-ci.yaml` + +The whole pipeline/stage/job configuration stands in the `.gitlab-ci.yaml` file. +A lot of documentation is provided by GitLab: https://docs.gitlab.com/ce/ci/. + +In short, this file defines the pipeline process, which contain stages. A stage +can contains jobs, which describes what to do. Jobs can be parallel and manual. + +Note that if you want to keep "produced" files between jobs, you have to use the +artifacts system. + +The file of this project define 4 stages: + - test → just prints out some env variables. + - build → creates the README.pdf. + - deploy → creates the release via the `deploy.sh` script. Need a variable. + - control → final steps, does nothing but echoing a string. + + + + + +## Set the variable + +To be able to use the API to create a release, you need to have a [Personal +Access +Tokens](https://docs.gitlab.com/ce/user/profile/personal_access_tokens.html) +which can be created in your [personal +account](https://gitlab.epfl.ch/profile/personal_access_tokens). In the +`deploy.sh` script, it's used as the `MY_TOKEN` variable. Once you get the +token, there are 2 ways to use it: + + - define it into the project (Project > Settings > CI/CD > Variables), which + means that other persons of the project will use it. + - define it when manually launching the pipeline: only you will be able to + use it but automatic trigger will fail. + +This is OK for this demo or small projects, but a better way to do it is to +create a "bot" user to whom you give some right on your repository and create +the personal access token from it. + +## Observe + +Now it time that you try out the pipeline. Either change something to this file +(I'm sure there's a lot of errors and missing/incorrect information) and push to +the repo, or go into your project CI/CD pipelines and use the "Run pipeline" +button. diff --git a/deploy.sh b/deploy.sh index 7aadf6f32aa9740ca67313387a619eab8f3d1428..5be38d30550305d4d03d6d3daf6dfd559ec2ca65 100755 --- a/deploy.sh +++ b/deploy.sh @@ -1,49 +1,42 @@ #!/usr/bin/env bash -PROJECT_ID=1298 +# +# This script create a release containg the README.pdf file from the +# https://gitlab.epfl.ch/idevelop/howtogitlabrunner/ project. +# set -e -x -env +PROJECT_ID=1298 -echo "Hello from deploy.sh" +# Print out the environment +# env +echo "Hello from deploy.sh" echo "My token is $MY_TOKEN" +# Should display "gitlab-runner" whoami -# -# -# eval `ssh-agent` -# -# ssh-add ~/.ssh/id_rsa -# -# ssh-add -l -# ssh-add -L - -# git clone git@gitlab.epfl.ch:idevelop/howtogitlabrunner.wiki.git wiki -# cp README.pdf wiki/public/ -# echo \n$(date '+%Y-%m-%d %H:%M:%S') >> wiki/home.md -# cd wiki -# git config user.email "gitlab@epfl.ch" -# git config user.name "gitlab-runner" -# git commit -am "New PDF file $(date '+%Y-%m-%d %H:%M:%S')" -# git push --push-option=ci.skip origin master -# https://docs.gitlab.com/ee/ci/yaml/README.html#skipping-jobs + +# Create a release name RELEASE_V=v$(date '+%s') echo "Creating release: ${RELEASE_V}" + +# Tagging the version git tag -f -a -m "My sweet release $(date '+%Y-%m-%d %H:%M:%S')" ${RELEASE_V} -echo "END OF TAGGING" + +# Be sure to have a remote to push to git remote set-url origin "git@gitlab.epfl.ch:idevelop/howtogitlabrunner.git" +# And actually push the tags git push -f --tags -echo "END OF PUSH" +# @domq's trick to be able to see the curl response in stderr tee_stderr="perl -pe STDERR->print(\$_);warn(\$!)if(\$!)" -echo "BEGIN UPLOAD" +# Uploading the README.pdf file UPLOADED_FILE_URL="$(curl --request POST --header "Private-Token: ${MY_TOKEN}" --form "file=@README.pdf" "https://gitlab.epfl.ch/api/v4/projects/$PROJECT_ID/uploads" | $tee_stderr | jq -r .url)" -echo "END UPLOAD" -echo "BEGIN POST RELEASE" +# Create the release until it finally work... for retry in $(seq 1 5); do if curl --request POST \ --header 'Content-Type: application/json' \ @@ -57,4 +50,3 @@ for retry in $(seq 1 5); do sleep 5 fi done -echo "END POST RELEASE" diff --git a/project_pipeline.jpg b/project_pipeline.jpg new file mode 100644 index 0000000000000000000000000000000000000000..95fa2a39695a6c06a19dfccfcb0d408d10d16421 Binary files /dev/null and b/project_pipeline.jpg differ