Skip to content

Commit 4c39994

Browse files
jeffmccunejeffmccune
committed
Add Gitlab CI jobs
Without this patch the control repository does not contain any scripts to execute CI jobs for Puppet code deployment. This patch addresses the problem by adding a Gitlab CI job configuration with three jobs defined across two stages. The test stage executes first with a Syntax and a Lint job executing in parallel. If an environment branch has been pushed, one matching `developemnt`, `testing`, `production` or starting with the prefix `playground`, the deploy stage is executed. The deploy stage requires a Gitlab secret environment variable, PUPPET_TOKEN. This environment variable is the puppet access token configured with Code Manager. The test stage runs the following syntax checks: * yaml files (*.yml, *.yaml) * json files (*.json) * bash scripts (*.sh) * ruby scripts (*.rb) * puppet manifests (*.pp) * erb files (*.erb) The behavior of the checks are to check only files modified relative to a base branch, defaulting to `production`. This avoids running syntax checks against files which have not been modified by the merge request. The check uses `git diff --name-status` to identify changed files across multiple commits in a topic branch. The lint checks rely on bundler and the Gemfile to install numerous lint checks. Gem libraries are installed in a per-job location in $HOME. This is an intentional compromise between installing into a system GEM_PATH element, which would create library conflicts with other jobs, and the job workspace, which would cause gem libraries to be installed from the network on each job invocation.
1 parent ac97852 commit 4c39994

File tree

5 files changed

+266
-0
lines changed

5 files changed

+266
-0
lines changed

‎.gitlab-ci.yml‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
---
2+
stages:
3+
- test
4+
- deploy
5+
6+
# Jobs are defined below
7+
Syntax Check:
8+
tags:
9+
- puppet agent
10+
stage: test
11+
script:
12+
- scripts/syntax_check.sh
13+
14+
Lint Check:
15+
tags:
16+
- puppet agent
17+
stage: test
18+
script:
19+
- scripts/lint_check.sh
20+
21+
# Unless TCP port 8170 is open to the PE master, this job must execute on the
22+
# PE Monolithic Master itself.
23+
Deploy to Puppet:
24+
tags:
25+
- puppet monolithic master
26+
stage: deploy
27+
variables:
28+
URL: https://puppet:8170/code-manager
29+
only:
30+
- development
31+
- testing
32+
- production
33+
- /^playground/
34+
script:
35+
- scripts/puppet_deploy.sh
36+
37+
# vim:tabstop=2
38+
# vim:shiftwidth=2
39+
# vim:expandtab

‎Gemfile‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
source ENV['GEM_SOURCE'] || 'https://rubygems.org'
2+
3+
if puppetversion = ENV['PUPPET_GEM_VERSION']
4+
gem 'puppet', puppetversion, require: false
5+
else
6+
gem 'puppet', require: false
7+
end
8+
9+
gem 'puppet-lint', '~> 2.0'
10+
gem 'puppet-lint-absolute_classname-check'
11+
gem 'puppet-lint-alias-check'
12+
gem 'puppet-lint-empty_string-check'
13+
gem 'puppet-lint-file_ensure-check'
14+
gem 'puppet-lint-file_source_rights-check'
15+
gem 'puppet-lint-leading_zero-check'
16+
gem 'puppet-lint-trailing_comma-check'
17+
gem 'puppet-lint-undef_in_function-check'
18+
gem 'puppet-lint-unquoted_string-check'
19+
gem 'puppet-lint-variable_contains_upcase'

‎scripts/lint_check.sh‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
#!/bin/bash
2+
3+
# Error out if there are any failures
4+
set -e
5+
set -o pipefail
6+
set -u
7+
8+
# Notes
9+
# xargs -P2 is used to run 2 parallel processes at once. This speeds up
10+
# performance on multi-core systems.
11+
12+
if [ -e /proc/cpuinfo ]; then
13+
cores=$(awk 'BEGIN { c = 0 }; $1 == "processor" { c++ }; END { print c }' /proc/cpuinfo)
14+
else
15+
cores=2
16+
fi
17+
18+
# Use Puppet Enterprise Ruby to check ruby and yaml files
19+
export PATH="/opt/puppetlabs/puppet/bin:/opt/puppetlabs/bin:$PATH"
20+
21+
# Localize Gems on a per-job basis to prevent conflicts
22+
gem_home="$(gem env gempath | cut -d: -f1)"
23+
# Trim off the leading part of $HOME
24+
gem_suffix=${gem_home##*/.gem/}
25+
# Set GEM_HOME to a job specific location
26+
export GEM_HOME="${HOME}/jobs/${CI_JOB_NAME:-lint}/gem/${gem_suffix}"
27+
28+
# If we need to install a gem, do so into HOME
29+
# e.g. /home/gitlab-runner/.gem/ruby/2.1.0
30+
export PATH="${GEM_HOME}/bin:$PATH"
31+
32+
echo '######## BEGIN DEPENDENCY SETUP #########'
33+
34+
# Display the gem environment
35+
gem env
36+
37+
if ! (which bundle 2>&1 >/dev/null); then
38+
gem install bundler --no-ri --no-rdoc
39+
fi
40+
41+
# List the files changes from $BASEBRANCH on stdout
42+
files_changed() {
43+
# File status flags:
44+
# M modified - File has been modified
45+
# C copy-edit - File has been copied and modified
46+
# R rename-edit - File has been renamed and modified
47+
# A added - File has been added
48+
# D deleted - File has been deleted
49+
# U unmerged - File has conflicts after a merge
50+
git diff --name-status "${BASEBRANCH:=production}" \
51+
| awk '$1 ~ /^[MCRA]$/' \
52+
| cut -f2-
53+
}
54+
55+
# Install dependencies
56+
bundle install
57+
58+
echo '######## END DEPENDENCY SETUP #########'
59+
echo
60+
echo
61+
echo '######## BEGIN LINT CHECKS #########'
62+
# Lint only the manifest files changed
63+
files_changed \
64+
| awk '/manifests\/.*\.(pp)$/' \
65+
| xargs --no-run-if-empty -t -P$cores -n1 \
66+
bundle exec puppet-lint
67+
68+
echo '######## END LINT CHECKS #########'
69+
70+
# vim:tabstop=2
71+
# vim:shiftwidth=2
72+
# vim:expandtab

‎scripts/puppet_deploy.sh‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
#!/bin/bash
2+
set -u
3+
4+
export PATH="/opt/puppetlabs/bin:$PATH"
5+
# Allow these environment variable to be overriden
6+
: ${URL:='https://puppet:8170/code-manager/v1/deploys'}
7+
# CI_BUILD_REF_NAME is a variable set by gitlab
8+
: ${ENVIRONMENT:="$CI_BUILD_REF_NAME"}
9+
10+
err() {
11+
echo "$1" >&2
12+
}
13+
14+
if [ -z "${PUPPET_TOKEN:-}" ]; then
15+
err "ERROR: PUPPET_TOKEN environment variable must be set!"
16+
err "SUGGESTION: Did you push to origin instead of upstream?"
17+
err "PUPPET_TOKEN must be set as an environment variable in CI"
18+
exit 1
19+
fi
20+
21+
if ! [ -x /opt/puppetlabs/bin/puppet-code ]; then
22+
err "ERROR: /opt/puppetlabs/bin/puppet-code does not exist"
23+
err "SUGGESTION: Install the puppet client tools"
24+
err "https://docs.puppet.com/pe/2016.4/install_pe_client_tools.html#install-on-a-linux-workstation"
25+
exit 2
26+
fi
27+
28+
# Save the token to a temporary file so we can use it with puppet code deploy
29+
scratch="$(mktemp -d)"
30+
remove_scratch() {
31+
[ -e "${scratch:-}" ] && rm -rf "$scratch"
32+
}
33+
trap remove_scratch EXIT
34+
# Subsequent calls to mktemp should be inside our scratch dir
35+
export TMPDIR="$scratch"
36+
37+
tokenfile="$(mktemp)"
38+
echo -n "$PUPPET_TOKEN" > "$tokenfile"
39+
40+
# Turn on debug logging after the token has been written to the file system
41+
set -x
42+
# Deploy the code
43+
puppet-code deploy \
44+
--service-url "$URL" \
45+
--token-file "$tokenfile" \
46+
--wait "${ENVIRONMENT}"
47+
rval=$?
48+
set +x
49+
50+
if [ $rval -ne 0 ]; then
51+
echo "ERROR: puppet-code deploy failed with exit code $rval" >&2
52+
exit $rval
53+
fi
54+
55+
echo "Exiting with exit value $rval"
56+
exit $rval
57+
58+
# vim:tabstop=2
59+
# vim:shiftwidth=2
60+
# vim:expandtab

‎scripts/syntax_check.sh‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
#!/bin/bash
2+
3+
# Error out if there are any failures
4+
set -e
5+
set -o pipefail
6+
set -u
7+
8+
# Notes
9+
# xargs -P2 is used to run 2 parallel processes at once. This speeds up
10+
# performance on multi-core systems.
11+
12+
if [ -e /proc/cpuinfo ]; then
13+
cores=$(awk 'BEGIN { c = 0 }; $1 == "processor" { c++ }; END { print c }' /proc/cpuinfo)
14+
else
15+
cores=2
16+
fi
17+
18+
# Use Puppet Enterprise Ruby to check ruby and yaml files
19+
export PATH="/opt/puppetlabs/puppet/bin:$PATH"
20+
21+
# List the files changes from $BASEBRANCH on stdout
22+
files_changed() {
23+
# File status flags:
24+
# M modified - File has been modified
25+
# C copy-edit - File has been copied and modified
26+
# R rename-edit - File has been renamed and modified
27+
# A added - File has been added
28+
# D deleted - File has been deleted
29+
# U unmerged - File has conflicts after a merge
30+
git diff --name-status "${BASEBRANCH:=production}" \
31+
| awk '$1 ~ /^[MCRA]$/' \
32+
| cut -f2-
33+
}
34+
35+
# Check the Puppetfile
36+
echo -n "Checking Puppetfile ... "
37+
ruby -c Puppetfile
38+
39+
files_changed \
40+
| awk '/\.(sh)$/' \
41+
| xargs --no-run-if-empty -t -P$cores -n1 \
42+
bash -n
43+
44+
# Check all YAML files
45+
# See: http://stackoverflow.com/questions/3971822/yaml-syntax-validator
46+
files_changed \
47+
| awk '/\.(yml|yaml)$/' \
48+
| xargs --no-run-if-empty -t -P$cores -n1 \
49+
ruby -r yaml -e 'YAML.load_file(ARGV[0])'
50+
51+
# Check all JSON files
52+
files_changed \
53+
| awk '/\.(json)$/' \
54+
| xargs --no-run-if-empty -t -P$cores -n1 \
55+
ruby -r json -e 'JSON.load(File.read(ARGV[0]))'
56+
57+
files_changed \
58+
| awk '/\.(rb)$/' \
59+
| xargs --no-run-if-empty -t -P$cores -n1 \
60+
ruby -c
61+
62+
# Check all erb files
63+
files_changed \
64+
| awk '/\.(erb)$/' \
65+
| xargs -l --no-run-if-empty -t -P$cores -n1 \
66+
bash -c 'erb -P -x -T- $0 | ruby -c'
67+
68+
# Check all Puppet manifest files
69+
files_changed \
70+
| awk '/manifests\/.*\.(pp)$/' \
71+
| xargs --no-run-if-empty -t -P$cores -n1 \
72+
puppet parser validate
73+
74+
# vim:tabstop=2
75+
# vim:shiftwidth=2
76+
# vim:expandtab

0 commit comments

Comments
 (0)