Creating repository from automated Debian Salsa builds

I'm not an Debian maintainer. Nor I want to be in future. My reasons to do all this is:

1. I want some specific software to be packaged for Debian
2. when I update packaging on salsa.debian.org I want to have available package for my devices
3. I don't want to bother with downloading packages from Salsa CI manually or compiling them again on these devices
4. I want also others to be able use my packages

These reasons motivated me to look into Salsa, GitLab API and Aptly deployment.

• package build process is handled by salsa, no need to worry about it
• ecologic - it's build only once

• no control about build process than offered by salsa-ci.yml configuration file

1. You did correctly setup debian/salsa-ci.yml and Gitlab settings (Debconf19 talk about Salsa CI)
2. you build for achitectures you'll be distributing with aptly

You can look at project configuration of https://salsa.debian.org/okias-guest/ as an example.

Aptly. First, you need to find a place, where you can host your repository. It's right, that Salsa-CI can offer you 1 repository per one pipeline, but that's not very pleasant having to switch repository everytime you build a package.

So, you need some VPS or physical machine with IPv6 (every provider should give you at least /64 range for free) or/and with IPv4 connectivity.

When you get it, just installing Debian, aptly and nginx should be enough.

Then it's a time to create user and generate your repository replated GPG key.

useradd -m repo && export USER=repo # create user repo
sudo -u $USER gpg --default-new-key-algo rsa4096 --gen-key # generate new key
sudo -u $USER mkdir -p /home/${USER}/.aptly/public/
sudo -u $USER GPG_TTY=$(tty) gpg --export > /home/${USER}/.aptly/public/public-key.asc # export key
sudo -u $USER aptly repo create -distribution unstable ${REPO_NAME}

Now we have our user and GPG key, we can move to creating our script gitlab_to_aptly.sh which we will place into /home/$USER/

Before continuing, please prepare your Bearer token, you'll learn how to do that from https://www.youtube.com/watch?v=0LsMC3ZiXkA . Fill USER= and TOKEN= inside the script.

Script does simple things

1. it does check for all your projects
2. for each project it check latest successful jobs
3. if there is new successful job, then it picks debian packages generated by salsa-ci and downloads and unpack them
4. then export them into aptly repository

#!/bin/sh
# SPDX-License-Identifier: GPL-3.0-only
# version 0.1; created by David Heidelberg <david@ixit.cz>

if [ `id -u` -eq 0 ]; then
  echo "Please DO NOT run as root!"
  exit
fi

USER= # your gitlab username
TOKEN= # watch https://www.youtube.com/watch?v=0LsMC3ZiXkA

ARCHS="amd64"
JOBS="build" # jobs must match to build == amd64;

ARTIF_UNPACK=debian/output

echo "* Downloading projects..."
PROJ_JSON=`curl -s -H 'Accept: application/json' -H "Authorization: Bearer ${TOKEN}" "https://salsa.debian.org/api/v4/users/${USER}/projects/"`
PROJECTS=`echo $PROJ_JSON | jq ".[].id" | xargs`
PUBLISH=0

check_new_jobs() {
	for id in $PROJECTS; do
		echo "* Checking jobs for project ${id}..."
		LAST_JOB_ID_FILE="proj_${id}_last_job.txt"
		LAST_JOB_ID=`curl -s -H 'Accept: application/json' -H "Authorization: Bearer ${TOKEN}" "https://salsa.debian.org/api/v4/projects/${id}/jobs?scope[]=success" | jq ".[0].id" | xargs`

		if [ -e "${LAST_JOB_ID_FILE}" ]; then
			JOB_ID=`cat ${LAST_JOB_ID_FILE}`
		else
			JOB_ID=0
		fi

		if [ $LAST_JOB_ID != "null" ] && [ $JOB_ID -ne $LAST_JOB_ID ]; then
			echo $LAST_JOB_ID > ${LAST_JOB_ID_FILE}
			echo "* Written new JOB ID $LAST_JOB_ID into file ${LAST_JOB_ID_FILE}."
			get_artifacts
			PUBLISH=1
		fi
		unset LAST_JOB_ID
	done
}

cleanup() {
        rm -vrf ./${ARTIF_UNPACK}/*
}

get_artifacts() {
        for job in $JOBS; do
                BRANCH="debian/latest"        
        	curl -o ${id}_${job}.zip -H 'Accept: application/json' -H "Authorization: Bearer ${TOKEN}" "https://salsa.debian.org/api/v4/projects/${id}/jobs/artifacts/${BRANCH}/download?job=${job}" && \
                unzip ${id}_${job}.zip && \
                rm ${id}_${job}.zip && \
                echo aptly command ID: ${id} JOB: ${job} done!
                rm ${ARTIF_UNPACK}/output.log
        done
}

publish() {
	if [ $PUBLISH -eq 0 ]; then
		echo "* Nothing to publish. Quiting..."
		exit 0
	fi

        REPO_NAME=ixit
        aptly repo add ${REPO_NAME} ${ARTIF_UNPACK} # add files
	aptly publish drop unstable # first we get rid of previously published repo
	APTLY_ARCHS=`echo $ARCHS | tr " " ,`
	aptly publish repo -batch -architectures "${APTLY_ARCHS}" -distribution unstable ${REPO_NAME}
}

cleanup
check_new_jobs
publish

Test script with

sudo -u $USER ./gitlab_to_aptly.sh

, if your /home/$USER/.aptly/public is populated, prepare nginx configuration. This configuration may differ a lot from your use. I'm assuming you have SSL configured (ssl.conf) and SSL certificates in place.

/etc/nginx/sites-available/repo.ixit.cz.conf

server {
    include ssl.conf;
    server_name repo.ixit.cz;
    ssl_certificate /etc/letsencrypt/live/ixit.cz/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ixit.cz/privkey.pem;

    location / {
        root /home/repo/.aptly/public/;
        autoindex on;
    }
}

systemctl reload nginx and check URL, if you see the repository.

In that case, you can try add it to the first device. Replace ixit with your repository name:

echo "deb https://repo.ixit.cz/ unstable main" > /etc/apt/sources.list.d/ixit.list # repository
curl -o /etc/apt/trusted.gpg.d/ixit.gpg https://repo.ixit.cz/public-key.asc # GPG key
apt update # shouldn't report any errors

now you can install package you desire and test.

Most likely, you'll want to regenerate your repository at least every hour.

This is how can look /etc/systemd/system/gitlab_to_aptly.service

[Unit]
Description=Radioalarm service

[Service]
User=repo
WorkingDirectory=/home/repo/tmp/
ExecStart=/home/repo/gitlab_to_aptly.sh

and /etc/systemd/system/gitlab_to_aptly.timer

[Unit]
Description=Update aptly repository every hour

[Timer]
OnBootSec=60min
OnUnitActiveSec=60min
Unit=gitlab_to_aptly.service

[Install]
WantedBy=timers.target