Creating repository from automated Debian Salsa builds

I'm not an Debian maintainer. Nor I want to be in future. My reasons to do all this is that:

1. I want some specific software to be packaged for Debian
2. when I update packaging on salsa.debian.org I want to have available package for computers and phones
3. I don't want to bother myself downloading packages from Salsa CI manually or compiling them again on the devices
4. I want also others to be able use my packages

These reasons motivated me to look into Salsa, GitLab API and Aptly deployment.

1. You have correctly setup Salsa CI (including debian/salsa-ci.yml and Gitlab settings)
2. you didn't disabled build for the architecture you planing to distribute

You can take https://salsa.debian.org/okias-guest/ as an example.

Aptly. First, you need to find a place, where you can host your repository. It's right, that Salsa-CI can offer you 1 repository per one pipeline, but that's not very pleasant having to switch repository everytime you build a package.

So, you need some VPS or physical machine with IPv6 (every provider should give you at least /64 range for free) or/and with IPv4 connectivity.

When you get it, just installing Debian, aptly and nginx should be enough.

Then it's a time to create user and generate your repository replated GPG key.

useradd -m repo && export USER=repo # create user repo
sudo -u $USER gpg --default-new-key-algo rsa4096 --gen-key # generate new key
sudo -u $USER mkdir -p /home/${USER}/.aptly/public/
sudo -u $USER GPG_TTY=$(tty) gpg --export > /home/${USER}/.aptly/public/public-key.asc # export key
sudo -u $USER aptly repo create -distribution unstable ${REPO_NAME}

Now we have our user and GPG key, we can move to creating our script gitlab_to_aptly.sh which we will place into /home/$USER/

Before continuing, please prepare your Bearer token, you'll learn how to do that from https://www.youtube.com/watch?v=0LsMC3ZiXkA . Fill USER= and TOKEN= inside the script.

Script does simple things

1. it does check for all your projects
2. for each project it check latest successful jobs
3. if there is new successful job, then it picks debian packages generated by salsa-ci and downloads and unpack them
4. then export them into aptly repository

#!/bin/sh
# SPDX-License-Identifier: GPL-3.0-only
# version 0.1; created by David Heidelberg <david@ixit.cz>

if [ `id -u` -eq 0 ]; then
  echo "Please DO NOT run as root!"
  exit
fi

USER= # your gitlab username
TOKEN= # watch https://www.youtube.com/watch?v=0LsMC3ZiXkA

ARCHS="arm64 amd64"
JOBS="build build\ arm64" # jobs must match to build == amd64;

ARTIF_UNPACK=debian/output

echo "* Downloading projects..."
PROJ_JSON=`curl -s -H 'Accept: application/json' -H "Authorization: Bearer ${TOKEN}" "https://salsa.debian.org/api/v4/users/${USER}/projects/"`
PROJECTS=`echo $PROJ_JSON | jq ".[].id" | xargs`
PUBLISH=0

check_new_jobs() {
	for id in $PROJECTS; do
		echo "* Checking jobs for project ${id}..."
		LAST_JOB_ID_FILE="proj_${id}_last_job.txt"
		LAST_JOB_ID=`curl -s -H 'Accept: application/json' -H "Authorization: Bearer ${TOKEN}" "https://salsa.debian.org/api/v4/projects/${id}/jobs?scope[]=success" | jq ".[0].id" | xargs`

		if [ -e "${LAST_JOB_ID_FILE}" ]; then
			JOB_ID=`cat ${LAST_JOB_ID_FILE}`
		else
			JOB_ID=0
		fi

		if [ $LAST_JOB_ID != "null" ] && [ $JOB_ID -ne $LAST_JOB_ID ]; then
			echo $LAST_JOB_ID > ${LAST_JOB_ID_FILE}
			echo "* Written new JOB ID $LAST_JOB_ID into file ${LAST_JOB_ID_FILE}."
			get_artifacts
			PUBLISH=1
		fi
		unset LAST_JOB_ID
	done
}

cleanup() {
        rm -vrf ./${ARTIF_UNPACK}/*
}

get_artifacts() {
        for job in $JOBS; do
        	curl -o ${id}_${job}.zip -H 'Accept: application/json' -H "Authorization: Bearer ${TOKEN}" "https://salsa.debian.org/api/v4/projects/${id}/jobs/artifacts/debian/latest/download?job=${job}" && \
                unzip ${id}_${job}.zip && \
                rm ${id}_${job}.zip && \
                echo aptly command ID: ${id} JOB: ${job} done!
                rm ${ARTIF_UNPACK}/output.log
        done
}

publish() {
	if [ $PUBLISH -eq 0 ]; then
		echo "* Nothing to publish. Quiting..."
		exit 0
	fi

        REPO_NAME=ixit
        aptly repo add ${REPO_NAME} ${ARTIF_UNPACK} # add files
	aptly publish drop unstable # first we get rid of previously published repo
        aptly publish repo -batch -architectures ${ARCHS} -distribution unstable ${REPO_NAME}
}

cleanup
check_new_jobs
publish

Test script with

sudo -u $USER ./gitlab_to_aptly.sh

, if your /home/$USER/.aptly/public is populated, prepare nginx configuration. This configuration may differ a lot from your use. I'm assuming you have SSL configured (ssl.conf) and SSL certificates in place.

/etc/nginx/sites-available/repo.ixit.cz.conf

server {
    include ssl.conf;
    server_name repo.ixit.cz;
    ssl_certificate /etc/letsencrypt/live/ixit.cz/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ixit.cz/privkey.pem;

    location / {
        root /home/repo/.aptly/public/;
        autoindex on;
    }
}

systemctl reload nginx and check URL, if you see the repository.

In that case, you can try add it to the first device. Replace ixit with your repository name:

echo "deb https://repo.ixit.cz/ unstable main" > /etc/apt/sources.list.d/ixit.list # repository
curl -o /etc/apt/trusted.gpg.d/ixit.gpg https://repo.ixit.cz/public-key.asc # GPG key
apt update # shouldn't write any errors

now you can install package you desire and test.

Most likely, you'll want to regenerate your repository at least every hour.

This is how can look /etc/systemd/system/gitlab_to_aptly.service

[Unit]
Description=Radioalarm service

[Service]
User=repo
WorkingDirectory=/home/repo/tmp/
ExecStart=/home/repo/gitlab_to_aptly.sh

and /etc/systemd/system/gitlab_to_aptly.timer

[Unit]
Description=Update aptly repository every hour

[Timer]
OnBootSec=60min
OnUnitActiveSec=60min
Unit=gitlab_to_aptly.service

[Install]
WantedBy=timers.target