This is an old revision of the document!
Creating repository from automated Debian Salsa builds
Motivation
I'm not an Debian maintainer. Nor I want to be in future. My reasons to do all this is that:
- I want some specific software to be packaged for Debian
- when I update packaging on salsa.debian.org I want to have available package for computers and phones
- I don't want to bother myself downloading packages from Salsa CI manually or compiling them again on the devices
- I want also others to be able use my packages
These reasons motivated me to look into Salsa, GitLab API and Aptly deployment.
Predisposition
- You have correctly setup Salsa CI in each project you want to include in your repo (including debian/salsa-ci.yml and Gitlab settings)
- you didn't disabled build for the architecture you planing to distribute
You can take https://salsa.debian.org/okias-guest/ as an example.
Where to start
Aptly. First, you need to find a place, where you can host your repository. It's right, that Salsa-CI can offer you 1 repository per one pipeline, but that's not very pleasant having to switch repository everytime you build a package.
So, you need some VPS or physical machine with IPv6 (every provider should give you at least /64 range for free) or/and with IPv4 connectivity.
When you get it, just installing Debian, aptly and nginx should be enough.
Then it's a time to create user and generate your repository replated GPG key.
Aptly
useradd -m repo && export USER=repo # create user repo sudo -u $USER gpg --default-new-key-algo rsa4096 --gen-key # generate new key sudo -u $USER mkdir -p /home/${USER}/.aptly/public/ sudo -u $USER GPG_TTY=$(tty) gpg --export > /home/${USER}/.aptly/public/public-key.asc # export key sudo -u $USER aptly repo create -distribution unstable ${REPO_NAME}
Now we have our user and GPG key, we can move to creating our script gitlab_to_aptly.sh which we will place into /home/$USER/
Before continuing, please prepare your Bearer token, you'll learn how to do that from https://www.youtube.com/watch?v=0LsMC3ZiXkA . Fill USER= and TOKEN= inside the script.
Script does simple things
- it does check for all your projects
- for each project it check latest successful jobs
- if there is new successful job, then it picks debian packages generated by salsa-ci and downloads and unpack them
- then export them into aptly repository
#!/bin/sh # SPDX-License-Identifier: GPL-3.0-only # version 0.1; created by David Heidelberg <david@ixit.cz> if [ `id -u` -eq 0 ]; then echo "Please DO NOT run as root!" exit fi USER= # your gitlab username TOKEN= # watch https://www.youtube.com/watch?v=0LsMC3ZiXkA ARCHS="amd64" JOBS="build" # jobs must match to build == amd64; ARTIF_UNPACK=debian/output echo "* Downloading projects..." PROJ_JSON=`curl -s -H 'Accept: application/json' -H "Authorization: Bearer ${TOKEN}" "https://salsa.debian.org/api/v4/users/${USER}/projects/"` PROJECTS=`echo $PROJ_JSON | jq ".[].id" | xargs` PUBLISH=0 check_new_jobs() { for id in $PROJECTS; do echo "* Checking jobs for project ${id}..." LAST_JOB_ID_FILE="proj_${id}_last_job.txt" LAST_JOB_ID=`curl -s -H 'Accept: application/json' -H "Authorization: Bearer ${TOKEN}" "https://salsa.debian.org/api/v4/projects/${id}/jobs?scope[]=success" | jq ".[0].id" | xargs` if [ -e "${LAST_JOB_ID_FILE}" ]; then JOB_ID=`cat ${LAST_JOB_ID_FILE}` else JOB_ID=0 fi if [ $LAST_JOB_ID != "null" ] && [ $JOB_ID -ne $LAST_JOB_ID ]; then echo $LAST_JOB_ID > ${LAST_JOB_ID_FILE} echo "* Written new JOB ID $LAST_JOB_ID into file ${LAST_JOB_ID_FILE}." get_artifacts PUBLISH=1 fi unset LAST_JOB_ID done } cleanup() { rm -vrf ./${ARTIF_UNPACK}/* } get_artifacts() { for job in $JOBS; do curl -o ${id}_${job}.zip -H 'Accept: application/json' -H "Authorization: Bearer ${TOKEN}" "https://salsa.debian.org/api/v4/projects/${id}/jobs/artifacts/debian/latest/download?job=${job}" && \ unzip ${id}_${job}.zip && \ rm ${id}_${job}.zip && \ echo aptly command ID: ${id} JOB: ${job} done! rm ${ARTIF_UNPACK}/output.log done } publish() { if [ $PUBLISH -eq 0 ]; then echo "* Nothing to publish. Quiting..." exit 0 fi REPO_NAME=ixit aptly repo add ${REPO_NAME} ${ARTIF_UNPACK} # add files aptly publish drop unstable # first we get rid of previously published repo aptly publish repo -batch -architectures ${ARCHS} -distribution unstable ${REPO_NAME} } cleanup check_new_jobs publish
Nginx
Test script with
sudo -u $USER ./gitlab_to_aptly.sh
, if your /home/$USER/.aptly/public is populated, prepare nginx configuration. This configuration may differ a lot from your use. I'm assuming you have SSL configured (ssl.conf) and SSL certificates in place.
/etc/nginx/sites-available/repo.ixit.cz.conf
server { include ssl.conf; server_name repo.ixit.cz; ssl_certificate /etc/letsencrypt/live/ixit.cz/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/ixit.cz/privkey.pem; location / { root /home/repo/.aptly/public/; autoindex on; } }
systemctl reload nginx and check URL, if you see the repository.
Testing it
In that case, you can try add it to the first device. Replace ixit with your repository name:
echo "deb https://repo.ixit.cz/ unstable main" > /etc/apt/sources.list.d/ixit.list # repository curl -o /etc/apt/trusted.gpg.d/ixit.gpg https://repo.ixit.cz/public-key.asc # GPG key apt update # shouldn't write any errors
now you can install package you desire and test.
Repository refreshing
Most likely, you'll want to regenerate your repository at least every hour.
This is how can look /etc/systemd/system/gitlab_to_aptly.service
[Unit] Description=Radioalarm service [Service] User=repo WorkingDirectory=/home/repo/tmp/ ExecStart=/home/repo/gitlab_to_aptly.sh
and /etc/systemd/system/gitlab_to_aptly.timer
[Unit] Description=Update aptly repository every hour [Timer] OnBootSec=60min OnUnitActiveSec=60min Unit=gitlab_to_aptly.service [Install] WantedBy=timers.target