This is an old revision of the document!
Creating repository from automated Debian Salsa builds
Motivation
I'm not an Debian maintainer. Nor I want to be in future. My reasons to do all this is that:
- I want some specific software to be packaged for Debian
- when I update packaging on salsa.debian.org I want to have available package for computers and phones
- I don't want to bother myself downloading packages from Salsa CI manually or compiling them again on the devices
- I want also others to be able use my packages
These reasons motivated me to look into Salsa, GitLab API and Aptly deployment.
Where to start
Aptly. First, you need to find a place, where you can host your repository. It's right, that Salsa-CI can offer you 1 repository per one pipeline, but that's not very pleasant having to switch repository everytime you build a package.
So, you need some VPS or physical machine with IPv6 (every provider should give you at least /64 range for free) or/and with IPv4 connectivity.
When you get it, just installing Debian, aptly and nginx should be enough.
Then it's a time to create user and generate your repository replated GPG key.
useradd -m repo && export USER=repo # create user repo sudo -u $USER gpg --default-new-key-algo rsa4096 --gen-key # generate new key sudo -u $USER mkdir -p /home/${USER}/.aptly/public/ sudo -u $USER GPG_TTY=$(tty) gpg --export > /home/${USER}/.aptly/public/public-key.asc # export key sudo -u $USER aptly repo create -distribution unstable ${REPO_NAME}
Now we have our user and GPG key, we can move to creating our script gitlab_to_aptly.sh which we will place into /home/$USER/
Before continuing, please prepare your Bearer token, you'll learn how to do that from https://www.youtube.com/watch?v=0LsMC3ZiXkA . Fill USER= and TOKEN= inside the script.
Script does simple things
- it does check for all your projects
- for each project it check latest successful jobs
- if there is new successful job, then it picks debian packages generated by salsa-ci and downloads and unpack them
- then export them into aptly repository
#!/bin/sh # SPDX-License-Identifier: GPL-3.0-only # version 0.1; created by David Heidelberg <david@ixit.cz> if [ `id -u` -eq 0 ]; then echo "Please DO NOT run as root!" exit fi USER= # your gitlab username TOKEN= # watch https://www.youtube.com/watch?v=0LsMC3ZiXkA ARCHS="arm64 amd64" JOBS="build build\ arm64" # jobs must match to build == amd64; ARTIF_UNPACK=debian/output echo "* Downloading projects..." PROJ_JSON=`curl -s -H 'Accept: application/json' -H "Authorization: Bearer ${TOKEN}" "https://salsa.debian.org/api/v4/users/${USER}/projects/"` PROJECTS=`echo $PROJ_JSON | jq ".[].id" | xargs` PUBLISH=0 check_new_jobs() { for id in $PROJECTS; do echo "* Checking jobs for project ${id}..." LAST_JOB_ID_FILE="proj_${id}_last_job.txt" LAST_JOB_ID=`curl -s -H 'Accept: application/json' -H "Authorization: Bearer ${TOKEN}" "https://salsa.debian.org/api/v4/projects/${id}/jobs?scope[]=success" | jq ".[0].id" | xargs` if [ -e "${LAST_JOB_ID_FILE}" ]; then JOB_ID=`cat ${LAST_JOB_ID_FILE}` else JOB_ID=0 fi if [ $LAST_JOB_ID != "null" ] && [ $JOB_ID -ne $LAST_JOB_ID ]; then echo $LAST_JOB_ID > ${LAST_JOB_ID_FILE} echo "* Written new JOB ID $LAST_JOB_ID into file ${LAST_JOB_ID_FILE}." get_artifacts PUBLISH=1 fi unset LAST_JOB_ID done } cleanup() { rm -vrf ./${ARTIF_UNPACK}/* } get_artifacts() { for job in $JOBS; do curl -o ${id}_${job}.zip -H 'Accept: application/json' -H "Authorization: Bearer ${TOKEN}" "https://salsa.debian.org/api/v4/projects/${id}/jobs/artifacts/debian/latest/download?job=${job}" && \ unzip ${id}_${job}.zip && \ rm ${id}_${job}.zip && \ echo aptly command ID: ${id} JOB: ${job} done! rm ${ARTIF_UNPACK}/output.log done } publish() { if [ $PUBLISH -eq 0 ]; then echo "* Nothing to publish. Quiting..." exit 0 fi REPO_NAME=ixit aptly repo add ${REPO_NAME} ${ARTIF_UNPACK} # add files aptly publish drop unstable # first we get rid of previously published repo aptly publish repo -batch -architectures ${ARCHS} -distribution unstable ${REPO_NAME} } cleanup check_new_jobs publish