====== Creating repository from automated Debian Salsa builds ======
{{ :salsa_to_aptly.png |}}
===== Motivation =====

I'm not an Debian maintainer. Nor I want to be in future.
My reasons to do all this is:
  - I want some specific software to be packaged for Debian
  - when I update packaging on salsa.debian.org I want to have available package for my devices
  - I don't want to bother with downloading packages from Salsa CI manually or compiling them again on these devices
  - I want also others to be able use my packages

These reasons motivated me to look into [[salsa.debian.org/|Salsa]], [[https://docs.gitlab.com/ee/api/|GitLab API]] and [[https://www.aptly.info/tutorial/|Aptly]] deployment.

==== Advantages ====
   * package build process is handled by salsa, no need to worry about it
   * ecologic - it's build only once

==== Disadvantages ====
   * no control about build process than offered by salsa-ci.yml configuration file

===== Predisposition =====
  - You did correctly setup **debian/salsa-ci.yml** and Gitlab settings ([[https://debconf19.debconf.org/talks/148-salsa-ci-debian-pipeline-for-developers/|Debconf19 talk about Salsa CI]])
  - you build for achitectures you'll be distributing with aptly

You can look at project configuration of [[https://salsa.debian.org/okias-guest/]] as an example.

===== Where to start =====

Aptly. First, you need to find a place, where you can host your repository. It's right, that Salsa-CI can offer you 1 repository per one pipeline, but that's not very pleasant having to switch repository everytime you build a package.

So, you need some VPS or physical machine with IPv6 (every provider should give you at least /64 range for free) or/and with IPv4 connectivity.

When you get it, just installing Debian, [[https://www.aptly.info/download/|aptly]] and nginx should be enough.

Then it's a time to create user and generate your repository replated GPG key.
==== Aptly ====
<code>
useradd -m repo && export USER=repo # create user repo
sudo -u $USER gpg --default-new-key-algo rsa4096 --gen-key # generate new key
sudo -u $USER mkdir -p /home/${USER}/.aptly/public/
sudo -u $USER GPG_TTY=$(tty) gpg --export > /home/${USER}/.aptly/public/public-key.asc # export key
sudo -u $USER aptly repo create -distribution unstable ${REPO_NAME}
</code>

Now we have our user and GPG key, we can move to creating our script **gitlab_to_aptly.sh** which we will place into **/home/$USER/**

Before continuing, please prepare your Bearer token, you'll learn how to do that from https://www.youtube.com/watch?v=0LsMC3ZiXkA . Fill **USER=** and **TOKEN=** inside the script.

Script does simple things
  - it does check for all your projects
  - for each project it check latest successful jobs
  - if there is new successful job, then it picks debian packages generated by salsa-ci and downloads and unpack them
  - then export them into aptly repository

<code>
#!/bin/sh
# SPDX-License-Identifier: GPL-3.0-only
# version 0.1; created by David Heidelberg <david@ixit.cz>

if [ `id -u` -eq 0 ]; then
  echo "Please DO NOT run as root!"
  exit
fi

USER= # your gitlab username
TOKEN= # watch https://www.youtube.com/watch?v=0LsMC3ZiXkA

ARCHS="amd64"
JOBS="build" # jobs must match to build == amd64;

ARTIF_UNPACK=debian/output

echo "* Downloading projects..."
PROJ_JSON=`curl -s -H 'Accept: application/json' -H "Authorization: Bearer ${TOKEN}" "https://salsa.debian.org/api/v4/users/${USER}/projects/"`
PROJECTS=`echo $PROJ_JSON | jq ".[].id" | xargs`
PUBLISH=0

check_new_jobs() {
	for id in $PROJECTS; do
		echo "* Checking jobs for project ${id}..."
		LAST_JOB_ID_FILE="proj_${id}_last_job.txt"
		LAST_JOB_ID=`curl -s -H 'Accept: application/json' -H "Authorization: Bearer ${TOKEN}" "https://salsa.debian.org/api/v4/projects/${id}/jobs?scope[]=success" | jq ".[0].id" | xargs`

		if [ -e "${LAST_JOB_ID_FILE}" ]; then
			JOB_ID=`cat ${LAST_JOB_ID_FILE}`
		else
			JOB_ID=0
		fi

		if [ $LAST_JOB_ID != "null" ] && [ $JOB_ID -ne $LAST_JOB_ID ]; then
			echo $LAST_JOB_ID > ${LAST_JOB_ID_FILE}
			echo "* Written new JOB ID $LAST_JOB_ID into file ${LAST_JOB_ID_FILE}."
			get_artifacts
			PUBLISH=1
		fi
		unset LAST_JOB_ID
	done
}

cleanup() {
        rm -vrf ./${ARTIF_UNPACK}/*
}

get_artifacts() {
        for job in $JOBS; do
                BRANCH="debian/latest"        
        	curl -o ${id}_${job}.zip -H 'Accept: application/json' -H "Authorization: Bearer ${TOKEN}" "https://salsa.debian.org/api/v4/projects/${id}/jobs/artifacts/${BRANCH}/download?job=${job}" && \
                unzip ${id}_${job}.zip && \
                rm ${id}_${job}.zip && \
                echo aptly command ID: ${id} JOB: ${job} done!
                rm ${ARTIF_UNPACK}/output.log
        done
}

publish() {
	if [ $PUBLISH -eq 0 ]; then
		echo "* Nothing to publish. Quiting..."
		exit 0
	fi

        REPO_NAME=ixit
        aptly repo add ${REPO_NAME} ${ARTIF_UNPACK} # add files
	aptly publish drop unstable # first we get rid of previously published repo
	APTLY_ARCHS=`echo $ARCHS | tr " " ,`
	aptly publish repo -batch -architectures "${APTLY_ARCHS}" -distribution unstable ${REPO_NAME}
}

cleanup
check_new_jobs
publish

</code>

==== Nginx ====

Test script with <code>sudo -u $USER ./gitlab_to_aptly.sh</code>, if your /home/$USER/.aptly/public is populated, prepare nginx configuration. This configuration may differ a lot from your use. I'm assuming you have SSL configured (ssl.conf) and SSL certificates in place.

**/etc/nginx/sites-available/repo.ixit.cz.conf**
<code>
server {
    include ssl.conf;
    server_name repo.ixit.cz;
    ssl_certificate /etc/letsencrypt/live/ixit.cz/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ixit.cz/privkey.pem;

    location / {
        root /home/repo/.aptly/public/;
        autoindex on;
    }
}
</code>

systemctl reload nginx and check URL, if you see the repository.

==== Testing it ====

In that case, you can try add it to the first device. Replace ixit with your repository name:

<code>
echo "deb https://repo.ixit.cz/ unstable main" > /etc/apt/sources.list.d/ixit.list # repository
curl -o /etc/apt/trusted.gpg.d/ixit.gpg https://repo.ixit.cz/public-key.asc # GPG key
apt update # shouldn't report any errors
</code>
now you can install package you desire and test.

==== Repository refreshing ====

Most likely, you'll want to regenerate your repository at least every hour.

This is how can look **/etc/systemd/system/gitlab_to_aptly.service**
<code>
[Unit]
Description=Radioalarm service

[Service]
User=repo
WorkingDirectory=/home/repo/tmp/
ExecStart=/home/repo/gitlab_to_aptly.sh
</code>
and **/etc/systemd/system/gitlab_to_aptly.timer**
<code>
[Unit]
Description=Update aptly repository every hour

[Timer]
OnBootSec=60min
OnUnitActiveSec=60min
Unit=gitlab_to_aptly.service

[Install]
WantedBy=timers.target
</code>