Posts in this category
- Automating Deployments: A New Year and a Plan
- Automating Deployments: Why bother?
- Automating Deployments: Simplistic Deployment with Git and Bash
- Automating Deployments: Building Debian Packages
- Automating Deployments: Debian Packaging for an Example Project
- Automating Deployments: Distributing Debian Packages with Aptly
- Automating Deployments: Installing Packages
- Automating Deployments: 3+ Environments
- Architecture of a Deployment System
- Introducing Go Continuous Delivery
- Technology for automating deployments: the agony of choice
- Automating Deployments: New Website, Community
- Continuous Delivery for Libraries?
- Managing State in a Continuous Delivery Pipeline
- Automating Deployments: Building in the Pipeline
- Automating Deployments: Version Recycling Considered Harmful
- Automating Deployments: Stage 2: Uploading
- Automating Deployments: Installation in the Pipeline
- Automating Deployments: Pipeline Templates in GoCD
- Automatically Deploying Specific Versions
- Story Time: Rollbacks Saved the Day
- Automated Deployments: Unit Testing
- Automating Deployments: Smoke Testing and Rolling Upgrades
- Automating Deployments and Configuration Management
- Ansible: A Primer
- Continuous Delivery and Security
- Continuous Delivery on your Laptop
- Moritz on Continuous Discussions (#c9d9)
- Git Flow vs. Continuous Delivery
Wed, 13 Jan 2016
Automating Deployments: Distributing Debian Packages with Aptly
Once a Debian package is built, it must be distributed to the servers it is to be installed on.
Debian, as well as all other operating systems I know of, use a pull model for that. That is, the package and its meta data are stored on a server that the client can contact, and request the meta data and the package.
The sum of meta data and packages is called a repository. In order to distribution packages to the servers that need them, we must set up and maintain such a repository.
In Debian land, packages are also signed cryptographically, to ensure packages aren't tampered with on the server or during transmission.
So the first step is to create a key pair that is used to sign this particular repository. (If you already have a PGP key for signing packages, you can skip this step).
The following assumes that you are working with a pristine system user that
does not have a gnupg keyring yet, and which will be used to maintain the
debian repository. It also assumes you have the
$ gpg --gen-key
This asks a bunch of questions, like your name and email address, key type and bit width, and finally a pass phrase. I left the pass phrase empty to make it easier to automate updating the repository, but that's not a requirement.
$ gpg --gen-key gpg (GnuPG) 1.4.18; Copyright (C) 2014 Free Software Foundation, Inc. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. gpg: directory `/home/aptly/.gnupg' created gpg: new configuration file `/home/aptly/.gnupg/gpg.conf' created gpg: WARNING: options in `/home/aptly/.gnupg/gpg.conf' are not yet active during this run gpg: keyring `/home/aptly/.gnupg/secring.gpg' created gpg: keyring `/home/aptly/.gnupg/pubring.gpg' created Please select what kind of key you want: (1) RSA and RSA (default) (2) DSA and Elgamal (3) DSA (sign only) (4) RSA (sign only) Your selection? 1 RSA keys may be between 1024 and 4096 bits long. What keysize do you want? (2048) Requested keysize is 2048 bits Please specify how long the key should be valid. 0 = key does not expire <n> = key expires in n days <n>w = key expires in n weeks <n>m = key expires in n months <n>y = key expires in n years Key is valid for? (0) Key does not expire at all Is this correct? (y/N) y You need a user ID to identify your key; the software constructs the user ID from the Real Name, Comment and Email Address in this form: "Heinrich Heine (Der Dichter) <email@example.com>" Real name: Aptly Signing Key Email address: firstname.lastname@example.org You selected this USER-ID: "Moritz Lenz <email@example.com>" Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O You need a Passphrase to protect your secret key. You don't want a passphrase - this is probably a *bad* idea! I will do it anyway. You can change your passphrase at any time, using this program with the option "--edit-key". We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy. ..........+++++ .......+++++ Not enough random bytes available. Please do some other work to give the OS a chance to collect more entropy! (Need 99 more bytes) ..+++++ gpg: /home/aptly/.gnupg/trustdb.gpg: trustdb created gpg: key 071B4856 marked as ultimately trusted public and secret key created and signed. gpg: checking the trustdb gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u pub 2048R/071B4856 2016-01-10 Key fingerprint = E80A D275 BAE1 DEDE C191 196D 078E 8ED8 071B 4856 uid Moritz Lenz <firstname.lastname@example.org> sub 2048R/FFF787F6 2016-01-10
Near the bottom the line starting with
pub contains the key
pub 2048R/071B4856 2016-01-10
We'll need the public key later, so it's best to export it:
$ gpg --export --armor 071B4856 > pubkey.asc
Preparing the Repository
There are several options for managing Debian repositories. My experience with debarchiver is mixed: Once set up, it works, but it does not give immediate feedback on upload; rather it communicates the success or failure by email, which isn't very well-suited for automation.
Instead I use aptly, which works fine from the command line, and additionally supports several versions of the package in one repository.
To initialize a repo, we first have to come up with a name. Here I call it
$ aptly repo create -distribution=jessie -architectures=amd64,i386,all -component=main internal Local repo [internal] successfully added. You can run 'aptly repo add internal ...' to add packages to repository. $ aptly publish repo -architectures=amd64,i386,all internal Warning: publishing from empty source, architectures list should be complete, it can't be changed after publishing (use -architectures flag) Loading packages... Generating metadata files and linking package files... Finalizing metadata files... Signing file 'Release' with gpg, please enter your passphrase when prompted: Clearsigning file 'Release' with gpg, please enter your passphrase when prompted: Local repo internal has been successfully published. Please setup your webserver to serve directory '/home/aptly/.aptly/public' with autoindexing. Now you can add following line to apt sources: deb http://your-server/ jessie main Don't forget to add your GPG key to apt with apt-key. You can also use `aptly serve` to publish your repositories over HTTP quickly.
As the message says, there needs to be a HTTP server that makes these files available. For example an Apache virtual host config for serving these files could look like this:
<VirtualHost *:80> ServerName apt.example.com ServerAdmin email@example.com DocumentRoot /home/aptly/.aptly/public/ <Directory /home/aptly/.aptly/public/> Options +Indexes +FollowSymLinks Require all granted </Directory> # Possible values include: debug, info, notice, warn, error, crit, # alert, emerg. LogLevel notice CustomLog /var/log/apache2/apt/access.log combined ErrorLog /var/log/apache2/apt/error.log ServerSignature On </VirtualHost>
After creating the logging directory (
/var/log/apache2/apt/), enabling the the virtual host
a2ensite apt.conf) and restarting Apache, the Debian repository
Adding Packages to the Repository
Now that the repository is set up, you can add a package by running
$ aptly repo add internal package-info_0.1-1_all.deb $ aptly publish update internal
Configuring a Host to use the Repository
Copy the PGP public key with which the repository is signed
pubkey.asc) to the host which shall use the repository, and
$ apt-key add pubkey.asc
Then add the actual package source:
$ echo "deb http://apt.example.com/ jessie main" > /etc/apt/source.list.d/internal
apt-get update, the contents of the repository are
available, and an
apt-cache policy package-info shows the
repository as a possible source for this package:
$ apt-cache policy package-info package-info: Installed: (none) Candidate: 0.1-1 Version table: *** 0.1-1 0 990 http://apt.example.com/ jessie/main amd64 Packages 100 /var/lib/dpkg/status
This concludes the whirlwind tour through debian repository management and thus package distribution. Next up will be the actual package installation.
I'm writing a book on automating deployments. If this topic interests you, please sign up for the Automating Deployments newsletter. It will keep you informed about automating and continuous deployments. It also helps me to gauge interest in this project, and your feedback can shape the course it takes.