Getting Started With Podman and Containers

Introduction

Containers are, in my (100% correct, based and objectively true) opinion, the best way to manage any kind of server. Kyun is in containers, the blog is in a container, the GitLab server this doc is fetched from is in a container.

In practice, containers behave a lot like virtual machines, only that they’re a lot less resource hogging, as they share the same Linux kernel as the host machine. In fact, a lot of people, including some hosting providers (not Kyun), use containers like they would use virtual machines, via LXC. Performance-wise, it’s almost the same as running the software straight on the host machine.

You’ve probably heard of Docker, and Podman is in pretty much every way a 1:1 compatible implementation of Docker, but a lot more lightweight: it doesn’t need root, it doesn’t need a daemon running, there’s no commercial licensing bullshit, it just works.

Note: A very important difference between Docker and Podman is that, by default, Podman doesn’t auto-start containers you had running after you reboot. Read Auto-Start On Boot for details.

Why Containers?

Remember back in 2022 when some loser that was maintaining some very popular package on npm pushed a malicious update that wiped all data on computers with Russian and Belarusian IPs?

If you were to execute the code from that package while using a targeted IP, one of two things could happen depending on whether you were running it in a container or not:

Scenario One - Bad Ending. You are not running it in a container. All of your data is lost. Quick, pull the power cord and run photorec. You do have a backup, right?

Scenario Two - Good Ending. You are running it in a container. Nothing happens, because the container doesn’t have access to the host filesystem, except for the directories you specifically give it access to. At most, you lost some user avatars or something like that. You probably didn’t even notice anything was wrong, because everything on the host and other containers is intact.

This is just a worst-case scenario example, but containers provide a lot of other benefits:

Containers Run Anywhere, Predictably

No more hunting for dependencies. No more dependency conflicts. No more manually downloading software or adding repos if it’s not in your distro’s repositories. It doesn’t matter if you’re on Arch, Debian, Ubuntu, Gentoo, NixOS, or even Alpine, containers run with the exact same behavior.

Extraordinarily Easy Backups & Configuration

The average containerless server filesystem structure looks something like this:

/etc/
| webapps/
| | gitlab/
| | | gitlab.yml
| | | ...
| | ...
| nginx/
| | nginx.conf
| | ...
| redis/
| | ...
| ...
/var/
| lib/
| | gitlab/
| | | ...
| | postgres/
| | | ...
| | ...
| www/
| | html/
| | | index.html
| | | style.css
| ...

Looks like a mess, because it is. Especially when you consider that among the files you actually care about are a bunch of random files and programs that you will never need to touch.

If you want to back this up or move to a new server, it’s either waste a bunch of time remembering which data you need and where it’s located, or waste even more time by giving up and setting everything up again from scratch.

And if you have any packages installed that you need in order to run your software, well, sucks for you, you’ll need to find and install them again on the new server.

Compare this with a potential container-based setup with the exact same software:

/srv/
| gitlab/
| | compose.yaml
| | gitlab.yml
| | data/
| | | redis/
| | | | ...
| | | postgres/
| | | | ...
| web/
| | compose.yaml
| | nginx.conf
| | mywebsite/
| | | index.html
| | | style.css

To back this up, all you need to do is copy the entire /srv directory.

Everything you will need to touch, e.g. configuration files and website assets, is put in very convenient places, while internal software data is neatly tucked away so that it doesn’t bother you but can still be easily backed up and/or moved.

Static or non-persistent files, such as the software binaries, remain in the container and are not mapped to the host filesystem.

All “packages” (container images) are defined and configured in the compose.yaml files, so when you move to a new server or restore the backup all you need to do is install Podman (+ compose) and run podman-compose up -d.

Containerized Networks

By default, whenever you run a container, it has access to the Internet, but the Internet doesn’t have access to it.

Let’s say you want to run a database server such as PostgreSQL on your server. If you’re not running it in a container, you have to do one of 2 things to fully secure the database:

Most people don’t do any of these 2, so you better make sure you have a good strong password and that there are no exploits in your database software, or you’re going to have a bad time.

Podman does the first by default. It creates a separate isolated network for every container, and you can manually create networks to connect containers together. You can also expose specific ports from the container to the Internet.

Getting Into Podman (no diddy)

You will need 2 packages: podman and podman-compose. Most distros have them, but if not, follow the official installation instructions for Podman and Podman Compose.

Let’s make sure everything was installed correctly by running the podman/hello image from the quay.io repository:

$ podman run quay.io/podman/hello

You should be met with this output:

Trying to pull quay.io/podman/hello:latest...
Getting image source signatures
Copying blob d08b40be6878 done
Copying config e2b3db5d4f done
Writing manifest to image destination
Storing signatures
!... Hello Podman World ...!

         .--"--.
       / -     - \
      / (O)   (O) \
   ~~~| -=(,Y,)=- |
    .---. /`  \   |~~
 ~/  o  o \~~~~.----. ~~
  | =(X)= |~  / (O (O) \
   ~~~~~~~  ~| =(Y_)=-  |
  ~~~~    ~~~|   U      |~~

Project:   https://github.com/containers/podman
Website:   https://podman.io
Documents: https://docs.podman.io
Twitter:   @Podman_io

Podman has 2 repositories (“registries”) configured by default: quay.io, which is managed by Red Hat (IBM), and docker.io, which is managed by, well, Docker.

If you run an image without specifying the repository (e.g. podman run podman/hello), it will ask you which repository to use (unless the name is in the common image list).

There are a bunch of container repositories, including Google’s gcr.io and GitHub’s ghcr.io, you can also run your own, but docker.io has the most images by far.

Commonly Used Podman Commands

I won’t go into depth, you can use the official Podman documentation or podman --help for that, but here are some examples of commands with frequently used arguments:

$ podman run -it --rm -v ./host:/container -p 1234:1234 docker.io/library/alpine

$ podman ps

Shows a list of containers that are currently running.

$ podman exec -it f4a46023442f /bin/ash

Runs /bin/ash inside the running container with ID f4a46023442f (you can find the ID using podman ps) and attaches it to the current terminal.

$ podman system prune

General housekeeping command. Run this if you’re running out of disk space and you don’t know why. Removes unused containers, images, networks and build cache (from building container images). Add –all to also remove all images that have no containers currently running them.

Compose

podman-compose is a tool that enables using YAML configuration files for managing your containers. It adds no special functionality to Podman itself, but it makes life a lot easier and I highly recommend you use it for anything more than one-off containers. The format is 100% compatible with Docker Compose (docker-compose.yml).

Again, I won’t go into much detail, you have the compose spec for that.

Let’s take this sample compose.yaml for running Nitter as an example:

services:
  nitter:
    image: docker.io/zedeus/nitter:latest
    restart: always
    ports:
      - "8080:8080" # expose port 8080 to the Internet
    depends_on:
      - redis # nitter won't start until redis does
    volumes:
      - ./nitter.conf:/src/nitter.conf

  redis:
    image: docker.io/library/redis:6-alpine
    restart: always
    command: redis-server --save 60 1 --loglevel warning
    volumes:
      - ./redis:/data

The container properties directly map to podman arguments. The redis configuration is equivalent to:

$ podman run \
 --restart always \
 --volume ./redis:/data \
 redis:6-alpine \
 redis-server --save 60 1 --loglevel warning

nitter.conf would look something like this:

[Server]
hostname = "your.domain"
title = "nitter"
address = "0.0.0.0"
port = 8080

[Cache]
redisHost = "redis"
redisPort = 6379
...

Notice how redisHost is set to just redis, and not an IP or proper domain name.

This works because, under the hood, podman-compose will create a new network with all of the containers in the compose.yaml file, and sets the hostnames of the internal container IPs to their respective service names defined in the file.

Some useful compose commands:

$ podman-compose up -d

Starts all of the containers defined in the compose file, in the background. If you omit the -d, it will attach the container outputs to your terminal.

$ podman-compose up -d containername

Starts container containername.

$ podman-compose down -v

Takes all containers down and removes their networks and files, except for external networks and files mounted from the host filesystem or external volumes.

$ podman-compose logs --tail 500 -f containername

Shows the last 500 lines of the log for the container containername and follows new entries in the log. Leave out containername to see logs for all containers.

Auto-Start On Boot

If you want your Podman containers to automatically start after you reboot, the easiest way to do it is to set all of your containers’ restart properties to always.

For example, instead of:

service:
  image: docker.io/node:current
  restart: unless-stopped

you should have:

service:
  image: docker.io/node:current
  restart: always

Then, enable the podman-restart service:

$ systemctl --user enable --now podman-restart
$ systemctl --user status podman-restart

● podman-restart.service - Podman Start All Containers With Restart Policy Set To Always
     Loaded: loaded (/usr/lib/systemd/user/podman-restart.service; enabled; preset: enabled)
     .....

Notice how, because Podman is rootless, you don’t need sudo to run any of these commands, even if you’re not the root user.

Make sure your user’s services start as soon as possible, even without starting a session:

$ sudo loginctl enable-linger $USER

Troubleshooting

If you’re running into “unknown command/argument/etc” errors:

This guide is written for the latest versions of Podman and Podman Compose (at the time of writing, of course).

Many stable distros, i.e. Debian, ship with older versions, even on the latest OS releases.

The Podman ecosystem is rapidly evolving, in part due to Red Hat’s recent involvement (going as far as entirely dropping support for Docker in favor of Podman in their distros), and as such it is recommended to use the newer versions.

The most common issue you’ll run into is an outdated podman-compose version. To install the latest version, no matter the distro: