Tools: No Docker Here: Welcome to Singularity 🔬

🤔 "Why Not Docker?" ## 📦 Image Management ## ▶️ Running Containers ## 📂 File System & Volumes ## ⚙️ Environment Variables ## 🌐 Networking ## 🔄 Running Services (Instances) ## 📄 Definition Files ## 🧪 Sandbox Mode ## 🎮 GPU Access ## 📤 Registry & Sharing ## 🎼 Multi-Container Orchestration ## 🔍 Troubleshooting ## 💡 What I Learned ## Where Singularity wins ## Watch out for ## 🎬 The End I just joined a new project — one that runs on a big HPC cluster. I opened the project README.md and saw something like this: I had no idea what singularity was. 😅 So I did what felt natural — typed the Docker command instead: And the terminal replied with: I messaged my project manager. His reply was short: "We don't use Docker here. We use Singularity." I stared at the message, thinking: "I have been using Docker for years. I know docker build, docker run, docker push like the back of my hand. And now none of that works here?" That's how it all started. I loved Docker. Years of packaging apps in containers, deploying to production, running ML training pipelines. It was part of how I worked every day. So I asked my project manager to install Docker on the cluster. His reply came quickly: "We can't. There are legal reasons we can't use Docker here. Company policy. But also — Singularity is a better fit for what we do." He didn't go into all the legal details, but explained the technical side in a Google Meet session. Docker needs a background daemon running all the time — a service that sits there waiting for your commands. On a shared HPC cluster where hundreds of researchers submit jobs, that adds complexity and overhead. Singularity doesn't need any daemon — you just run it directly. No background process. And you are the same user inside the container as you are outside. No switching to root, no permission confusion. That last part sounded too good to be true. But it was true. 💡 Docker wants to isolate your app from the system. Singularity wants to integrate your app with the system. That made sense. I started learning Singularity and writing down every Docker command I knew next to its Singularity equivalent. This article is that cheat sheet. First thing I needed was a Python image. In Docker, I would type docker pull python:3.11. In Singularity, the command is almost the same — with one small twist: See that docker:// prefix? All my Docker Hub images still work. Every image I had ever used — GPU-enabled, data science, notebook servers — still available. But instead of layers hidden in Docker's storage, I got a single file: That .sif file is the image. A real file in my directory. I can cp it, scp it to another node, or rsync it anywhere. Try that with Docker — you need a registry, accounts, push, pull... With Singularity, you just copy a file. Want to delete it? rm python_3.11.sif. No docker system prune, no dangling images. In Docker, I would do docker run python:3.11 python script.py. In Singularity, the keyword is exec instead of run (run exists too — more on that in a second): This is where Singularity first surprised me. I typed singularity shell python_3.11.sif, then ls — and saw all my files. My notebooks. My training scripts. My config files. Everything from my home folder, right there. In Docker, you see an empty filesystem unless you mount your folder with -v. In Singularity, your home directory, current working directory, /tmp, and system paths like /proc and /sys are automatically available ✅. So my old Docker habit: No -v. No -w. Your current directory is already there. When you need folders outside your home directory, use --bind: You can also make binds read-only: --bind /shared/datasets:/data:ro. This one caught me off guard. In Docker, the container starts with a clean environment. If you need an API key inside, you pass it explicitly: Singularity does the opposite — it inherits your entire host environment by default ✅. Your $PATH, your custom variables — all available inside the container. At first I thought this was great. Then I hit a bug where my container's Python was fighting with my host's Python paths because environment variables were leaking in. That's when I learned about --cleanenv: My advice: always use --cleanenv for training runs. The inherited environment is handy for quick interactive work, but for anything reproducible, you want a clean slate. In Docker, every container gets its own isolated network. Want to run a notebook server? You need port mapping: In Singularity, there is no network isolation. The container uses the host network directly. Start a service on port 8888 inside the container, and it is on port 8888 on your machine. No mapping needed ✅: Running notebook servers, dashboards, monitoring tools — no more figuring out port mappings. Just start the service and go to localhost:port. The downside? Two users on the same node starting something on port 8888 will conflict. But our cluster setup handles that by assigning different ports. Sometimes you need something running in the background — a notebook server, a database, a monitoring dashboard. In Docker, you use -d to detach. Singularity has instances: One important difference: in Docker, the daemon keeps your containers alive even if you log out. In Singularity, instances are tied to your session. If you disconnect from the cluster, they stop. For long-running services, you need a job scheduler — which is usually how HPC clusters work anyway. Every Docker user knows the Dockerfile. Singularity has its own version called a definition file (.def). Different structure, same idea — a recipe for building your image. The same thing as a Singularity definition file: Most of the time, you don't need a .def file at all. If you already have a Docker image, convert it directly: I only started writing .def files when I needed a custom environment that didn't exist on Docker Hub. For everything else, docker:// was enough. When I need to experiment with new packages before committing to a build, I create a writable sandbox — a draft environment I can mess around in, then freeze into a clean image: In Docker, you would do docker run -it myimage bash then docker commit, but the sandbox approach feels more intentional. This is where Singularity really shines. In Docker, you need --gpus all: In Singularity, you add --nv: That's it. No nvidia-docker, no container runtime config, no Docker daemon GPU passthrough setup. Just --nv and your NVIDIA drivers are available. For AMD GPUs, it's --rocm. Combined with a job scheduler, a typical training job looks like this: Submit the job and check the results later. In Docker, sharing an image means pushing to a registry. Both sides need accounts and permissions. In Singularity, you can push to the Singularity Library: But what I actually do: just copy the file. Anyone on the cluster can run singularity exec /shared/containers/team-env.sif python train.py and get the exact same environment. Same packages, same versions, same GPU libraries. No registry, no login. Just a file on a shared filesystem. Singularity has no built-in docker-compose alternative ⚠️. If you come from a world of docker-compose up with web servers, databases, and caches all wired together, this will feel like a step back. But on HPC, you usually don't need it. Most workloads are single-container jobs. Your training script runs inside one environment, reads data from shared storage, and writes checkpoints to scratch space. When you do need multiple services, you have a few options: 📜 Option 1: A simple shell script. Start each service as a Singularity instance, connect them through localhost (since they share the host network): 🔧 Option 2: singularity-compose — a community tool that reads YAML files similar to docker-compose.yml. It works for simple setups, but it is not actively maintained. Here are the problems I hit and how I solved them: 🐍 Host Python leaking in — my $PYTHONPATH and other environment variables were being inherited by the container, causing import errors. Fix: always use --cleanenv for training runs. ✏️ "Read-only file system" errors — Singularity images are read-only by default. If your script tries to write to /opt or /usr, it will fail. Fix: write to $HOME, use --bind, or use sandbox mode. 🔌 Port conflicts — two users starting a service on the same port will conflict since Singularity shares the host network. Fix: always pick a random port, or let your job scheduler assign one. 💾 Disk space — .sif files can be large (multi-GB for GPU images) and there is no layer sharing. Fix: keep shared images in /shared/containers/ instead of each person having their own copy. That command not found on my first day scared me. I thought none of my Docker experience would transfer. But it did 😊. Every Docker image still works — just add docker:// in front. Every concept — images, containers, mounts, environment variables — still applies. Same mental model, different tool. Singularity taught me something unexpected: isolation is not always the goal. Docker keeps containers separate from the host. But on a shared cluster, you actually want the container to feel like part of the system. You want your files there. You want the GPU drivers to just work. You want to submit a job and not worry about daemon sockets and port mappings. My workflow now: Docker on my MacBook for local testing. Singularity on the cluster for real training. Same images, same Dockerfiles, different last mile. They are not competitors. They answer different questions: Sometimes the second question is the right one. If you found this useful, follow me here on dev.to and check out my GitHub. Templates let you quickly answer FAQs or store snippets for re-use. Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse CODE_BLOCK: singularity exec docker://python:3.11 python train_model.py --data /shared/datasets/train Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: singularity exec docker://python:3.11 python train_model.py --data /shared/datasets/train CODE_BLOCK: singularity exec docker://python:3.11 python train_model.py --data /shared/datasets/train COMMAND_BLOCK: docker run python:3.11 python train_model.py --data /shared/datasets/train Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: docker run python:3.11 python train_model.py --data /shared/datasets/train COMMAND_BLOCK: docker run python:3.11 python train_model.py --data /shared/datasets/train CODE_BLOCK: bash: docker: command not found Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: bash: docker: command not found CODE_BLOCK: bash: docker: command not found CODE_BLOCK: singularity pull docker://python:3.11 Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: singularity pull docker://python:3.11 CODE_BLOCK: singularity pull docker://python:3.11 COMMAND_BLOCK: ls -lh python_3.11.sif # -rwxr-xr-x 1 dalirnet dalirnet 385M Feb 25 09:15 python_3.11.sif Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: ls -lh python_3.11.sif # -rwxr-xr-x 1 dalirnet dalirnet 385M Feb 25 09:15 python_3.11.sif COMMAND_BLOCK: ls -lh python_3.11.sif # -rwxr-xr-x 1 dalirnet dalirnet 385M Feb 25 09:15 python_3.11.sif CODE_BLOCK: singularity exec python_3.11.sif python --version Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: singularity exec python_3.11.sif python --version CODE_BLOCK: singularity exec python_3.11.sif python --version COMMAND_BLOCK: docker run -v $(pwd):/workspace -w /workspace python:3.11 python analysis.py Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: docker run -v $(pwd):/workspace -w /workspace python:3.11 python analysis.py COMMAND_BLOCK: docker run -v $(pwd):/workspace -w /workspace python:3.11 python analysis.py CODE_BLOCK: singularity exec docker://python:3.11 python analysis.py Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: singularity exec docker://python:3.11 python analysis.py CODE_BLOCK: singularity exec docker://python:3.11 python analysis.py CODE_BLOCK: singularity exec \ --bind /shared/datasets:/data \ --bind /scratch/$USER:/scratch \ myimage.sif python train.py --data /data/train --output /scratch/checkpoints Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: singularity exec \ --bind /shared/datasets:/data \ --bind /scratch/$USER:/scratch \ myimage.sif python train.py --data /data/train --output /scratch/checkpoints CODE_BLOCK: singularity exec \ --bind /shared/datasets:/data \ --bind /scratch/$USER:/scratch \ myimage.sif python train.py --data /data/train --output /scratch/checkpoints COMMAND_BLOCK: docker run -e API_KEY=abc123 myimage python train.py Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: docker run -e API_KEY=abc123 myimage python train.py COMMAND_BLOCK: docker run -e API_KEY=abc123 myimage python train.py COMMAND_BLOCK: # Clean environment — recommended for reproducible experiments singularity exec --cleanenv myimage.sif python train.py # Clean env + only the variables you need singularity exec --cleanenv --env API_KEY=abc123 myimage.sif python train.py # Or use an env file singularity exec --cleanenv --env-file .env myimage.sif python train.py Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: # Clean environment — recommended for reproducible experiments singularity exec --cleanenv myimage.sif python train.py # Clean env + only the variables you need singularity exec --cleanenv --env API_KEY=abc123 myimage.sif python train.py # Or use an env file singularity exec --cleanenv --env-file .env myimage.sif python train.py COMMAND_BLOCK: # Clean environment — recommended for reproducible experiments singularity exec --cleanenv myimage.sif python train.py # Clean env + only the variables you need singularity exec --cleanenv --env API_KEY=abc123 myimage.sif python train.py # Or use an env file singularity exec --cleanenv --env-file .env myimage.sif python train.py COMMAND_BLOCK: docker run -p 8888:8888 mynotebook-image Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: docker run -p 8888:8888 mynotebook-image COMMAND_BLOCK: docker run -p 8888:8888 mynotebook-image CODE_BLOCK: singularity exec myimage.sif python -m notebook --port 8888 Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: singularity exec myimage.sif python -m notebook --port 8888 CODE_BLOCK: singularity exec myimage.sif python -m notebook --port 8888 CODE_BLOCK: FROM python:3.11-slim RUN pip install numpy pandas scikit-learn matplotlib COPY analysis.py /app/ WORKDIR /app CMD ["python", "analysis.py"] Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: FROM python:3.11-slim RUN pip install numpy pandas scikit-learn matplotlib COPY analysis.py /app/ WORKDIR /app CMD ["python", "analysis.py"] CODE_BLOCK: FROM python:3.11-slim RUN pip install numpy pandas scikit-learn matplotlib COPY analysis.py /app/ WORKDIR /app CMD ["python", "analysis.py"] CODE_BLOCK: Bootstrap: docker From: python:3.11-slim %post pip install numpy pandas scikit-learn matplotlib %files analysis.py /app/ %environment export LC_ALL=C %runscript cd /app exec python analysis.py Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: Bootstrap: docker From: python:3.11-slim %post pip install numpy pandas scikit-learn matplotlib %files analysis.py /app/ %environment export LC_ALL=C %runscript cd /app exec python analysis.py CODE_BLOCK: Bootstrap: docker From: python:3.11-slim %post pip install numpy pandas scikit-learn matplotlib %files analysis.py /app/ %environment export LC_ALL=C %runscript cd /app exec python analysis.py CODE_BLOCK: singularity build myenv.sif docker://myregistry/gpu-image:latest Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: singularity build myenv.sif docker://myregistry/gpu-image:latest CODE_BLOCK: singularity build myenv.sif docker://myregistry/gpu-image:latest COMMAND_BLOCK: singularity build --sandbox myenv/ myenv.def # create writable folder singularity shell --writable myenv/ # shell in and install stuff # pip install some-new-package # python -c "import some_new_package" # test it sudo singularity build myenv.sif myenv/ # freeze when happy Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: singularity build --sandbox myenv/ myenv.def # create writable folder singularity shell --writable myenv/ # shell in and install stuff # pip install some-new-package # python -c "import some_new_package" # test it sudo singularity build myenv.sif myenv/ # freeze when happy COMMAND_BLOCK: singularity build --sandbox myenv/ myenv.def # create writable folder singularity shell --writable myenv/ # shell in and install stuff # pip install some-new-package # python -c "import some_new_package" # test it sudo singularity build myenv.sif myenv/ # freeze when happy COMMAND_BLOCK: docker run --gpus all gpu-image:latest python train.py Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: docker run --gpus all gpu-image:latest python train.py COMMAND_BLOCK: docker run --gpus all gpu-image:latest python train.py CODE_BLOCK: singularity exec --nv gpu-image.sif python train.py Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: singularity exec --nv gpu-image.sif python train.py CODE_BLOCK: singularity exec --nv gpu-image.sif python train.py CODE_BLOCK: singularity exec --nv \ --bind /shared/datasets:/data \ --bind /scratch/$USER:/scratch \ gpu-image.sif \ python train.py \ --data /data/train \ --output /scratch/checkpoints \ --epochs 100 \ --batch-size 512 \ --gpus 4 Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: singularity exec --nv \ --bind /shared/datasets:/data \ --bind /scratch/$USER:/scratch \ gpu-image.sif \ python train.py \ --data /data/train \ --output /scratch/checkpoints \ --epochs 100 \ --batch-size 512 \ --gpus 4 CODE_BLOCK: singularity exec --nv \ --bind /shared/datasets:/data \ --bind /scratch/$USER:/scratch \ gpu-image.sif \ python train.py \ --data /data/train \ --output /scratch/checkpoints \ --epochs 100 \ --batch-size 512 \ --gpus 4 CODE_BLOCK: singularity remote login singularity push myimage.sif library://myname/default/myimage:v1.0 Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: singularity remote login singularity push myimage.sif library://myname/default/myimage:v1.0 CODE_BLOCK: singularity remote login singularity push myimage.sif library://myname/default/myimage:v1.0 CODE_BLOCK: cp myimage.sif /shared/containers/team-env.sif Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: cp myimage.sif /shared/containers/team-env.sif CODE_BLOCK: cp myimage.sif /shared/containers/team-env.sif CODE_BLOCK: #!/bin/bash singularity instance start postgres.sif db sleep 5 singularity instance start --env DATABASE_URL=postgresql://localhost/mydb tracker.sif tracker singularity instance list Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: #!/bin/bash singularity instance start postgres.sif db sleep 5 singularity instance start --env DATABASE_URL=postgresql://localhost/mydb tracker.sif tracker singularity instance list CODE_BLOCK: #!/bin/bash singularity instance start postgres.sif db sleep 5 singularity instance start --env DATABASE_URL=postgresql://localhost/mydb tracker.sif tracker singularity instance list - No daemon — just run it - Single-file images — cp for instant reproducibility - Same user inside and outside — no permission headaches - Simple GPU access — just --nv - Works on shared HPC clusters without special privileges - Building images needs root (--fakeroot or --remote as workaround) - No network isolation - No built-in compose/orchestration - No layer caching — full rebuild every time - Host environment can leak in — always use --cleanenv - 🐳 Docker: "How do I isolate this app?" - 🔬 Singularity: "How do I bring this app into the researcher's environment?"