Docker/Apptainer HATS@LPC

General Information

Docker and Apptainer are platform as a service products which use OS-level virtualization within containers. Containerization brings a high level of flexibility, isolation, and reproducibility to your workflow. It also allows pieces of software to be run on an OS other than the one for which they were designed. This hands-on tutorial will introduce the Docker and Apptainer environments and show how working within a container is a powerful alternative to our traditional computer cluster centric workflow. We will discuss the advantages and disadvantages of working within a container as well as the differences between Docker and Apptainer. Participants will also be introduced to some CMS/CVMFS focused containers and will get a chance to explore those environments. Within these containers users will have access to CVMFS and all of the software associated with it (i.e. CMSSW) as well as more graphics oriented software, like cmsShow, through X11 and VNC.

What: An opinionated introduction to using Docker and Apptainer as software development tools for use with CMSSW.

Who: The course is aimed at anyone looking to containerize their workflow. You don't need to have any previous knowledge of the tools that will be presented at the workshop.

Where: This training will take place online. The instructors will provide you with the information you will need to connect to this meeting.

When: July 16, 2024. Add to your Google Calendar.

Requirements: Participants must bring a laptop with a Mac, Linux, or Windows operating system (not a tablet, Chromebook, etc.) that they have administrative privileges on. They should have a few specific software packages installed (listed on the setup page).

References: Many of the modules were taken from the Introduction to Docker (also) tutorial and the Software containers for CMSSW tutorial.

Accessibility: We are dedicated to providing a positive and accessible learning environment for all. Please notify the instructors in advance of the workshop if you require any accommodations or if there is anything we can do to make this workshop more accessible to you.

Contact: Please email pedrok at fnal.gov for more information.

Prerequisites

Follow the setup instructions on the setup page.

Have access to a computing cluster with Apptainer installed on it. We will be using the CMSLPC cluster for this demo (account request directions).

(optional) A VO registered grid certificate (request directions) setup on your local machine (setup directions)

(optional) A GitLab account (CMS users have access to CERN’s GitLab instance)

Schedule

Asynchronous (do before we meet)

	Setup	Download files required for the lesson
00:00	1. Introduction	What are containers? What is Docker? What is it used for? What are its components? How is Docker different on OSX/Windows/Linux?
00:10	2. Pulling Images	How are images downloaded? How are images distinguished?
00:25	3. Running Containers	How are containers run? How do you monitor containers? How are containers exited? How are containers restarted?
00:40	4. Removal of Containers and Images	How do you cleanup old containers? How do you delete images?
00:50	5. File I/O with Containers	How do containers interact with my local file system?
01:00	Finish

Synchronous (lessons to be covered live)

00:00	6. Accessing CVMFS From Docker Locally	How can I access CVMFS from my computer? How can I access CVMFS from Docker?
00:45	7. Using the cms-cvmfs-docker Image	What is so special about this image? What problems does it solve and what still remain? How do I interact with this image?
01:25	8. Using Full CMSSW Containers	How can I obtain a standalone CMSSW container? What are the advantages and disadvantages of this type of container?
01:35	9. Running Containers on CMSLPC/LXPLUS Using Apptainer	How can I run a container on CMSLPC/LXPLUS?
02:05	10. Using unpacked.cern.ch	What is unpacked.cern.ch? How can I use unpacked.cern.ch?
02:20	11. Container Security	What are the best practices when it comes to container security? What are the Fermilab security dos and don’ts?
02:40	Finish

Bonus (time permitting)

00:00	12. Writing Dockerfiles and Building Images	How are Dockerfiles written? How are Docker images built?
00:30	13. Using CMD and ENTRYPOINT in Dockerfiles	How are default commands set in Dockerfiles?
00:30	14. Gitlab CI for Automated Environment Preservation	How can GitLab CI and Docker work together to automatically preserve my analysis environment? What do I need to add to my GitLab repo(s) to enable this automated environment preservation?
01:20	15. SSH Credentials	How do I access my SSH credentials within a container?
01:30	16. Building derived images from the cms-cvmfs-docker base image	Why is it harder to build derived images for this container? When do I need to use a workaround to build a derived image and when is it okay to use standard methods? Are there any limitations on what typed of derived images I can and cannot build?
02:10	17. Using Buildah and Podman	Why should I use Buildah and Podman? Are these cross-platform solutions? What limitation or extras do Buildah and Podman possess compared to Moby BuildKit and Docker?
03:10	18. Containerizing a CMSSW Working Area	How do I create an image which contains my CMSSW working area? How can I mount CVMFS inside that image?
03:50	19. Advanced Usage of Apptainer	How do I modify the default behavior of Apptainer when entering a container? How can I submit batch jobs when inside an Apptainer container? How can I use Apptainer when I’m already inside an Apptainer container?
04:30	Finish

The actual schedule may vary slightly depending on the topics and exercises chosen by the instructor.