Docker for Dummies (Part-3 Docker Compose)

Often, we find ourselves in scenarios where our primary service requires support from additional services. For instance, your Express server might need a PostgreSQL or MongoDB instance. How do you containerize such a setup? In this project, we will delve into precisely that.
I understand that this tutorial is quite comprehensive and may take 2-3 days to fully grasp, but those who complete it will be well-equipped to install any Docker Compose-based project locally.

This project will simple express ecommerce project with just single endpoint that uses PostgreSQL, caches value in Redis with expiry of 1 minute in Redis. Simplest multi-service system (in my opinion).

For this project, there is a directory named ecommerce-search-microservice. Inside, we have multiple files, such as seed.sql, main.js, .env, and package.json. This blog will be divided into three parts. First, we will cover how to set up this project without Docker. Then, we will add Dockerfile(s) and docker-compose.yml. Finally, we will initialize the same project using Docker Compose and learn a few concepts about shared networks and shared volumes.

Starting Project without Docker Compose

As we already know, we will need PostgreSQL and Redis instances. So, you need to install them first. You can check out these links for installation:

Redis Installation: redis.io/docs/latest/operate/oss_and_stack/..

PostgreSQL Installation: postgresql.org/download

Since we already have Docker installed, let's use it to have "containerized versions" of Redis and PostgreSQL instances.

docker run --name my-redis -d -p 6379:6379 redis:latest # Start Redis Container
docker run --name my-postgres -e POSTGRES_USER=your_username -e POSTGRES_PASSWORD=your_password -e POSTGRES_DB=your_database -d -p 5432:5432 postgres:latest # Postgresql

Steps to start this project

1. Firstly, start the above PostgreSQL and Redis instances.

2. Now, edit your .env file according to the environment variables in your PostgreSQL and Redis configuration.

3. Seed the data using seed.sql. I used an vscode extension for this task.

4. Install the dependency with npm install and run the project with npm start

If everything went well, then your api should be live at http://localhost:3000/search?q=headphones.

Run this multiple times you will see source of the data as well.

{
  "source": "database", //other value is redis
  "result": [
    {
      "id": 3,
      "product_name": "Sony Headphones-d012",
      "price": "59.99"
    },
    {
      "id": 4,
      "product_name": "Sony Headphones-d012",
      "price": "59.99"
    },
    {
      "id": 9,
      "product_name": "Sony Headphones-d012",
      "price": "59.99"
    }
  ]
}

Introduction to Docker Compose

Although running this project is a simple task, I still need to explain the steps. When a new developer joins a company, they need to install multiple software like Redis, PostgreSQL, etc. This can be tedious. What if we could do all the above steps with a single command? Let me introduce you to "Docker Compose." With Docker Compose, you can specify the project's requirements within the project itself. For example, this project will require Redis, PostgreSQL, RabbitMQ, etc.

Lets start containerising this express app.

Steps to Containerization

1. [RECOMMENDATION] First, shut down your Docker container if it is already running.

2. Create Dockerfile and .dockerignore files. I am putting the .env file in the .dockerignore file because I don't want it to be copied with the COPY . . command in the Dockerfile. In our case, this file will not change at all.

     FROM node:22-alpine
   
     # Create app directory
     WORKDIR /usr/src/app
   
     # Install app dependencies
     COPY package*.json ./
   
     RUN npm install
   
     # Bundle app source
     COPY . .
     EXPOSE 5000
     CMD ["npm", "start"]
   

3. Install Docker Compose (if you don't have it already). To check whether your system have compose. Use docker-compose --version or docker compose --version.

   

4. Create a Docker Compose file named compose.yml. We will explain this code in the next point. To run this Compose configuration, you need to execute docker-compose up compose.yml in the ecommerce project directory (make sure to check out the main branch).

     # NOTE: mentioning version is now deprecated. 
     version: "3.6"  
   
     networks:
       ecommerce_network:
         driver: bridge
   
     volumes:
       postgres_data:
   
     services:
       postgres:
         image: postgres:alpine
         ports:
           - 5432:5432
         environment:
           POSTGRES_DB: postgres
           POSTGRES_USER: postgres
           POSTGRES_PASSWORD: mysecretpassword
           POSTGRES_HOST_AUTH_METHOD: scram-sha-256
         networks:
           - ecommerce_network
         volumes:
           - postgres_data:/var/lib/postgresql/data
   
       redis:
         image: redis:6.2.5-alpine
         networks:
           - ecommerce_network
       api:
         build:
           context: .
           dockerfile: Dockerfile
         depends_on:
           - postgres
           - redis
         ports:
           - 5000:5000
         volumes:
           - ./node_modules:/usr/src/app/node_modules
           - .:/usr/src/app
         networks:
           - ecommerce_network
         command: npm run start 
         environment:
           PORT: 5000
           PGUSER: postgres
           PGHOST: postgres
           PGDATABASE: postgres
           PGPASSWORD: mysecretpassword
           PGPORT: 5432
           REDIS_URL: redis://redis:6379
   

5. Here is the explanation of the code above. It mainly defines services, their environment variables, port mapping, and network-and-volume-layers. The first few things are easy to understand. However, networks and volumes are deeper topics that we will cover later in this blog.

   

Networks and Volume in Docker (and Docker Compose)

1. Network Isolation and Volume Allocation are very common containerization techniques, as we want maximum independence among containers and minimum security risk/uncertainty. Now, we will discuss about these topics briefly.

2. Network Isolation

   1. We need a network because it allows containers to communicate with each other and with the outside world. Docker containers can’t talk to each other by default. When a service is running on localhost, it means localhost is relative to the container itself, not the host machine. In a multi-container setup with Docker Compose, each service runs on localhost relative to its own network but also shares a common default network to communicate with each other. However, we can create our own networks. For example, I have created my own network called ecommerce_network.

      

   2. You might be wondering why we need to create an additional network when one is already provided. The answer is controlled access (i.e., our ability to decide which container can access another container, preventing malicious containers from accessing sensitive information) and ecosystem isolation (i.e., if one ecosystem is compromised, the others remain unaffected due to network isolation).

   3. Commands to make your own docker network.

        # create a network
        docker network create my_custom_network
        # build your image with standard command
        docker build -t image_tag .
        # start your image with network(you can start multiple services with same network)
        docker run -d -p 3000:3000 --name backend --network my_custom_network image_tag
      

   4. In our e-commerce example, we have a single network named ecommerce_network, which can only be accessed from inside, not from outside. However, we can still access each individual component from outside, such as when we need to run Redis insights or an SQL viewer, by using global port mapping to the host machine.

3. Volume/Disk Allocation:

   1. We also have volume mapping for PostgreSQL data because Docker containers are transient by nature. This means that when we stop a container, all the data inside it is lost as the system reclaims memory back to the host. To prevent this data loss, we allocate some space on the host machine itself. This allocated space remains intact even when the container is shut down. We then map this allocated space to the container, ensuring that the data persists across container restarts.

      For example, we might use a command like this to create a volume:

        docker volume create pgdata
      

      And then, when running the PostgreSQL container, we map this volume to the appropriate directory inside the container:

        docker run -d --name postgres -v pgdata:/var/lib/postgresql/data postgres
      

   2. This setup ensures that our PostgreSQL data is stored on the host machine and not lost when the container stops.

   3. It's important to note that different services might have different storage locations. For instance, Redis might store its data in a different directory compared to PostgreSQL. Therefore, you need to create and map volumes accordingly for each service.

   4. [PERSONAL ADVICE]:To be honest, I relied heavily on ChatGPT to understand and implement this part because the storage locations and volume mapping can vary significantly between different services.

   5. This approach allows us to maintain data integrity and persistence across various services running in Docker containers.

4. I think now, you can read this diagram to get mental model, arrows denotes the direction of information dependence(instead of direction of information flow.)

   

Ending

I have shared everything I learned about Docker last month. The only topic I haven't covered is Docker multi-stage builds. You can find tutorials and videos on that; it's not too hard if you understand the basics.

I know this blog might seem overwhelming because I've covered a lot in one place. But trust me, if you try to build the three projects I mentioned, you'll grasp Docker basics. I suggest spending 24-48 hours containerizing an application with the help of ChatGPT and some tutorials or videos. If you have any questions, feel free to comment, and I'll respond. Cool.

Bye-bye.