Docker for Dummies (Part-3 Docker Compose)
Multi app/services architecture with Docker Compose
Often, we find ourselves in scenarios where our primary service requires support from additional services. For instance, your Express server might need a PostgreSQL or MongoDB instance. How do you containerize such a setup? In this project, we will delve into precisely that.
I understand that this tutorial is quite comprehensive and may take 2-3 days to fully grasp, but those who complete it will be well-equipped to install any Docker Compose-based project locally.
This project will simple express ecommerce project with just single endpoint that uses PostgreSQL, caches value in Redis with expiry of 1 minute in Redis. Simplest multi-service system (in my opinion).
For this project, there is a directory named ecommerce-search-microservice
. Inside, we have multiple files, such as seed.sql
, main.js
, .env
, and package.json
. This blog will be divided into three parts. First, we will cover how to set up this project without Docker. Then, we will add Dockerfile(s) and docker-compose.yml
. Finally, we will initialize the same project using Docker Compose and learn a few concepts about shared networks and shared volumes.
Starting Project without Docker Compose
As we already know, we will need PostgreSQL and Redis instances. So, you need to install them first. You can check out these links for installation:
Redis Installation: redis.io/docs/latest/operate/oss_and_stack/..
PostgreSQL Installation: postgresql.org/download
Since we already have Docker installed, let's use it to have "containerized versions" of Redis and PostgreSQL instances.
docker run --name my-redis -d -p 6379:6379 redis:latest # Start Redis Container
docker run --name my-postgres -e POSTGRES_USER=your_username -e POSTGRES_PASSWORD=your_password -e POSTGRES_DB=your_database -d -p 5432:5432 postgres:latest # Postgresql
Steps to start this project
Firstly, start the above PostgreSQL and Redis instances.
Now, edit your
.env
file according to the environment variables in your PostgreSQL and Redis configuration.Seed the data using
seed.sql
. I used an vscode extension for this task.Install the dependency with
npm install
and run the project withnpm start
If everything went well, then your api should be live at http://localhost:3000/search?q=headphones
.
Run this multiple times you will see source
of the data as well.
{
"source": "database", //other value is redis
"result": [
{
"id": 3,
"product_name": "Sony Headphones-d012",
"price": "59.99"
},
{
"id": 4,
"product_name": "Sony Headphones-d012",
"price": "59.99"
},
{
"id": 9,
"product_name": "Sony Headphones-d012",
"price": "59.99"
}
]
}
Introduction to Docker Compose
Although running this project is a simple task, I still need to explain the steps. When a new developer joins a company, they need to install multiple software like Redis, PostgreSQL, etc. This can be tedious. What if we could do all the above steps with a single command? Let me introduce you to "Docker Compose." With Docker Compose, you can specify the project's requirements within the project itself. For example, this project will require Redis, PostgreSQL, RabbitMQ, etc.
Lets start containerising this express app.
Steps to Containerization
[RECOMMENDATION] First, shut down your Docker container if it is already running.
Create
Dockerfile
and.dockerignore
files. I am putting the.env
file in the.dockerignore
file because I don't want it to be copied with theCOPY . .
command in the Dockerfile. In our case, this file will not change at all.FROM node:22-alpine # Create app directory WORKDIR /usr/src/app # Install app dependencies COPY package*.json ./ RUN npm install # Bundle app source COPY . . EXPOSE 5000 CMD ["npm", "start"]
Install Docker Compose (if you don't have it already). To check whether your system have compose. Use
docker-compose --version
ordocker compose --version
.Create a Docker Compose file named
compose.yml
. We will explain this code in the next point. To run this Compose configuration, you need to executedocker-compose up compose.yml
in the ecommerce project directory (make sure to check out themain
branch).# NOTE: mentioning version is now deprecated. version: "3.6" networks: ecommerce_network: driver: bridge volumes: postgres_data: services: postgres: image: postgres:alpine ports: - 5432:5432 environment: POSTGRES_DB: postgres POSTGRES_USER: postgres POSTGRES_PASSWORD: mysecretpassword POSTGRES_HOST_AUTH_METHOD: scram-sha-256 networks: - ecommerce_network volumes: - postgres_data:/var/lib/postgresql/data redis: image: redis:6.2.5-alpine networks: - ecommerce_network api: build: context: . dockerfile: Dockerfile depends_on: - postgres - redis ports: - 5000:5000 volumes: - ./node_modules:/usr/src/app/node_modules - .:/usr/src/app networks: - ecommerce_network command: npm run start environment: PORT: 5000 PGUSER: postgres PGHOST: postgres PGDATABASE: postgres PGPASSWORD: mysecretpassword PGPORT: 5432 REDIS_URL: redis://redis:6379
Here is the explanation of the code above. It mainly defines services, their environment variables, port mapping, and network-and-volume-layers. The first few things are easy to understand. However, networks and volumes are deeper topics that we will cover later in this blog.
Networks and Volume in Docker (and Docker Compose)
Network Isolation and Volume Allocation are very common containerization techniques, as we want maximum independence among containers and minimum security risk/uncertainty. Now, we will discuss about these topics briefly.
Network Isolation
We need a network because it allows containers to communicate with each other and with the outside world. Docker containers canโt talk to each other by default. When a service is running on localhost, it means localhost is relative to the container itself, not the host machine. In a multi-container setup with Docker Compose, each service runs on localhost relative to its own network but also shares a common default network to communicate with each other. However, we can create our own networks. For example, I have created my own network called
ecommerce_network
.You might be wondering why we need to create an additional network when one is already provided. The answer is controlled access (i.e., our ability to decide which container can access another container, preventing malicious containers from accessing sensitive information) and ecosystem isolation (i.e., if one ecosystem is compromised, the others remain unaffected due to network isolation).
Commands to make your own docker network.
# create a network docker network create my_custom_network # build your image with standard command docker build -t image_tag . # start your image with network(you can start multiple services with same network) docker run -d -p 3000:3000 --name backend --network my_custom_network image_tag
In our e-commerce example, we have a single network named
ecommerce_network
, which can only be accessed from inside, not from outside. However, we can still access each individual component from outside, such as when we need to run Redis insights or an SQL viewer, by using global port mapping to the host machine.
Volume/Disk Allocation:
We also have volume mapping for PostgreSQL data because Docker containers are transient by nature. This means that when we stop a container, all the data inside it is lost as the system reclaims memory back to the host. To prevent this data loss, we allocate some space on the host machine itself. This allocated space remains intact even when the container is shut down. We then map this allocated space to the container, ensuring that the data persists across container restarts.
For example, we might use a command like this to create a volume:
docker volume create pgdata
And then, when running the PostgreSQL container, we map this volume to the appropriate directory inside the container:
docker run -d --name postgres -v pgdata:/var/lib/postgresql/data postgres
This setup ensures that our PostgreSQL data is stored on the host machine and not lost when the container stops.
It's important to note that different services might have different storage locations. For instance, Redis might store its data in a different directory compared to PostgreSQL. Therefore, you need to create and map volumes accordingly for each service.
[PERSONAL ADVICE]:To be honest, I relied heavily on ChatGPT to understand and implement this part because the storage locations and volume mapping can vary significantly between different services.
This approach allows us to maintain data integrity and persistence across various services running in Docker containers.
I think now, you can read this diagram to get mental model, arrows denotes the direction of information dependence(instead of direction of information flow.)
Ending
I have shared everything I learned about Docker last month. The only topic I haven't covered is Docker multi-stage builds. You can find tutorials and videos on that; it's not too hard if you understand the basics.
I know this blog might seem overwhelming because I've covered a lot in one place. But trust me, if you try to build the three projects I mentioned, you'll grasp Docker basics. I suggest spending 24-48 hours containerizing an application with the help of ChatGPT and some tutorials or videos. If you have any questions, feel free to comment, and I'll respond. Cool.
Bye-bye.