Docker Containers Troubleshooting - Debug , Fix , Run , Scale

Yeah i know, I will ask my Gen AI agent to write , debug, fix my code. 

Thats what many of us think now a days. What if your AI agent dint see some issue before hand and you need to fix its mistakes. What if you need to review the fix before you roll out to prod environment.

Being a developer never stops, may that be working with your old school IDEs writing your own code, or working with your code writen by a Gen AI agent. Always you should know what you are doing.

Just like everyone else i asked Claude to create me a sample Docker compose project and it did a great job.

It started with a simple arhcitecture diagram, explined me what different components are doing.

Then it spit out the code , and in first glance it all looks so perfect. I moved the code to my RHEL10 VM on Virtualbox to test it out.

If you wanna try it by your self, here is the code hosted on Github. https://github.com/pratappilaka24/Docker-Compose-Troubleshooting

I am excited and ran it. It started fine

then this happened.

My App container provisioning failed its health check. Its showing unhealthy 

Checked the logs and logs seems fine.

Next step is to inspect the container to see why the health check is failing.

Here is the issue. Claude suggested i use "python:3.12-slim" image to run my app container, but dint realized, it dint come with curl installed. From above inspect command we find the missing dependency.

Let me fix it by adding the curl installation in Dockerfile.

We chnaged the definition of our App container image.

For this to work, you need to stop app container, remove app container and remove app image.

Thus next compose run, docker will recreate new image based on updated definition, and create app container based on it.

Here we go again.

My app container failed to start again. What AI dint know is that i am already running httpd service on my RHEL vm, where posrts 80 and 443 are pre allocated. 

So now we change nginx.conf and docker-compose files to run the app on a different port.

After all these fixes, we were able to deploy all App, DB, Redis and DB containers successfully. 

Testing the app by curl http://localhost.

Its an 500 internal service error. Something to do with our code. Lets check the logs.

Lets start with nginx container logs. this doesnt show much information apart form there is a 500 internal server error. this means app is crashing.

Next step is to check app container logs.

Wait a second, why does app container logs says Visits table in postgres db doesnot exists.

But if you check in our code , db-init.sql script is creating the table when postgres db container is provisioned.

After some research , found out the db-init.sql, also called seed script runs only once when the volume is created. but when we decomissioned containers , the voulme is left behind. 

When we run Docker compose again, the containers were recreated and due to existing volume , the seed script never ran, thus failing to run the seed script creating the required postgres table.

So this time we take down all the containers including the volume.

Now we will recreate the contianers using docker compose and test our app.

Vallah , the containers deployed and the user hits on the app and the db is registering the user hits, and redis is caching the count.  If you look at the output, you can see the count is being fetched from cache for cuncurrent hits. 

Now look at how we can scale up the app containers. Initial Docker compose ps shows 1 of app, db, nginx and redis containers. Now scale up app to 3 and you can see 3 app containers serving the user hits in union.

Now scaling down one app container.

At last by digging through the logs and figuring out issues and fixing things is what makes us Engineers and Developers, not just writing code.