Published on

Debugging Rails Connectivity Issues with the DB, Queue and Sidekiq on AWS

Authors

I recently had to debug connectivity issues to a Rails app in containers deployed to a cluster. Being new to Rails this was not immediately obvious. I had to debug what was causing DB, Sidekiq and queue connectivity issues on a newly deployed app.

I eventually managed to track down the issue by debugging as follows

Checking Environment Details

System Level

First, let us check if the app has the correct DB connection string. To do this go into the container running your rails app.

In the normal bash terminal run: env

In the output of env, you should see an environment variable matching what you pass to the container. This variable should have the DB connection string as its value.

Database Connectivity

Next, we need to confirm that our container and host can actually physically connect to the DB. This will let us rule out AWS security group configuration issues if we can connect. If we cannot it likely means the DB security group's inbound rule has to be updated to allow connections from the host where the container is running.

Run the following on the host and then again in the container to confirm we can connect to the DB without issues:

telnet dbHostURI dbPort
  • This article from AWS describes this process in more detail. Using this approach is very useful in picking up networking issues.
  • If you are using an alpine container you will need to install telnet as follows: apk add busybox-extras

If you connect you will be given the option of disconnecting (this means that your host can see the DB and is authorized to hit the DB URI)

We now know we can successfully connect so there are no issues with security groups or other networking issues. Let us now confirm the DB connection settings are correct. In the host or the container run the following

psql -Atx -U yourDbUserName -d yourDbName -h dbHostURI -p dbPort

You will be prompted for a password. Type/paste it in. If all is good you should be dropped into a psql shell.

Rails Level

The last step in debugging is confirming that Rails has the right config.

To do this go onto the machine/container running your rails app, then:

  • cd to the root of your Rails project
  • Run rails console to put you into a console in your app's environment
  • Type: ActiveRecord::Base.connection_config

If the connection string returned matches the one you tested earlier then you should be all good.

Redis Queue Connectivity

To confirm Sidekiq will work properly we first need to ensure that we have no connectivity issues to the queue. To do this we will install the redis-cli on the host machine.

Then using the cli try to connect as follows:

redis-cli -h yourRedisURI

In the above:

  • Use -p if the URL has a different port

To confirm that Redis responds in the cli type ping you should get back PONG

If you need to install Redis on the host follow these steps

If connectivity is working at this point, Sidekiq should have no issues connecting to the queue as long as the connection string is correct.

Sidekiq: ArgumentError: invalid uri scheme

At one stage I ran into this error.

This happens if you do not prepend redis:// to the front of your Redis URI.