A while back, when I was just starting to get more hands-on with AWS and infrastructure, I ran into a performance issue that took me longer than I’d like to admit to figure out.

At the time, we were upgrading one of our Rails applications to a newer version.

Instead of upgrading the existing EC2 instance, we decided to provision a new one and deploy the application there. It felt like the safer approach since we could leave the old server untouched until we were confident everything was working as expected.

We already had an RDS PostgreSQL instance that had been running without any issues, so we decided to reuse it instead of provisioning a new database and migrating all the data.

The deployment itself went smoothly. The application booted up, everything looked healthy, and there were no obvious errors.

Then we started testing it.

The application felt noticeably slower than before. Nothing was failing, but every page that interacted with the database had a slight delay. It wasn’t enough to make the application unusable, but it was enough to know something wasn’t right.

Since this happened immediately after the upgrade, we assumed the problem was in the application.

We started by checking the usual suspects:

  • Recent code changes
  • SQL queries
  • Missing indexes
  • N+1 queries
  • Gem updates

Everything looked normal.

The SQL queries were executing quickly, PostgreSQL wasn’t under any unusual load, and the EC2 instance wasn’t showing signs of resource exhaustion either.

After spending quite a bit of time looking at the application, we decided to step back and look at the infrastructure instead.

That’s when we noticed it.

The new EC2 instance had been provisioned in a different AWS region than our existing RDS instance.

When we originally built the application, both resources were deployed in the same region. During the upgrade, we only created a new EC2 instance and reused the existing database. None of us thought to double-check that they were still in the same region.

That small oversight turned out to be the entire problem.

Every database query now had to travel between AWS regions before PostgreSQL could execute it.

A single query might only add a small amount of latency, but Rails applications rarely execute just one query per request. By the time a page finished loading, that extra network latency had accumulated across dozens of database calls.

The database wasn’t slow.

The application wasn’t slow.

The network path between them was.

Once we found the issue, the fix was straightforward.

We provisioned a new RDS instance in the same region as the EC2 server, migrated the data, updated the application’s database configuration, and tested again.

The improvement was immediate.

Response times returned to what we expected, and we didn’t have to change a single line of application code.

Looking Back

At the time, I was mostly focused on getting the application running. I wasn’t thinking much about how the infrastructure itself could affect performance.

Looking back, it makes perfect sense. If your application and database are talking to each other constantly, they should be as close to each other as possible.

Since then, checking which region my services are running in has become one of the first things I verify whenever I’m provisioning new infrastructure or migrating an application.

It’s a simple check, but it’s one that I learned the hard way.