If you have a Rails application that connects to an external, read-only database, there are a few ways to wire things up in the test suite. In my scenario, I needed some data seeded into that external database to adequately test the flow of my application.

If you are in a similar situation, this article may be of some help. I found several posts in my research but some were outdated, some contained rather complicated and verbose rake task overrides which I ultimately did not need, and others just did not work as advertised when tried in my environment.

After consulting members of my team, here is the solution that we ended up finding.

Context

We have a base class for all active record models that interact with the external database that looks like this.

module MyExternal
  class Base < ApplicationRecord
    include ReadonlyExternalDatabase

    self.abstract_class = true

    connects_to database: { reading: :external_database, writing: :external_database }
  end
end

ReadonlyExternalDatabase restricts major writes from going through to this external database as an extra layer of protection.

module ReadonlyExternalDatabase
  def readonly?
    true
  end

  def destroy(*)
    raise ActiveRecord::ReadOnlyRecord
  end

  def delete(*)
    raise ActiveRecord::ReadOnlyRecord
  end

  def self.destroy_all(*)
    raise ActiveRecord::ReadOnlyRecord
  end

  def self.delete_all(*)
    raise ActiveRecord::ReadOnlyRecord
  end
end

Solution

First, the default declaration for this external database has replica: true since it is pointing to another database.

# database.yml
external_database_default: &external_database_default
  <<: *default
  replica: true

However, the test database is not a replica.

# database.yml
test:
  primary:
    <<: *default
    database: my_app_test
  external_database:
    <<: *external_database_default
    replica: false
    database: external_database_test

Due to the limitation of having a read-only database, seeding the data becomes a bit more difficult. If that isn't a requirement for you, you could just include a schema file for the external database in db/ and run your own seed file against it. Instead of doing a schema load, I decided to load the database schema and populate it with data via an .sql file. To get this file, I loaded my development copy of this external database to a state that I thought would be good for my test suite, and ran this command.

pg_dump -U -d external_database_development -f external_database_dump.sql

Some fixup was required in this file. I had to go through and replace any instances of the user (which happened to be my name) with "postgres."

I also ran into an error message that indicated I needed to include some additional postgres plugins in that file, which I did.

This SQL file represents the instructions to create the tables and populate them, but it can be very large. I decided to store this file in db/dump and I opted to gzip the file to save space.

gzip db/dump/external_database_dump.sql

Rake Task

Next, I added a rake task to apply this SQL file.

# lib/tasks/db.rake

namespace :db do
  namespace :external do
    task load: :environment do
      if File.exist?(Rails.root.join(external_database_filename))
        system("PGPASSWORD='' gunzip < #{external_database_filename} | psql -U postgres external_database_test")
      end
    end
  end
end

def external_database_filename
  'db/dump/external_database_dump.sql.gz'
end

This rake task runs a system command that un-gzips the SQL file and pipes the contents into a psql command which ultimately executes the SQL.

You may not want to run this before all your tests, since it could be slow. We added a flag that rspec can pick up on which will cause the SQL to only be executed once and if the test being run requires it.

# spec/support/external_database.rb
require 'rake'
Rails.application.load_tasks

RSpec.configure do |config|
  config.when_first_matching_example_defined(external_database: true) do
    connection = MyExternal::Base.connection
    database_not_yet_built = connection.execute("SELECT table_name FROM information_schema.tables WHERE table_schema = 'public'").none?
    Rake::Task['db:external:load'].invoke if database_not_yet_built
  end
end

# example of using the flag
RSpec.describe MyExternal::MyModel, :external_database, type: :model do

And with all that, it was working! Now, in my tests, I can do things like this to interact with the seeded data.

MyExternal::Project.find_by(...)

Semaphore

A few changes needed to be made in the semaphore configuration to get these tests working on CI. Because there was no schema file for the new external database, doing a typical bin/rails db:setup db:test:prepare would throw an error because ultimately it was trying to load the schema for each attached database. Instead, we did the following.

 jobs:
   - name: RSpec Unit Tests
     commands:
         - bin/rails db:create db:schema:load:primary

Hopefully you found this helpful!

Update for Rails 7.1

After upgrading to Rails 7.1.4.2 some issues were uncovered. Setting replica: false for the test database no longer worked.

This command no longer worked as a check for whether or not the db had been created.

database_not_yet_built = connection.execute("SELECT table_name FROM information_schema.tables WHERE table_schema = 'public'").none?

I changed the rspec setup to be the following:

RSpec.configure do |config|
  config.before(:suite) do
    ApplicationRecord.connection.execute("drop database if exists external_database")
    ApplicationRecord.connection.execute("create database external_database")
    Rake::Task['db:external:load'].invoke
  end
end

Now set the external database back to replica: true and add database_tasks: false

external_default: &external_default
  <<: *default
  database_tasks: false

And in CI I changed it from

bin/rails db:create db:schema:load:primary

back to

bundle exec rails db:setup db:test:prepare

Since it is now a replica again, you can delete the schema file for the external database.