The definitive guide to tackle technical deb in the Rails project

How to survive to a legacy monolithic Ruby on Rails application with the help of Rails Engines. As a disclosure, this article is not about extracting modules from the host app to a Rails Engine but using the power of a Rails Engine to let you rethink and rewrite a codebase which needs to be.

If you have ever be confronted to a Rails project it scares you because the codebase is so messy you are not confident to touch anything at all, even understand what is going on in some part of the code because 80% of the code is about fixing edge cases and the flaw of the initial design. Your conclusion reaches quickly the fact everything is just worth to throw away but as surprising as it is it works and it is out of the question to start a re-write from scratch, because, well, it works. Too often executives don’t understand why a codebase, which works, need to be rewritten and why they need to put time and money on that, but this article is not about management.

Let’s pretend the startup was just funded and start hiring, you have onboarding ahead, you want to start to put some processes in place right away like Rubocop, Yard Doc, clean test suite, speed up deliveries and so on.

Rubocop is a great tool but it doesn’t apply well in an existing project is going to hurt badly your git history which you really need in order to understand and find why some pieces of code were introduced for.

Adding code documentation in an existing project it’s painful and you don’t have time for that anyway.

So let’s start with a new fresh project! Shall we?

To make that article truly useful for anyone that wants to apply it in his/her project let follow a concrete example.

Let’s say we have an App that let you comment part of the reports. In purpose, we have written a old fashion apps which have on top of the hierarchy Organization, which have projects and those projects have a bunch of reports attached, which you can add notes on them.

{{domain}}/organizations/{{organization_id}}/projects/{{project_id}}/reports/{{report_id}}/notes

Representation could be something like Tesla > Model 3 > 2019 Q4 Shanghai Report to have an idea in mind.

Besides the old naive and rigid approach of this CRUD REST API you conclude project shouldn’t be mandatory or let say this is a business requirement, clients complain that they would be able to add reports directly to the organizations because it makes the whole workflow easier. They always can sort them out later on, or never. They may want to put the same report in different project sections and so one… This is the first feature you are asked to do.

Ahead you think that project and note models are used like filter/tag and can easily be rethink as a polymorphic model, more or that later, but for now you are not interesting to touch them.

So we are going to take down user, organization and report models into the Rails Engine and let project and note models into the Host Rails App for further removing. That way we will explore how to make that mix play well together.

You think, good timing to refactor a bit that code and behaviour, you want to introduce JSON API and be able to ask reports like that

{{domain}}/v1/organizations/{{organization_id}}/reports?include=project&filter[project_name_in]=Annual-Report-2017,Annual-Report-2018&fields[project]=name

It’s definitely sexier, isn’t it? And add a lot of flexibility to the API.

The host app

This part it’s optional. You can skip this section if you are not interested to follow step by step. The whole code is here, you can go through the commits which will show you the detailed steps I took, and/or following the README.

Ruby Version 2.6.5

I skip everything I don’t need.

rails _6.0.1_ new annotable_app --api \
  --database=postgresql \
  --skip-action-mailer \
  --skip-action-mailbox \
  --skip-action-text \
  --skip-puma \
  --skip-action-cable \
  --skip-sprockets \
  --skip-javascript \
  --skip-turbolinks \
  --skip-test \
  --skip-webpack-install

As well I like to load the only dependencies I’m going to use instead of loading the entirely rails stack.

# Gemfile
# gem 'rails', '~> 6.0.1'
gem 'actionpack', '~> 6.0.1'
gem 'activemodel', '~> 6.0.1'
gem 'activerecord', '~> 6.0.1'
gem 'activesupport', '~> 6.0.1'
gem 'railties', '~> 6.0.1'

You can revise your middleware as well and remove what you are not using, either is always good to repassing the middleware stack just to have it in mind.

bin/rails middleware

And remove all you are not going to use

# config/application.rb
config.middleware.delete ActionDispatch::Cookies
config.middleware.delete Rack::Sendfile
config.middleware.delete ActionDispatch::Static
config.middleware.delete ActiveSupport::Cache::Strategy::LocalCache::Middleware
config.middleware.delete ActionDispatch::ActionableExceptions
config.middleware.delete ActionDispatch::Callbacks

Zeitwerk offer to log its activity, as we are going to Improving engine functionality by Overriding Models and Controllers in the Host App it would be a good idea to have an eye on what autoload does, we are going to see why later on.

# config/initializers/autoloader.rb
Rails.autoloaders.logger = Rails.logger

As we are going to generate some resources, do you a big favour and override all the templates you need, the Rails ones, RSpec ones if you plan to use RSpec.

Is not totally obvious, so if you have some difficulties don’t hesitate to have a look to my answer regarding the subject in stackoverflow.com.

Or create your own generator if it’s needed

bin/rails generate generator — help

NOTE: If you are using RSpec the generator is broken, have a look at my PR https://github.com/rspec/rspec-rails/pull/2217

Once your app is setup correctly you can quickly create it with the following commands:

bin/rails generate scaffold organization name:string
bin/rails g scaffold user name:string email:string organization:references
bin/rails g scaffold project name:string organization:references
bin/rails g scaffold report name:string content:text project:references
bin/rails g scaffold note title:string content:text report:references

The Rails Engine

This part it’s optional too. You can skip this section if you are not interested to follow step by step. The whole code is here, you can go through the commits which will show you the detailed steps I took, and/or following the README.

First of all, we need to find a suitable name. If the name of a Rails App is not really a matter as it influences almost nothing it is another story when it comes to the Rails Engine name. This name is essential because it will be present all over the code. So keep it short and meaningful. For our example, we are going to choose Annotable because the App treats to annotate reports.

We are only interested in Rails API-Only here. So I will remove the unnecessary dependencies.

rails _6.0.1_ plugin new annotable --mountable --api \
  --database=postgresql \
  --skip-action-mailer \
  --skip-action-mailbox \
  --skip-action-text \
  --skip-active-storage \
  --skip-puma \
  --skip-action-cable \
  --skip-sprockets \
  --skip-javascript \
  --skip-turbolinks \
  --skip-test \
  --dummy-path=spec/dummy

in the annotable.gemspec we can change accordingly

# annotable.gemspec
# spec.add_dependency "rails", "~> 6.0.1"
# Load only what you need
spec.add_dependency "actionpack", "~> 6.0.1"
spec.add_dependency "activemodel", "~> 6.0.1"
spec.add_dependency "activerecord", "~> 6.0.1"
spec.add_dependency "activesupport", "~> 6.0.1"
spec.add_dependency "activejob", "~> 6.0.1"
spec.add_dependency "railties", "~> 6.0.1"

If, like me, you prefer Rspec over Testunit, do install it but keep in mind that rspec-rails is not going change the generator configuration in a Rails Engine like it does in an Rails App, you have to explicitly define them.

# lib/annotable/engine.rb
config.generators do |g|
  g.test_framework :rspec, fixture: true
  g.fixture_replacement :fabrication
  g.api_only = true
  g.orm :active_record, primary_key_type: :uuid
  g.templates << File.expand_path('../templates', __dir__)
end

Here we define all the generator we need. Note we need to explicitly add the templates directory for overriding them, a Rails App look into this directory implicitly but not in the Rails Engine.

Just to see how far we can mix everything between the Rails Engine and the Rails App I choose another Fixture Generator we will see how we articulate that in the host app later on.

As the plan is to rewrite entirely the App we don’t want to stick with the classic ID as Integer sequence but using UUID instead, adding Rubocop and YardStick a the very beginning make everything easier, I tried once on the existing project is definitely note doable.

For Rails JSON API there are some different implementations out there, I will use this one jsonapi.rb which is simple and quite powerful thanks to Ransack.

{{domain}}/annotable/organizations/{{organization_id}}/reports?filter[name_cont]=Annual-Report

Go through the commits details to see how everything is configured, but with the templates in place, this reduces more or less to call generators for the three models we need. Organization, User, Report.

Mount the Rails Engine

Now we reach the interesting part, how to plug our Rails Engine into the Rails app and make everything work smoothly.

So go back to our Rails App.

We plug the Rails Engine, it’s quite simple and straightforward

Add the gem to your Gemfile

# Gemfile
gem ‘annotable’, path: '../annotable' # local development

We mount the API as V1 to separate the new behavior from the old one.

# config/routes.rb
Rails.application.routes.draw do
  resources :users
  resources :organizations do
    resources :projects do
      resources :reports do
        resources :notes
      end
    end
  end
  mount Annotable::Engine, at: 'v1'
end

We copy the Rails Engine migrations into our Rails App

bin/rails railties:install:migrations

At that stage we can run the server and see the enpoint response correclty

curl -X GET \
  'http://localhost:3000/v1/users
{
    "links": {
        "self": "http://localhost:3000/v1/users?include=organization",
        "current": "http://localhost:3000/v1/users?include=organization&page[number]=1"
    },
    "data": []
}

Okay, now we want to see our data coming through, without breaking the legacy API

There are several techniques to use the Rails Engine code into your App, I recommend you to check out the chapter Improving engine functionality of the Rails Engine guide but for our purpose, we are going to use the technique of reopening the class because our goal is to replace our Rails App not enriched the functionalities. We really want to switch completely.

We want to use the class Annotable::Organization instead of Organization.

But remember, the model Annotable::Organization doesn’t have any knowledge about project and doesn't have projects association, so we are going to add it.

# app/models/organization.rb
Annotable::Organization.class_eval do
  has_many :projects, class_name: 'Project', primary_key: :legacy_id
end
Organization = Annotable::Organization

As you’ve have surely already spotted we declare another primary_key as a reference, indeed we primary want to use UUID in our system, however, the legacy app talks with normal ID. We want to let every model of our Rails App continue to talk with classic ID but let the new model using UUID instead.

in order to do so, we add a new field to the model Annotable::Organization

add_column :annotable_organizations, :legacy_id, :bigint

we keep track of the id sequence value at it is.

organizations_id_seq_value = select_value("SELECT NEXTVAL('organizations_id_seq')")

We create a sequence for our new legacy_id attribute.

sql = <<-SQL.squish
  CREATE SEQUENCE public.annotable_organizations_legacy_id_seq
   START WITH #{organizations_id_seq_value}
   INCREMENT BY 1
   NO MINVALUE
   NO MAXVALUE
   CACHE 1
SQL
execute(sql)

We need to change the foreign key constraint

sql = <<-SQL.squish
  ALTER TABLE annotable_organizations ADD CONSTRAINT annotable_organizations_legacy_id_uniq UNIQUE (legacy_id)
SQL
execute(sql)
remove_foreign_key :projects, :organizations

See commits or code for all details

We tell the model project how to find the new model Annotable::Organization

class Project < ApplicationRecord
  belongs_to(
    :organization,
    class_name: 'Annotable::Organization',
    primary_key: :legacy_id,
    required: true
  )
end

Now the legacy project and the new organization provided by our Engine works well.

Before being able to run our application we need to enrich the code of the engine to make it aware of project.

# config/initializers/annotable.rb
require 'annotable'
require Annotable::Engine.root
  .join('app/controllers/annotable/organizations_controller.rb')
module OrganizationControllerCustomFields
  module ClassMethods
    def allowed_includes
      (super.dup + [:projects]).freeze
    end
    def allowed_filterables
      (super.dup + ['projects_name']).freeze
    end
  end
  def self.prepended(base)
    class << base
      prepend(ClassMethods)
    end
  end
end
Annotable::OrganizationsController
  .prepend(OrganizationControllerCustomFields)

remember project doesn’t exist for our Engine, so now we are able to do this kind of request through the V1 API

curl -X GET \
  'http://localhost:3000/v1/organizations?include=users,projects&filter[name_eq]=Big Corp&fields[organization]=name&fields[user]=email&fields[project]=name

There is one last trick, in development mode autoload is going to reload all in the app/ directory as long as your Engine, but nothing into config/ so to prevent in dev mode to have the following

NoMethodError (undefined method `projects' for #<Annotable::Organization:0x00007fe78008b078>)

we need to tell autoload not to reload neither organization controller nor the organization model.

# config/initializers/zeitwerk.rb
require 'annotable'
require Annotable::Engine.root
 .join('app/models/annotable/organization.rb')
require Annotable::Engine.root
 .join('app/serializers/annotable/organization_serializer.rb')
require Annotable::Engine.root
 .join('app/controllers/annotable/organizations_controller.rb')
Rails.autoloaders.main.ignore(
 Annotable::Engine.root.join('app/models/annotable/organization.rb')
)
Rails.autoloaders.main.ignore(
 Annotable::Engine.root
  .join('app/serializers/annotable/organization_serializer.rb')
)
Rails.autoloaders.main.ignore(
 Annotable::Engine.root
  .join('app/controllers/annotable/organizations_controller.rb')
)
require_dependency 
  Rails.root.join('app/models/organization.rb')
require_dependency
  Rails.root.join('config/initializers/annotable.rb')

Now we can access both through the legacy API and through the new V1 API.

The next step would be to get rid of the project and note for a polymorphic tag model because it’s exactly how those model are used here. Once done, and the front end team catches up the new API, you will be able to remove all the old code and have only the new codebase working, and you are done!

Summary

In summary, Rails Engine can let you get progressively rid of the legacy code, make your daily basis a lot of better. Increasing the code quality, add the new features asked and rewriting your app in a safe and controlled environment. Speed up your development process and deliveries. You can introduce new processes along the way and make new onboarding easier. You can enhance the API behaviour and let the legacy code still running at it was.

Conclusion

It’s an interesting way to use Rails Engine, however, I would recommend having clear in mind the schedule of the entire rewrite, I strongly encourage not to stay in that mixed state forever, as certainly new edge cases will popup. But it would be a great way to deal with the wrong codebase in a safer way.

I really hope this article may help people to deal with bad design and technical debt. I wish I wouldn’t have to write this kind of article but if you have a bit of experience in Rails world you are really likely to have been exposed to that total kind of a mess. You might be recently hired, the company looks great, what they are doing makes you feel you are in the right place but once you discover the codebase you are about quitting right away, and you might just do.