The definitive guide to tackle technical deb in the Rails project
How to survive to a legacy monolithic Ruby on Rails application with the help of Rails Engines. As a disclosure, this article is not about extracting modules from the host app to a Rails Engine but using the power of a Rails Engine to let you rethink and rewrite a codebase which needs to be.
If you have ever be confronted to a Rails project it scares you because the codebase is so messy you are not confident to touch anything at all, even understand what is going on in some part of the code because 80% of the code is about fixing edge cases and the flaw of the initial design. Your conclusion reaches quickly the fact everything is just worth to throw away but as surprising as it is it works and it is out of the question to start a re-write from scratch, because, well, it works. Too often executives don’t understand why a codebase, which works, need to be rewritten and why they need to put time and money on that, but this article is not about management.
Let’s pretend the startup was just funded and start hiring, you have onboarding ahead, you want to start to put some processes in place right away like Rubocop, Yard Doc, clean test suite, speed up deliveries and so on.
Rubocop is a great tool but it doesn’t apply well in an existing project is going to hurt badly your git history which you really need in order to understand and find why some pieces of code were introduced for.
Adding code documentation in an existing project it’s painful and you don’t have time for that anyway.
So let’s start with a new fresh project! Shall we?
To make that article truly useful for anyone that wants to apply it in his/her project let follow a concrete example.
Let’s say we have an App that let you comment part of the reports. In purpose, we have written a old fashion apps which have on top of the hierarchy Organization, which have projects and those projects have a bunch of reports attached, which you can add notes on them.
{{domain}}/organizations/{{organization_id}}/projects/{{project_id}}/reports/{{report_id}}/notesRepresentation could be something like Tesla > Model 3 > 2019 Q4 Shanghai Report to have an idea in mind.
Besides the old naive and rigid approach of this CRUD REST API you conclude project shouldn’t be mandatory or let say this is a business requirement, clients complain that they would be able to add reports directly to the organizations because it makes the whole workflow easier. They always can sort them out later on, or never. They may want to put the same report in different project sections and so one… This is the first feature you are asked to do.
Ahead you think that project and note models are used like filter/tag and can easily be rethink as a polymorphic model, more or that later, but for now you are not interesting to touch them.
So we are going to take down user, organization and report models into the Rails Engine and let project and note models into the Host Rails App for further removing. That way we will explore how to make that mix play well together.
You think, good timing to refactor a bit that code and behaviour, you want to introduce JSON API and be able to ask reports like that
{{domain}}/v1/organizations/{{organization_id}}/reports?include=project&filter[project_name_in]=Annual-Report-2017,Annual-Report-2018&fields[project]=nameIt’s definitely sexier, isn’t it? And add a lot of flexibility to the API.
The host app
This part it’s optional. You can skip this section if you are not interested to follow step by step. The whole code is here, you can go through the commits which will show you the detailed steps I took, and/or following the README.
Ruby Version 2.6.5
I skip everything I don’t need.
rails _6.0.1_ new annotable_app --api \
--database=postgresql \
--skip-action-mailer \
--skip-action-mailbox \
--skip-action-text \
--skip-puma \
--skip-action-cable \
--skip-sprockets \
--skip-javascript \
--skip-turbolinks \
--skip-test \
--skip-webpack-installAs well I like to load the only dependencies I’m going to use instead of loading the entirely rails stack.
# Gemfile# gem 'rails', '~> 6.0.1'gem 'actionpack', '~> 6.0.1'
gem 'activemodel', '~> 6.0.1'
gem 'activerecord', '~> 6.0.1'
gem 'activesupport', '~> 6.0.1'
gem 'railties', '~> 6.0.1'
You can revise your middleware as well and remove what you are not using, either is always good to repassing the middleware stack just to have it in mind.
bin/rails middlewareAnd remove all you are not going to use
# config/application.rbconfig.middleware.delete ActionDispatch::Cookies
config.middleware.delete Rack::Sendfile
config.middleware.delete ActionDispatch::Static
config.middleware.delete ActiveSupport::Cache::Strategy::LocalCache::Middleware
config.middleware.delete ActionDispatch::ActionableExceptions
config.middleware.delete ActionDispatch::Callbacks
Zeitwerk offer to log its activity, as we are going to Improving engine functionality by Overriding Models and Controllers in the Host App it would be a good idea to have an eye on what autoload does, we are going to see why later on.
# config/initializers/autoloader.rb
Rails.autoloaders.logger = Rails.loggerAs we are going to generate some resources, do you a big favour and override all the templates you need, the Rails ones, RSpec ones if you plan to use RSpec.
Is not totally obvious, so if you have some difficulties don’t hesitate to have a look to my answer regarding the subject in stackoverflow.com.
Or create your own generator if it’s needed
bin/rails generate generator — helpNOTE: If you are using RSpec the generator is broken, have a look at my PR https://github.com/rspec/rspec-rails/pull/2217
Once your app is setup correctly you can quickly create it with the following commands:
bin/rails generate scaffold organization name:string
bin/rails g scaffold user name:string email:string organization:references
bin/rails g scaffold project name:string organization:references
bin/rails g scaffold report name:string content:text project:references
bin/rails g scaffold note title:string content:text report:referencesThe Rails Engine
This part it’s optional too. You can skip this section if you are not interested to follow step by step. The whole code is here, you can go through the commits which will show you the detailed steps I took, and/or following the README.
First of all, we need to find a suitable name. If the name of a Rails App is not really a matter as it influences almost nothing it is another story when it comes to the Rails Engine name. This name is essential because it will be present all over the code. So keep it short and meaningful. For our example, we are going to choose Annotable because the App treats to annotate reports.
We are only interested in Rails API-Only here. So I will remove the unnecessary dependencies.
rails _6.0.1_ plugin new annotable --mountable --api \
--database=postgresql \
--skip-action-mailer \
--skip-action-mailbox \
--skip-action-text \
--skip-active-storage \
--skip-puma \
--skip-action-cable \
--skip-sprockets \
--skip-javascript \
--skip-turbolinks \
--skip-test \
--dummy-path=spec/dummyin the annotable.gemspec we can change accordingly
# annotable.gemspec# spec.add_dependency "rails", "~> 6.0.1"# Load only what you need
spec.add_dependency "actionpack", "~> 6.0.1"
spec.add_dependency "activemodel", "~> 6.0.1"
spec.add_dependency "activerecord", "~> 6.0.1"
spec.add_dependency "activesupport", "~> 6.0.1"
spec.add_dependency "activejob", "~> 6.0.1"
spec.add_dependency "railties", "~> 6.0.1"
If, like me, you prefer Rspec over Testunit, do install it but keep in mind that rspec-rails is not going change the generator configuration in a Rails Engine like it does in an Rails App, you have to explicitly define them.
# lib/annotable/engine.rbconfig.generators do |g|
g.test_framework :rspec, fixture: true
g.fixture_replacement :fabrication
g.api_only = true
g.orm :active_record, primary_key_type: :uuid
g.templates << File.expand_path('../templates', __dir__)
end
Here we define all the generator we need. Note we need to explicitly add the templates directory for overriding them, a Rails App look into this directory implicitly but not in the Rails Engine.
Just to see how far we can mix everything between the Rails Engine and the Rails App I choose another Fixture Generator we will see how we articulate that in the host app later on.
As the plan is to rewrite entirely the App we don’t want to stick with the classic ID as Integer sequence but using UUID instead, adding Rubocop and YardStick a the very beginning make everything easier, I tried once on the existing project is definitely note doable.
For Rails JSON API there are some different implementations out there, I will use this one jsonapi.rb which is simple and quite powerful thanks to Ransack.
{{domain}}/annotable/organizations/{{organization_id}}/reports?filter[name_cont]=Annual-ReportGo through the commits details to see how everything is configured, but with the templates in place, this reduces more or less to call generators for the three models we need. Organization, User, Report.
Mount the Rails Engine
Now we reach the interesting part, how to plug our Rails Engine into the Rails app and make everything work smoothly.
So go back to our Rails App.
We plug the Rails Engine, it’s quite simple and straightforward
Add the gem to your Gemfile
# Gemfile
gem ‘annotable’, path: '../annotable' # local developmentWe mount the API as V1 to separate the new behavior from the old one.
# config/routes.rbRails.application.routes.draw do
resources :users
resources :organizations do
resources :projects do
resources :reports do
resources :notes
end
end
end mount Annotable::Engine, at: 'v1'
end
We copy the Rails Engine migrations into our Rails App
bin/rails railties:install:migrationsAt that stage we can run the server and see the enpoint response correclty
curl -X GET \
'http://localhost:3000/v1/users{
"links": {
"self": "http://localhost:3000/v1/users?include=organization",
"current": "http://localhost:3000/v1/users?include=organization&page[number]=1"
},
"data": []
}
Okay, now we want to see our data coming through, without breaking the legacy API
There are several techniques to use the Rails Engine code into your App, I recommend you to check out the chapter Improving engine functionality of the Rails Engine guide but for our purpose, we are going to use the technique of reopening the class because our goal is to replace our Rails App not enriched the functionalities. We really want to switch completely.
We want to use the class Annotable::Organization instead of Organization.
But remember, the model Annotable::Organization doesn’t have any knowledge about project and doesn't have projects association, so we are going to add it.
# app/models/organization.rbAnnotable::Organization.class_eval do
has_many :projects, class_name: 'Project', primary_key: :legacy_id
end
Organization = Annotable::Organization
As you’ve have surely already spotted we declare another primary_key as a reference, indeed we primary want to use UUID in our system, however, the legacy app talks with normal ID. We want to let every model of our Rails App continue to talk with classic ID but let the new model using UUID instead.
in order to do so, we add a new field to the model Annotable::Organization
add_column :annotable_organizations, :legacy_id, :bigintwe keep track of the id sequence value at it is.
organizations_id_seq_value = select_value("SELECT NEXTVAL('organizations_id_seq')")We create a sequence for our new legacy_id attribute.
sql = <<-SQL.squish
CREATE SEQUENCE public.annotable_organizations_legacy_id_seq
START WITH #{organizations_id_seq_value}
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1
SQL
execute(sql)We need to change the foreign key constraint
sql = <<-SQL.squish
ALTER TABLE annotable_organizations ADD CONSTRAINT annotable_organizations_legacy_id_uniq UNIQUE (legacy_id)
SQL
execute(sql)remove_foreign_key :projects, :organizations
See commits or code for all details
We tell the model project how to find the new model Annotable::Organization
class Project < ApplicationRecord
belongs_to(
:organization,
class_name: 'Annotable::Organization',
primary_key: :legacy_id,
required: true
)
endNow the legacy project and the new organization provided by our Engine works well.
Before being able to run our application we need to enrich the code of the engine to make it aware of project.
# config/initializers/annotable.rbrequire 'annotable'require Annotable::Engine.root
.join('app/controllers/annotable/organizations_controller.rb')module OrganizationControllerCustomFields
module ClassMethods
def allowed_includes
(super.dup + [:projects]).freeze
end
def allowed_filterables
(super.dup + ['projects_name']).freeze
end
end def self.prepended(base)
class << base
prepend(ClassMethods)
end
end
end
Annotable::OrganizationsController
.prepend(OrganizationControllerCustomFields)
remember project doesn’t exist for our Engine, so now we are able to do this kind of request through the V1 API
curl -X GET \
'http://localhost:3000/v1/organizations?include=users,projects&filter[name_eq]=Big Corp&fields[organization]=name&fields[user]=email&fields[project]=nameThere is one last trick, in development mode autoload is going to reload all in the app/ directory as long as your Engine, but nothing into config/ so to prevent in dev mode to have the following
NoMethodError (undefined method `projects' for #<Annotable::Organization:0x00007fe78008b078>)we need to tell autoload not to reload neither organization controller nor the organization model.
# config/initializers/zeitwerk.rbrequire 'annotable'require Annotable::Engine.root
.join('app/models/annotable/organization.rb')
require Annotable::Engine.root
.join('app/serializers/annotable/organization_serializer.rb')
require Annotable::Engine.root
.join('app/controllers/annotable/organizations_controller.rb')Rails.autoloaders.main.ignore(
Annotable::Engine.root.join('app/models/annotable/organization.rb')
)
Rails.autoloaders.main.ignore(
Annotable::Engine.root
.join('app/serializers/annotable/organization_serializer.rb')
)
Rails.autoloaders.main.ignore(
Annotable::Engine.root
.join('app/controllers/annotable/organizations_controller.rb')
)require_dependency
Rails.root.join('app/models/organization.rb')require_dependency
Rails.root.join('config/initializers/annotable.rb')
Now we can access both through the legacy API and through the new V1 API.
The next step would be to get rid of the project and note for a polymorphic tag model because it’s exactly how those model are used here. Once done, and the front end team catches up the new API, you will be able to remove all the old code and have only the new codebase working, and you are done!
Summary
In summary, Rails Engine can let you get progressively rid of the legacy code, make your daily basis a lot of better. Increasing the code quality, add the new features asked and rewriting your app in a safe and controlled environment. Speed up your development process and deliveries. You can introduce new processes along the way and make new onboarding easier. You can enhance the API behaviour and let the legacy code still running at it was.
Conclusion
It’s an interesting way to use Rails Engine, however, I would recommend having clear in mind the schedule of the entire rewrite, I strongly encourage not to stay in that mixed state forever, as certainly new edge cases will popup. But it would be a great way to deal with the wrong codebase in a safer way.
I really hope this article may help people to deal with bad design and technical debt. I wish I wouldn’t have to write this kind of article but if you have a bit of experience in Rails world you are really likely to have been exposed to that total kind of a mess. You might be recently hired, the company looks great, what they are doing makes you feel you are in the right place but once you discover the codebase you are about quitting right away, and you might just do.
