README.md

Outboxer

Background

Outboxer helps you migrate Ruby on Rails applications to best practice event driven architecture fast.

It was created out of a need to addresses the dual write problem that can occur when attempting to insert an Event row into an SQL database and then queuing an EventCreatedJob into redis to handle that event e.g.

user_created_event = UserCreatedEvent.create!(...)

# ☠️ process crashes

UserCreatedJob.perform_async(user_created_event.id)

❌ job was not queued to redis

🪲 downstream state is now inconsistent

Usage

1. Queue an outboxer message after an event is created

# app/event.rb

class Event < ApplicationRecord
  after_create do |event|
    Outboxer::Message.queue(messageable: event)
  end
end

Note: This ensures an event is always created in the same transaction as the queued outboxer message referencing it.

2. Define a new event

# app/models/user_created_event.rb

class UserCreatedEvent < Event; end

3. Define a new job to handle the new event

# app/jobs/user_created_job.rb

class UserCreatedJob
  include Sidekiq::Job

  def perform(event_id, event_type)
    # your code to handle UserCreatedEvent here
  end
end

4. Route the new event to the new job

# app/jobs/event_created_job.rb

class EventCreatedJob
  include Sidekiq::Job

  def perform(event_id, event_type)
    job_class_name = event_type.sub(/Event\z/, "Job")
    job_class = job_class_name.safe_constantize
    job_class.perform_async(event_id, event_type)
  end
end

5. Queue the event created job in the outboxer_publisher script block

# bin/publisher

Outboxer::Publisher.publish_message(...) do |message|
  EventCreatedJob.perform_async(
    message[:messageable_id], message[:messageable_type])
end

6. Run sidekiq

7. Run bin/outboxer_publisher

8. Create new event in console

UserCreatedEvent.create!

9. Observe logs

For more details, see the wiki page: How Outboxer works

Installation

1. add gem to gemfile

gem 'outboxer'

2. install gem

bundle install

3. generate schema, publisher and tests

bin/rails g outboxer:install

Usage

1. review generated event schema and model

Event schema

# db/migrate/create_events.rb

class CreateEvents < ActiveRecord::Migration[7.0]
  def up
    create_table :events do |t|
      t.bigint :user_id
      t.bigint :tenant_id

      t.string :eventable_type, limit: 255
      t.bigint :eventable_id
      t.index [:eventable_type, :eventable_id]

      t.string :type, null: false, limit: 255
      t.send(json_column_type, :body)
      t.datetime :created_at, null: false
    end
  end

  # ...
end

Event model

# app/models/event.rb

class Event < ApplicationRecord
  self.table_name = "events"

  # associations

  belongs_to :eventable, polymorphic: true

  # validations

  validates :type, presence: true, length: { maximum: 255 }

  # callbacks

  after_create do |event|
    Outboxer::Message.queue(messageable: event)
  end
end

2. migrate schema

bin/rake db:migrate

3. define new event using STI

# app/models/accountify/contact_created_event.rb

module Accountify
  class ContactCreatedEvent < Event
  end
end

4. create new event in application service

# app/services/accountify/contact_service.rb

module Accountify
  module ContactService
    module_function

    def create(user_id:, tenant_id:, email:)
      ActiveRecord::Transaction.execute do
        contact = Contact.create!(tenant_id: tenant_id, email: email)

        event = ContactCreatedEvent.create!(
          user_id: user_id, tenant_id: tenant_id,
          eventable: contact, body: { "email" => email }
        )

        [contact, event]
      end
    end
  end
end

contact, event = Accountiy::ContactService.create(...)

5. add event created job to route event

Following the convention Context::ResourceVerbEvent -> Context::ResourceVerbJob

# app/jobs/accountify/contact_created_job.rb

module Accountify
  class ContactCreatedJob
    include Sidekiq::Job

    def perform_async(args)
      # your handler code here
    end
  end
end

Note: this can be customised in the generated EventCreatedJob

6. run publisher

bin/outboxer_publisher

Note: The outboxer publisher supports many options.

7. run sidekiq

bin/sidekiq

Note: Enabling superfetch is strongly recommend, to preserve consistency across services.

8. open rails console

bin/rails c

9. call service

contact, event = Accountify::ContactService.create(user_id: 1, tenant_id: 1, email: '[email protected]')

10. observe transactional consistency

TRANSACTION (0.5ms)  BEGIN
Accountify::Contact Create             (1.9ms)  INSERT INTO "accountify_contacts" ...
Accountify::ContactCreatedEvent Create (1.8ms)  INSERT INTO "events" ...
Outboxer::Message Create               (3.2ms)  INSERT INTO "outboxer_messages" ...
TRANSACTION (0.7ms)  COMMIT
=> {:id=>1, :events=>[{:id=>1, :type=>"Accountify::ContactCreatedEvent"}]}

Testing

1. start test

OutboxerIntegration::TestService.start

TRANSACTION (0.5ms)  BEGIN
OutboxerIntegration::Test Create             (1.9ms)  INSERT INTO "outboxer_integration_tests" ...
OutboxerIntegration::TestStartedEvent Create (1.8ms)  INSERT INTO "events" ...
Outboxer::Message Create                     (3.2ms)  INSERT INTO "outboxer_messages" ...
TRANSACTION (0.7ms)  COMMIT
=> {:id=>1, :events=>[{:id=>1, :type=>"OutboxerIntegration::TestStartedEvent"}]}

2. ensure test completes

OutboxerIntegration::Test.find(1).events
=>
[#<OutboxerIntegration::TestStartedEvent:0x0000000105749158
  id: 1,
  user_id: nil,
  tenant_id: nil,
  eventable_type: "OutboxerIntegration::Test",
  eventable_id: 1,
  type: "OutboxerIntegration::TestStartedEvent",
  body: {"test"=>{"id"=>1}},
  created_at: 2025-01-11 23:37:36.009745 UTC>,
 #<OutboxerIntegration::TestCompletedEvent:0x0000000105748be0
  id: 2,
  user_id: nil,
  tenant_id: nil,
  eventable_type: "OutboxerIntegration::Test",
  eventable_id: 1,
  type: "OutboxerIntegration::TestCompletedEvent",
  body: {"test"=>{"id"=>1}},
  created_at: 2025-01-11 23:48:38.750419 UTC>]

3. run spec

bin/rspec spec/outboxer_integration/test_started_spec.rb

This is what the generated spec is testing, to continue to ensure you have end to end confidence in your stack.

Management

Outboxer provides a sidekiq like UI to help manage your messages

Publishers

Messages

rails

config/routes.rb

require 'outboxer/web'

Rails.application.routes.draw do
  mount Outboxer::Web, at: '/outboxer'
end

rack

config.ru

require 'outboxer/web'

map '/outboxer' do
  run Outboxer::Web
end

Contributing

If you'd like to help out, take a look at the project's open issues.

Also join our discord and ping bedrock-adam if you'd like to contribute to upcoming features.

License

This gem is available as open source under the terms of the GNU Lesser General Public License v3.0.