Skip to content

Upgrading from Sprockets to jsbundling‐rails

Jump to bottom
Janell Huyck edited this page May 20, 2024 · 27 revisions

Updating to jsbundling-rails

Background

The asset pipeline continues to evolve. There's a somewhat-helpful blog post here that talks about it. It looks like the new Rails 7 does allow for the use of Sprockets, but Sprockets has trouble handling modern JavaScript functionality, including the ES-6 format.

We needed to update how Javascript is handled on Treatment Database to add modern functionality like Stimulus and adding a WYSIWYG text editor. Initially, I had planned to update just to Webpacker from Sprockets, but Webpacker has a security issue that is fixed by upgrading from Webpacker to webpack with jsbundling-rails. Therefore, I decided it would be better to skip the migration to Webpacker and go straight to jsbundling-rails from Sprockets.

Quick note: "Webpack" is NOT the same thing as "Webpacker". It's really easy to get these two confused, and both ChatGPT and Google Searches will often assume you meant "Webpacker" when you type "Webpack" to look things up. Be on the lookout for this so you don't end up wandering down the wrong path. We are only using Webpack.

jsbundling-rails uses Webpack behind the scenes to handle files. The Webpack documentation is surprisingly readable and can be found at https://webpack.js.org/concepts/.

This wiki page will talk about what was involved in the migration and how the new jsbundling-rails set-up is laid out, what the different components are, and why I made the choices I did. The following sections are meant to be reviewed in order, but you may need to jump around and refer back to them if you're attempting to upgrade a different app.

The following is based off what I did in the following PR: https://github.com/uclibs/treatment_database/pull/497

Updating Node

Updating Node

Our server was set to use Node version 12.22.12. As of this writing, the LTS version of Node is 20, with version 22 released 3 weeks ago. Version 22 is intended to be the next LTS version. Version 12.22.12 does not support several dependencies that I needed to install in upgrading the app. I needed to use version 18.

We talked about this problem and decided to change our server from serving one specific Node version to using NVM with a default and allowing individual apps to use a .nvmrc file to specify what version of Node is needed for the app.

Using NVM and the .nvmrc file

If you wanted to change what version of Node you were using on your local machine to version 18.20.2, you would run nvm use 18.20.2. With the addition of the .nvmrc file, you now run nvm use. NVM will look for a .nvmrc file in the directory you're in and will apply the version written in the file.

The new .nvmrc file

A .nvmrc file is very simple - you simply put in what version of node you want to use, and nothing else. Our app uses Node version 18.20.2, so this is our .nvmrc file:

18.20.2
Installing Webpack and JSBundling Rails

Installing Webpack and JSBundling Rails

The next step in this migration is to install and configure Webpack since jsbundling-rails utilizes Webpack under the hood to manage JavaScript bundling.

Steps to Install

  • If you have the gem gem 'sprockets-rails' in your Gemfile, remove it.
  • Add gem 'jsbundling-rails' to your Gemfile and run bundle install.
  • Run yarn add webpack webpack-cli to add the Webpack and webpack-cli dependencies to your package.json file.
  • Run rails javascript:install:webpack to install and configure Webpack.
Updating the package.json file

Updating the package.json File

The package.json file is a manifest for your JavaScript dependencies and project metadata, similar to how a Gemfile works for Ruby gems, specifying which packages to install and how to configure scripts and tools for a project.

It needs to have all the dependencies that will handle loading and running the front end of the app. Here's how ours ended up, after including all the dependencies for handling JavaScript, css, and images:

{
  "name": "treatment_database",
  "private": true,
  "dependencies": {
    "@rails/activestorage": "^7.1.3-2",
    "@rails/ujs": "^7.1.3-2",
    "bootstrap": "^4.3.1",
    "jquery": "^3.5.1",
    "mini-css-extract-plugin": "^2.9.0",
    "popper.js": "^1.16.1",
    "turbolinks": "^5.2.0",
    "webpack": "^5.91.0",
    "webpack-cli": "^5.1.4"
  },
  "scripts": {
    "build": "webpack --config webpack.config.js"
  },
  "devDependencies": {
    "@babel/core": "^7.24.5",
    "@babel/preset-env": "^7.24.5",
    "babel-loader": "^9.1.3",
    "css-loader": "^7.1.1",
    "file-loader": "^6.2.0",
    "sass": "^1.77.0",
    "sass-loader": "^14.2.1",
    "style-loader": "^4.0.0",
    "webpack-manifest-plugin": "^5.0.0"
  }
}

For any missing dependency that you want to add, run yarn add package-name@version and that will add it to the "dependencies" section. If you want to add a devDependencies, which is like the section in the Gemfile for development gems, run yarn add package-name@version --dev.

It's fine to copy/paste these dependencies into your package.json file. If you do this, you will need to run yarn install after changing the package.json file. That's similar to manually adding a gem to your Gemfile and then running bundle install.

When a dependency is added, either by yarn add or by adding manually to package.json and then running yarn install, it will create or update a yarn.lock file. You don't edit anything in the yarn.lock file. This is the equivalent of the Gemfile.lock file, but for your front-end (yarn-managed) dependencies.

If you think you've messed something up or if things aren't running right, it's fine to delete the yarn.lock file and run yarn install again to get a fresh installation of the dependencies.

Note:

In the sample package.json above, take a look at the section:

  "scripts": {
    "build": "webpack --config webpack.config.js"
  }

This sets up the yarn build command for you to run from the terminal. It says that when you run yarn build it should use webpack, and that the config file for webpack is "webpack.config.js".

Creating a Webpack Config File

Creating a Webpack Config File

Think of Webpack like a pipeline, where your JavaScript, CSS, and image files enter in their raw, development-friendly format. Through this pipeline, they are transformed, optimized, and output as fewer, optimized files that are easier and faster for browsers to load. The configuration file webpack.config.js simply tells Webpack how to conduct this process in terms of:

  • Where to start and end the process.
  • How to handle different file types.
  • How to name the output files for efficient caching.
  • How to keep track of everything via a manifest.

Webpack assumes everything is in production mode. It has a development mode that could be slightly faster for development, but I found with our small Treatment Database that trying to manage both a production mode and a development mode added needless complexity, and running Webpack in production mode all the time worked fine.

It could be different with a larger codebase. There, you might see noticeable performance improvements for development mode.

The webpack.config.js file looks intimidating to someone not familiar with that format. Let's take a look at it. Here's how ours ended up:

const path = require('path');
const MiniCssExtractPlugin = require('mini-css-extract-plugin');
const { WebpackManifestPlugin } = require('webpack-manifest-plugin');

module.exports = (env, argv) => {
  const isProduction = argv.mode === 'production';

  return {
    mode: isProduction ? 'production' : 'development',
    devtool: isProduction ? 'source-map' : 'eval-cheap-module-source-map',
    entry: {
      application: './app/javascript/application.js'
    },
    output: {
      path: path.resolve(__dirname, 'public', 'build'),
      publicPath: '/build/',
      filename: 'javascripts/[name]-[contenthash].js'
    },
    module: {
      rules: [
        {
          test: /\.css$/,
          use: [MiniCssExtractPlugin.loader, 'css-loader']
        },
        {
          test: /\.scss$/,
          use: [MiniCssExtractPlugin.loader, 'css-loader', 'sass-loader']
        },
        {
          test: /\.js$/,
          exclude: /node_modules/,
          use: {
            loader: 'babel-loader',
            options: {
              presets: ['@babel/preset-env']
            }
          }
        },
        {
          test: /\.(png|jpe?g|svg)$/i,
          use: [
            {
              loader: 'file-loader',
              options: {
                outputPath: 'assets/images/',
                publicPath: '/build/assets/images',
                name: '[name]-[hash].[ext]',
              },
            },
          ],
        },
      ]
    },
    plugins: [
      new MiniCssExtractPlugin({
        filename: 'stylesheets/[name]-[contenthash].css',
      }),
      new WebpackManifestPlugin({
        publicPath: '/build/',
        writeToFileEmit: true,
        generate: (seed, files) => {
          return files.reduce((manifest, file) => {
            const name = path.basename(file.name, path.extname(file.name));
            const ext = path.extname(file.name);
            manifest[name + ext] = file.path.replace(/^.*\/build\//, '');
            return manifest;
          }, seed);
        }
      })
    ],
  };
}

Let's break it down into chunks. Here's the main points:

Basic Configuration

  • entry: Specifies the starting point of your application. Webpack will begin its process here, in this case with ./app/javascript/application.js.

  • output: Defines where the bundled files will be placed. In this setup:

    • path: Where to output the files on the disk (under public/build in your project directory).
    • publicPath: The base path for all assets within the application.
    • filename: Pattern for naming the bundled JavaScript files, where [name] is replaced by the entry name and [contenthash] helps in cache busting by appending a unique hash generated from the file content.

Rules

module.rules: these are the instructions for how to process different types of files. When you see a list of things after the "use" line, it uses the last item first, then the next to last item, etc. So [MiniCssExtractPlugin.loader, 'css-loader'] gets executed first with 'css-loader' and then with the MiniCssExtractPlugin.loader. This is relevant if you're troubleshooting why something won't compile.

  • CSS: Uses MiniCssExtractPlugin.loader and css-loader to process .css files. The plugin extracts CSS into separate files.
  • Sass/SCSS: Similar to CSS but includes sass-loader to handle Sass files.
  • JavaScript: Uses babel-loader to transpile modern JavaScript to backward-compatible versions using Babel, particularly focusing on @babel/preset-env for handling modern JavaScript syntax.
  • Images: Uses file-loader for processing image files like PNG, JPEG, and SVG, placing them in a specific directory and modifying the file names to include a hash.

The end result of these rules is that we've told Webpack how to handle css, scss, javascript, and image files. These should not need to be handled through Sprockets anymore, and any reference to them in the old Sprockets setup (app/assets/javascripts/*) can be removed.

Plugins

  • MiniCssExtractPlugin: Extracts CSS into separate files named according to the [name]-[contenthash].css pattern for caching purposes.

  • WebpackManifestPlugin: Generates a manifest file, useful for managing assets. It maps the original file names to the output filenames, which can include a hash. This plugin is especially handy for integrating with Rails' asset management.

Creating a New application.js File
Creating a New application.js File A new application.js file was created to serve as the main entry point for all JavaScript. This file integrates various libraries and modules.

Structure Modules: Import and configure JavaScript modules and libraries. Stimulus Controllers: Setup and initialize Stimulus controllers if used. Example javascript Copy code // Importing modules import Stimulus from 'stimulus'; import './styles/application.scss';

// Application code document.addEventListener('DOMContentLoaded', () => { console.log('Application loaded'); });

Where webpack Looks for Files
Where Webpack Looks for Files Configuring where Webpack looks for files is crucial to ensure that all assets are correctly processed and bundled.

Configuration Context: Set the context in webpack.config.js to define the base directory for entry points and imports. Resolve: Use the resolve option to specify how modules should be resolved. Example javascript Copy code module.exports = { context: path.resolve(__dirname, 'app/frontend'), resolve: { extensions: ['.js', '.jsx', '.scss'] }, module: { // Module rules } };

Updating .circleci/config.yml
Updating .circleci/config.yml To accommodate the changes in JavaScript management, the .circleci/config.yml file was updated to include steps for installing Node.js dependencies and building assets using Webpack.

Key Updates Node.js Version: Ensure the CI environment uses the correct Node.js version. Build Steps: Add commands to install dependencies and execute yarn build. Example yaml Copy code version: 2 jobs: build: docker: - image: cimg/node:18.20 steps: - checkout - run: yarn install - run: yarn build - save_cache: paths: - node_modules key: v1-dependencies-{{ checksum "package.json" }}

Updating the SCSS files
Updating the SCSS Files Transitioning to jsbundling-rails and Webpack required updates to how SCSS files are managed.

Changes in SCSS Handling Import Paths: Adjusted the import paths in SCSS files to align with the new directory structure enforced by Webpack. Webpack Integration: Configured Webpack to compile SCSS files into CSS, including setting up loaders such as sass-loader and css-loader. Example Here’s how the SCSS configuration part might look in webpack.config.js:

javascript Copy code module: { rules: [ { test: /.scss$/, use: ['style-loader', 'css-loader', 'sass-loader'] } ] }

Updating the Image files
Updating the Image Files With the shift to jsbundling-rails, handling images through Webpack became necessary.

Image Directory Moved all image assets to a directory that Webpack monitors, typically under app/assets/images.

Webpack Configuration Configured Webpack to handle image file processing using file-loader or similar plugins to ensure they are bundled correctly with the JavaScript assets.

Example Here’s a sample Webpack module rule for images:

javascript Copy code module: { rules: [ { test: /.(png|svg|jpg|jpeg|gif)$/i, type: 'asset/resource' } ] }

Building the Front End Running yarn build
Building the Front End Running yarn build is a crucial step to compile all JavaScript and CSS assets.

Command bash Copy code yarn build This command triggers Webpack to process and bundle all specified assets according to the configurations set in webpack.config.js.

About the Manifest
About the Manifest Understanding how Webpack generates and utilizes the manifest file is essential for integrating with Rails.

Manifest Usage Purpose: The manifest file maps the names of source files to their corresponding output file, which may include a hash for caching purposes. Integration: Rails uses this manifest to determine the correct assets to serve. Example Here's how you might read the manifest in Rails:

ruby Copy code manifest_path = Rails.root.join('public', 'packs', 'manifest.json') manifest = JSON.parse(File.read(manifest_path))

Creating Helper Functions
Creating Helper Functions Helper functions in Rails can be used to dynamically link to the assets specified in the Webpack manifest.

Functions Asset Path Helper: Create a helper to return the path of a given asset from the manifest. Example ruby Copy code def webpack_asset_path(filename) manifest_path = Rails.root.join('public', 'packs', 'manifest.json') manifest = JSON.parse(File.read(manifest_path)) "/packs/#{manifest[filename]}" end

Updating application.html.erb
Updating application.html.erb The main layout file application.html.erb was updated to use the new JavaScript and CSS files generated by Webpack.

Changes Script Tags: Updated to reference the bundled JavaScript files. Link Tags: Updated to include the compiled CSS files. Example html Copy code

<title>Treatment Database</title> <%= stylesheet_link_tag webpack_asset_path('application.css') %> <%= javascript_include_tag webpack_asset_path('application.js') %> <%= yield %>
Updating .gitignore
Updating .gitignore As part of the migration, updates to the .gitignore file were made to exclude unnecessary files and directories from being tracked by Git.

Additions Node Modules: node_modules/ Webpack Output: /public/packs/ Temporary Files: *.log, *.pid Example plaintext Copy code /node_modules/ /public/packs/ /tmp/ *.log *.pid

Updating the README
Updating the README The project README was updated to include instructions on setting up the development environment, running builds, and deploying the application.

Sections to Include Setup: Instructions on installing dependencies and setting up the local environment. Building: How to build assets for development and production. Deployment: Steps to deploy the application using updated technologies. Example markdown Copy code

Treatment Database

Setup

To set up your local development environment, follow these steps:

  1. Install dependencies: 
    bundle install
yarn install

Set up the database: bash Copy code rails db:setup Building To build the frontend assets, run:

bash Copy code yarn build Deployment Follow the deployment instructions detailed in config/deploy/qa.rb.

vbnet Copy code

This Markdown document should serve as a comprehensive guide for your team or any developers working on the project, explaining each step of the process in detail. Each section is designed to be self-contained while forming part of a coherent overall guide to the migration process.

Clone this wiki locally