Deploying a Craft CMS Site with Gulp and DeployHQ's Automated Build and SSH Commands

Technology and web development processes are always changing for us, as we find better, faster, simpler, more efficient ways to do things. I often look back at the work we did a couple years ago (or even a number of months back) and note how our tools and code have changed. The same is true of the method we use to build and deploy a website. We’ve recently revamped how we do that and wanted to share how we deploy Craft CMS websites using a Gulp workflow and utilizing DeployHQ’s Build and SSH commands.

What’s the Point?

There was a a lot of experimentation and failure involved in setting up this build process. What was the pain point? What problems were we trying to solve? Why bother? (For one, client work was a bit slow this month. When paid work is slow, it’s a great time to spend working on your business, such as reviewing processes to see what can be updated and improved!)

The Problems

We used to use the Minimee plugin to minify our static assets (JS/CSS files) on Craft 2 sites because it was drop dead simple, and we needed no part of a build process to make it happen. If we had CSS or JS that changed, Minimee would see it, quickly concatenate and compress those files, and create a hashed filename, replacing references in our layout. It was a sort of set it and forget it plugin, so we went with it.

The issues with Minimee:

• The dev (nice as he is!) hasn’t updated the plugin for Craft 3
• It’s a CMS specific solution. What if we are building something outside of Craft and want similar functionality?
• As a CMS solution, it’s yet another plugin that needs to be maintained and updated. Overuse of plugins leads to potential problems updating the CMS down the line, if plugins aren’t kept up-to-date as well
• Some user is going to get punished with a slow(er) load time when Minimee decides it’s time to go to work and create new files. Admittedly, in most cases, this is a negligible delay, but using a method other than Minimee will ensure that this is a non-issue.

This was a prime opportunity to review the rather basic Gulp file and workflow we’ve been using for the past few years, to see how we could improve it to also take care of that which Minimee had been doing for us.

Goals and Requirements

We wanted a few things we hadn’t had before out of this update as long as we were going to the trouble.

1. Our local files should have source maps for better debugging. It’s been suggested to keep source maps off of a production site for security reasons, so why not - let’s keep those source maps out of production and off the live server. We take security of our sites (and our client sites) pretty seriously, so if there are steps we can take to eliminate opportunities for bad behavior, we will take them.

2. We want concatenated, compressed and renamed asset files on production. For cache busting purposes, we would like to ensure the filenames themselves are rewritten, and not relying on a query string. In other words, we want our files to look like the top version, not the bottom:

There are references out there for why renaming files for cache busting is a more reliable way to ensure users are getting the latest files. Almost all cache-busting articles will lead you back to a post made in 2008 about query strings, so it’s not clear how relevant this in in 2018, but because we can easily rename the file, and update our links to it, we’ll go that route to be safe.

3. This setup should work with Craft CMS 3, as a Minimee replacement. The other advantage of doing this entirely with Gulp is that this solution isn’t Craft specific. There are Craft plugins that supposedly will rewrite references in files (layouts/templates) based on a rev-manifest, but we want to do this straight up in Gulp. (I could not get either of the Revving plugins to work but I’ll list them here for reference, and in case you want to try that route):

Asset Rev: https://github.com/clubstudioltd/craft-asset-rev

NSM Asset Rev: https://github.com/newism/craft3-asset-rev

4. Only compress and rewrite references to the files on the production server. That means we’ll keep all the local development files concatenated (but unminified) and utilizing source maps, and all the production stuff will be concatenated, compressed, and use no maps.

5. Last but not least, we want to keep all the “built” or “assembled” files out of source control. This includes the “build” folder and source maps, and the “dist” folder that our totally optimized files will live in (that will only ever exist on the production server unless we generate one locally for testing).

To accomplish the above, we only need two things: Gulp, and the available Build and SSH commands provided by our deployment service, DeployHQ.

Our Directory Setup

The root of a basic Craft 3 project of ours looks like this:

Of those, the files we’ll be concerned with in this article are the following:

We’ll touch on all of those files at some point below.

Two other directories will be created - the /public/assets/build directory, and the /public/assets/dist directory - but the contents of these are generated entirely by Gulp, and neither will be added to source control. Build will be created in our local sandboxes, and dist will exist only on the live server. We’ll get to all that in a minute.

Setting up Gulp

Getting this process down gave us a chance to review our Gulp file, and to really understand what’s in it and why. We tried to ensure we were using popular, maintained packages, so we don’t have to find replacements any time soon.

The Gulp Packages

To do what we want, we installed a number of npm packages for Gulp, which you can install via the command line (I’m going to make an assumption that if you’re reading this you’re at least slightly familiar with Gulp already, and might know how to find packages like Gulp Sass on the NPM site and be able to enter npm install gulp-sass --save-dev in the command line. If you’re not familiar with those basics, may I suggest reading An Introduction to Gulp by Nick Salloum, which is where I got a basic grasp of things.)

With those packages installed, our package.json file looks something like this:

Now we need to require those packages in our gulpfile.js, so we can use them. First I’ll break down why we need each one. In order:

gulp
The toolkit itself.

gulp-autoprefixer
This automatically adds browser prefixes to CSS style rules, based on parameters we specify (e.g., last two versions of all major browsers).

gulp-clean-css
A CSS optimizer used to minimize CSS files.

gulp-concat
Concatenates files, so if we have 10 CSS files, this jams them all into one.

gulp-csslint
CSS Lint is a tool to help point out problems and errors in CSS.

gulp-modify-css-urls
Allows us to prepend our CDN domain to relative URLs in the CSS file (e.g., paths to fonts or images). Is only ever used on our “production” task.

gulp-rev
Provides static asset revisioning by appending a hash to file names. This is the main thing the Minimee plugin provided that we can recreate with our Gulp task. For example, main.min.css might be renamed main-14b65d3c80.min.css.

gulp-rev-rewrite
This allows us to rewrite occurrences of filenames which have been renamed by gulp-rev. In other words, if our _layout already was referencing main-14b65d3c80.min.css and we change some styles and update the CSS to main-25c76e4d91.min.css, this plugin will replace that string (again, this only will happen in production when we run a special gulp prod task, which I’ll get into shortly)

gulp-sass
Compiles our source SCSS files to CSS.

gulp-sourcemaps
Useful for development, source maps provide a way of mapping code within a compressed or concatenated file back to the original position in a source file. So if our CSS is compiled from 20 different SCSS files, when we inspect element in dev tools, we know exactly which SCSS files the element styles came from.

gulp-uglify
Minifies JavaScript files.

run-sequence
Runs a sequence of gulp tasks in the specified order. Because we’re still using Gulp 3, we use run sequence to ensure that tasks are run in a certain order. If you’re using Gulp 4, you won’t need this, because “series” and “parallel” are built in.

del
Removes files and folders. We want to keep our dist directory clean, so every time we rebuild and rename the files, we’ll want to get rid of the previous ones. No need to use a gulp plugin for this, because the plain node module does the job perfectly.

We don’t use anything like BrowserSync, or image optimization gulp tasks, because we’re totally fine refreshing a window ourselves and optimizing images used as parts of site elements before we stick them in the repository.

Our entire gulpfile.js looks like this:

With that set up, let’s look at what we need to do with Craft.

Setting up Craft

.env.php variable

We’ll be using assets in “build” when developing locally. The live site will use assets from the “dist” folder. As mentioned before, both of these directories will be kept entirely out of source control, and both are created entirely by Gulp.

We need to set an environment variable so our site knows whether to use the build (development) or dist (production) files. We’ll first add a variable to our .env.php file. This will also work if you use Craft “out of the box” with the included .env file - which we don’t do, in part because it’s recommended to not use .env in production. Seriously. The authors said it themselves in the project’s README on Github:

For that reason, we use a modified version of Andrew Welch’s Craft Multi-Environment setup, which uses a .env.php file. I’m not going to get into a debate here about which to use: suffice to say, no matter what style environment file you use for your Craft multi-environment setup, you will need a variable so that your local development site will know to use the build files, and production will know to use the dist files. We decided to call it USE_PROD_ASSETS:

All of our environment variables get prefixed with CRAFTENV_ automatically, just like in Andrew’s script.

Then, with that in mind, in site/config/general.php, we add a config variable that references that environment variable, so we can later set up a conditional in our templates:

Revision Templates

We start by adding two very small files in a “revisions” folder we put inside our templates directory, and we .gitignore both of them:

css.html
This only contains one line: /assets/dist/css/main.min.css

js.html
This too, only contains one line: /assets/dist/js/main.min.js

When our gulp prod task runs, these files will get rewritten with the hashes from the rev manifest file, so they’ll change every time our CSS or JS is updated. We don’t want to deploy them to the server and overwrite the hashed values out there with the base values we have in these files to start.

The Layout

In our layout template, we add a few variables at the top, and add a line for the CSS, and towards the bottom we add one line where we include all our JS:

First we’re setting two variables to be equal to the contents of the files they’re referencing. Then we’re using that variable in another variable that has a conditional to check whether we want to use the prod assets or not: if we do, output the variable, otherwise just output a path to our build files. Then we just output those in the correct link or script tag.

The .gitignore File

We need to ignore all our built files, and the files that will get re-written with every build, so we make sure to add the following to our .gitignore file:

Because all the built files are ignored, when a developer pulls, they simply have to remember to run ‘gulp’ or ‘gulp watch’ in order to generate the most up-to-date builds of the CSS and JS, and their source maps.

That’s the extent of setting up files in Craft. All that’s left to do is set up DeployHQ.

Setting up DeployHQ

We’ve used DeployHQ for many years to deploy our sites over SFTP from Github and Bitbucket repositories. DeployHQ offers both SSH commands and Build commands when deploying, and these are the last piece of the puzzle that will allow us to do everything we want to do.

I found the DeployHQ docs a little sparse when it came to explaining how to use these commands, so there was a lot of trial and error involved, and a number of support emails back and forth with the patient and detailed support team. The process is worked out now and I can share with you what we’ve learned.

Build Commands

We’ll set up two Build Commands: npm install and gulp prod.

These are very simple - in this case the name of the command is the exact same thing as the command itself.

Cached Build Files

DeployHQ allows you to cache build files, which in our case comes in very handy for the node_modules directory, which has zillions of files in it. The deployments go much faster if these files are cached, which is easy to set up with one line in the DeployHQ settings. Under Cached Build Files, add one file path for node_modules/**.

Excluded Files

In addition to putting node_modules in our .gitignore file (so it never gets deployed to the production server), we also have to add node_modules to our excluded files list, so that DeployHQ does not copy the node_modules directory from the build server to the live server after it has run the build commands.

On DeployHQ, there are two ways you can exclude files from being deployed to the server: either use a .deployignore file or use the Excluded Files settings under your project.

If you’re using the Excluded files settings in your project, those might look like this:

You can add a line each for node_modules and node_modules/**.

If you’re using a .deployignore file (which is what we do), we add those lines to it:

To keep things as fast as possible DeployHQ also only uploads files back to your server that are changed since the last build. Now that the Build Commands are set, we need to add our SSH Command to finish up.

SSH Commands

Because we’re using Craft 3, we often need to run composer install after we’ve pulled from the repository, in the event any of us at Masuga Design has installed or updated Craft itself, or a third party plugin. In the case of our production server, we use an SSH command to run composer install on the server after we deploy all the files, so that we don’t have to manually SSH into the server every time, cd to the right directory, and run it ourselves.

We only need to add a single command here, which we add “After Deployment” - or after any files have been copied back from the build server. The command is: cd %path% && composer install. Note that %%path%% is a DeployHQ variable that represents the base server path we’re deploying to. So we’re instructing DeployHQ (which already has SSH access to our server) to change to the directory in which it deployed items, and to run composer install for us.

We let it run for a very reasonable 5 minutes (the lowest setting) on every deployment. There is no real issue running it on every deployment, because if there is nothing to install or update, the command does nothing and is over with nearly immediately.

One cool thing about the DeployHQ deployments with the Build and SSH commands in place is that you can see them in action if you’re watching a deployment. They open up little terminal windows and you can see the server run gulp prod for you, and then composer install. It’s nice to be able to see what’s going on as it happens.

Results

It took a lot of testing to get our latest Gulp setup right, and to make sure it worked. As with anything remotely technical, by the time you read this we may have already found better methods - everything in the web industry is a work in progress.

We have this build process working on a few Craft 3 sites now (as well as a few Craft 2 sites, where we use the Build commands for Gulp, but skip the SSH command for composer) and it’s working very well.

We now have a complete process in place that takes only a few minutes to set up for each project. It ensures that users are getting the latest compressed CSS and JS in completely renamed files, and hits all the other goals we set out to achieve in our workflow.