Integrating Eleventy with gulp, upstream JS

  Filed in: gulp, Eleventy, EMBL

Drip. Drop. Drip. Opening the tap of of upstream data objects for Eleventy. Thanks to Flickr user Paul B for the image [CC License].

Eleventy notes that it, "works great with data — use both front matter and external data files" but the static site generator stops short of working well with upstream in-memory data objects for local development.

This post is prompted by a project for a highly-modular design system as part of my work for EMBL — a leading laboratory for the life sciences. We're using Eleventy with gulp (for task running and building) and Fractal (for design components).

We wanted to:

  1. Have gulp trigger builds of Eleventy
  2. Utilise Eleventy’s watch/refresh commands for local development
  3. Send variables to Eleventy’s JavaScript data files from gulp (or other Node JS tasks)

There are methods to achieve part 1 but not with part 2 and 3. But we made a solution.

tl;dr

Still here? Great. Read on.


Scenario

Here's some really useful data, please use it

Fractal creates a really useful JavaScript object that lists component file structure, the contents of those files and compiles nested templates.

An illustrative example of the file structure:

fractalComponents = {
component: “myComponent”,
files: {
  “myComponent.css”: {
     “Contents of the CSS”
   },
  “myComponent.njk”: {
     “Contents of the NJK template”
   },
  “CHANGELOG.md”: {
     “Contents of the changelog”
   },
  “README.md”: {
     “Contents of the readme”
   }
   … and so on


Also, Fractal enables component Nunjucks templates can be invoked like:

<!-- Could not render component '@myNavComponent' - component not found. -->

Which is great as it allows us to npm install components and not have to move/symlink the Nunjucks templates into Eleventy src/ directory — and it keeps syntax consistent across environments.

In principle we could get Eleventy to do the same tasks, scanning the file system and rendering our templates — but we didn’t want to duplicate our build process — especially considering that both Eleventy and Fractal were already running in Node JS.

So, that means we've got some in-memory data we'd really like for Eleventy to receive, and receive updates to if a component is added or edited.

This is important for local development

In our use case we're editing files locally and want a Task A (Fractal) to be able to see changes, updates its data and feed it to Task B (Eleventy).

Eleventy's current design is a non-issue for our production builds. For production, Fractal generates the data and just hands it off to Eleventy. (That said: I could envision a scenario where part of the build process the Eleventy process might want to feed data to some Process C for dynamic rendering.)


So what are you asking for?

Designing our solution

As previously mentioned, Eleventy has a very nice feature supporting JavaScript data files. And as with most watch command's Eleventy's watch observes only file-system changes — that means we need our upstream task (gulp) to be able to trigger an Eleventy rebuild for local development.

So a conceptual example for our desired scenario looks like the below.

  1. Have some upstream data that we want to pass to Eleventy.
// Generate a sample list of all files in a scope outside of Eleventy
gulp.task('file-list', function () {
global.fileList = []; // we could pass by not using a `global`, but this is the simplest for an example 
return gulp.src(['./somePath/**/*.{njk,html,js,md}'])
  .pipe(through.obj(function (file, enc, cb) {
    global.fileList.push(file.path);
    cb(null);
  }));
});

  1. Have an Eleventy data file pull in a variable.
// ./src/site/_data/fileList.js
// Capture the sample list of all files from gulp
// for demonstration Gulp integration with Eleventy
module.exports = {
files: global.fileList
};

  1. On a targeted change event, have Gulp invoke ask Eleventy to rebuilt.
// Watch something for changes
gulp.task('watch', function() {
gulp.watch(['./src/**/*.{njk,html,js,md'], gulp.series('file-list', 'eleventy:reload'));
});

// Or another scenario with an `.on` event triggering a refresh
let fractal = require(fractalConfig).initialize();
fractal.components.on('updated', function() {
elev.restart();
elev.write();
}

// Refresh eleventy
gulp.task('eleventy:reload', function(done) {
elev.restart()
elev.write()
});

Recap: we want to do some local development, let a parent process update a variable and and then ask Eleventy to trigger a rebuild, pulling in the new data by the Eleventy JS data file.

Aside

A child process won’t get us there

Unless you like dumping memory to disk

One method we initially considered for our need was to use Node’s child_process. This is quite clean and is used below by zellwk.com; from zellwk/zellwk.com/blob/master/gulp/eleventy.js

const exec = require('child_process').exec

const eleventy = cb => {
const command = 'eleventy'

exec(command, function (err, stdout, stderr) {
  console.log(stdout)
  console.log(stderr)
  cb(err)
})
}

Using this method you’re also able to run Eleventy with a nice callback on completion — the downside to this method is there’s no clean way to pass in-memory objects to the child_process, you’d need to stringify your variables:

require('child_process').fork('./child.js', [], { env: { FOO: 'bar' } });

For us there are two deal breakers with this method:

  • The object we want to pass is quite large and we'd rather not risk issues with stringification.
  • We'd also need to destroy and re-invoke Eleventy every time during local development, losing access to elev.restart() and elev.write()


Eleventy, can you hear me?

Making it happen

Eleventy’s entry is cmd.js and we need access to elev — but that unfortunately is inside a try statement.

So I forked 11ty’s cmd.js in the local project. Those changes better integrate with external JS with a few minor changes but it's all about a key change: module.exports = elev;

In this way Gulp, or any other Node process, can now use Eleventy as a child task.

How? Like this

  1. Set up Eleventy using our forked local command file.
// Prepare eleventy
process.argv.push('--config=eleventy.js'); // Eleventy config
const elev = require('./eleventy-cmd.js');

  1. We aks eleventy to do its initial build.
gulp.task('eleventy:build', function(done) {
elev.write().then(function() {
  console.log('Done building 11ty');
  done();
});
});

  1. Do a deep rebuild of Eleventy when a file outside of Eleventy’s scope changes or an event trigger is received.
gulp.task('eleventy:reload', function(done) {
elev.restart()
elev.write()
});

This change works well for us but we'll of course need to make sure our local cmd.js incorporates any upstream changes (we forked it from Eleventy 0.9.0)


Enough talking

Here's some code to try

I made a demo repository using this approach that you can:

What's next

  • I'll likely make an issue on Eleventy about supporting this
  • If that doesn't get support (or I feel inspired) I may also make an npm gulp-eleventy-example package