Node memory leak running assemble

[bitdepth]

Hey @jonschlinkert, we are experiencing a very similar issue to #871 at the moment and our team are also considering porting to another static site generator.

Version

• assemble 0.24.3 with base-watch 0.1.3
• node 8.2.1

Description

Adding content to the partials and pages causes the build time to grow until eventually node runs out of memory and the build falls over.

Error Message

[09:27:50] starting assemble:default task
[09:27:54] finished assemble:default task
[09:27:54] finished assemble 4s
[09:27:59] starting assemble 
[09:27:59] starting assemble:default task
[09:28:04] finished assemble:default task
[09:28:04] finished assemble 4s
[09:28:09] starting assemble 
[09:28:09] starting assemble:default task
[09:28:14] finished assemble:default task
[09:28:14] finished assemble 5s
[09:28:20] starting assemble 
[09:28:20] starting assemble:default task
[09:28:49] finished assemble:default task
[09:28:49] finished assemble 28s
[09:29:00] starting assemble 
[09:29:00] starting assemble:default task
[09:29:47] finished assemble:default task
[09:29:47] finished assemble 47s
[09:30:03] starting assemble 
[09:30:03] starting assemble:default task
[09:36:44] finished assemble:default task
[09:36:44] finished assemble 7m
[09:44:38] starting assemble 
[09:44:38] starting assemble:default task

<--- Last few GCs --->

[75558:0x102801600]  3337194 ms: Mark-sweep 1400.3 (1589.9) -> 1400.4 (1589.4) MB, 1124.6 / 0.0 ms  allocation failure GC in old space requested
[75558:0x102801600]  3338282 ms: Mark-sweep 1400.4 (1589.4) -> 1400.1 (1556.9) MB, 1087.6 / 0.0 ms  last resort 
[75558:0x102801600]  3339414 ms: Mark-sweep 1400.1 (1556.9) -> 1400.0 (1556.4) MB, 1130.9 / 0.0 ms  last resort 


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x3744d469bbd9 <JS Object>
    1: DoJoin(aka DoJoin) [native array.js:~96] [pc=0xc476b4c3e3d](this=0x3744d4682241 <undefined>,p=0x3a0b5197bc41 <JS Array[2]>,q=2,E=0x3744d46822c1 <true>,A=0x3306fc442f21 <Very long string[9414]>,z=0x3744d46822f1 <false>)
    2: Join(aka Join) [native array.js:~121] [pc=0xc476b4f2388](this=0x3744d4682241 <undefined>,p=0x3a0b5197bc41 <JS Array[2]>,q=2,A=0x3306fc442f21 <Very long string[9414]...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/Users/t4004214/.nvm/versions/node/v8.2.1/bin/node]
 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/Users/t4004214/.nvm/versions/node/v8.2.1/bin/node]
 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/Users/t4004214/.nvm/versions/node/v8.2.1/bin/node]
 4: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/Users/t4004214/.nvm/versions/node/v8.2.1/bin/node]
 5: v8::internal::Runtime_StringBuilderJoin(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/t4004214/.nvm/versions/node/v8.2.1/bin/node]
 6: 0xc476a8040bd
 7: 0xc476b4c3e3d
Abort trap: 6
t4004214@macla214572 build-fail (master)*$ 

assemblefile.js

'use strict';

const assemble = require('assemble');
const watch = require('base-watch');
const handlebars_helpers = require('handlebars-helpers');

const app = assemble();

app.use(watch());

app.task('default', () => {

	// console.log('moo');
	app.data('./assemble/data/*.json');
	app.layouts('./assemble/templates/layouts/**/*.hbs');
	app.partials('./assemble/templates/components/**/*.hbs');
	app.pages('./assemble/pages/**/*.{md,hbs}');
	app.helpers([handlebars_helpers(),'./assemble/helpers/**/*.js']);

	return app.toStream('pages')
		.pipe(app.renderFile())
		.pipe(app.dest('./dist'));
});

app.task('watch', () => {
	app.watch(['./assemble/**/*.hbs', './assemble/data/**/*.json', './assemble/helpers/**/*.js'],  ['default']);
});

module.exports = app;

I have gone through some of the suggestions in this #871 and the build runs fine until i add the partials directory so i believe the error is caused by some combination of the data and the partials working together.

I have isolated the assemble stuff and added it to a private repo which i can share with you.

[assemblebot]

@bitdepth Thanks for the issue! If you're reporting a bug, please be sure to include:

• The version of assemble you are using.
• Your assemblefile.js (This can be in a gist)
• The commandline output. (Screenshot or gist is fine)
• What you expected to happen instead.

If your issue is related to one of the following, please open an issue there:

• grunt-assemble Issues with using assemble in grunt or the grunt-assemble library.
• handlebars-helpers Issues with using handlebars helpers from the handlebars-helpers library.

[doowb]

@bitdepth if you add me to the repo, I'll take a look when I have a chance.

[jonschlinkert]

It's most likely not an assemble bug. As with the other issue that you commented on, it's most likely caused by something you're doing wrong, like merging data with cyclical references (most likely in a helper).

If you've narrowed it down to the partials, then you should be able to narrow it down to a specific partial by eliminating them one at a time, or by replacing the contents of each partial with an empty string.

This is what @doowb or I would end up doing first. After that, if the problem still persists and we're not able to find a user error, we'll dig deeper in assemble's code (or more likely the code of a helper) to figure out where the bug is.

[jonschlinkert]

you probably should not be reloading helpers every time .watch is called. If you need to do that, you'll probably need to pass a new handlebars instance every time (call handlebars.create()).

edit: does the problem persist after removing watch?

[struct78]

I've had the same issue with memory leaks while running assemble in a watch task - it appears to be a problem with Handlebars rather than assemble from what I could tell, but I couldn't get a definitive answer of where the problem lies.

My solution was to use gulp-cached before calling assemble.renderFile()

  assemble.task('html.build', function(){
    return assemble.toStream('pages')
                  .pipe(cache('html.build'))
                  .pipe(assemble.renderFile())
                  .pipe(rename({ extname:'.html' }))
                  .pipe(assemble.dest('dist/html'));
  });

Then modify the watch task, this has to be done because you need to bust the cache when any partials or data files are changed.

assemble.task('watch', function(){
     assemble.watch([
      'src/**/*.hbs',
      'src/**/data/*.json'
    ], [
      'html'
    ], function(done){
       browserSync.reload();
       done();
    }).on('change', function(file) {
      // NOTE: gulp-cached keeps a copy of the handlebars page, but not the partials or data files
      // So when these files are modified, gulp-cached just skips over them and no changes are reflected
      // This callback function was created to remove the relevant page from the cache if any of the
      // partials or data files are modified
      var directory = path.dirname(file).split(path.sep);
      var folder = directory.pop();
      var paths = ['data', 'partials'];
      var cacheBust = false;

      paths.forEach(function(path) {
        if (folder === path) {
          cacheBust = true;
        }
      });

      // Check the key exists in the cache first
      if (cache.caches['html.build'] && cacheBust) {
        for (var key in cache.caches['html.build']) {
          var cacheFolder = path.dirname(key);
          var parentPath = directory.join(path.sep);

          // Paths match, so delete it from the folder
          // We also need to remove any global pages from the cache
          if (cacheFolder.endsWith(parentPath) || cacheFolder.endsWith('html')) {
            delete cache.caches['html.build'][key];
          }
        }
      }
    });

Not ideal, but our watch tasks really fly now.

[jonschlinkert]

it appears to be a problem with Handlebars rather than assemble

That's similar to what I was thinking. My hunch is that it's a combination of helper binding and watch, or something similar that is causing a memory leak. I'm going to look into this as soon as I get some free time in the next few days.