Allow parallel execution of tasks

[patriksvensson]

Image the following task graph which start at 1 and ends at 5:

      1
     /|\
    / | \
   2  3  4
    \ | /
     \|/
      5

Normally, we would do a topographic sort that gives us the order 1 -> 2 -> 3 -> 4 -> 5, but since this example graph is diamond shaped, we could actually run task 2,3 and 4 in parallel, by treating them as a single task which would result in the execution order 1 -> (2 | 3 | 4) -> 5.

Since most tasks in a Cake script probably will be I/O bound, this isn't always what you would want, but theoretically this might shave some seconds of a big build script if applied to the right kind of tasks. We could make this functionality opt-in to prevent it from actually being slower.

[RichiCoder1]

I'd piggy back on this and also (re?)mention the idea of async tasks.

[aabenoja]

Took a stab at this with this spike. Not ideal, but at least it cut down our 8-minute build time down to five.

[jnm2]

A big issue with parallelism is viewing the log output. I came up with a solution that buffers output so that the log looks exactly like the tasks are executed sequentially, but they are in fact simultaneous. As soon as the first action is finished, all the second action's buffered output is dumped immediately and you get to watch the second action continue live- or, if it's also finished, you see the third. Etc.

This can be used in your build scripts without modifying Cake.

https://gist.github.com/jnm2/a8a39b67a584ad555360102407049ae2

Task("Example")
    .Does(() => RunSimultaneously(
        () =>
        {
             StartProcess("ping", new ProcessSettings().WithArguments(a => a.Append("github.com")));
             Information("FINISHED 1");
        },
        () =>
        {
             RunSimultaneously(() =>
             {
                 StartProcess("ping", new ProcessSettings().WithArguments(a => a.Append("192.123.45.67")));
                 Information("FINISHED 2");
             },
             () =>
             {
                 StartProcess("ping", new ProcessSettings().WithArguments(a => a.Append("twitter.com")));
                 Information("FINISHED 3");
             });
        }
    ));

I'd love for some of this to be baked into Cake's dependency graph analysis without even needing to manually specify RunSimultaneously. That would be fantastic!

[pitermarx]

might i suggest this for declaring parallel taks. Feel free to ignore if you think it doesn't make sense:

Task("Default")
.IsDependentOn("Build.Net", "Build.Js");

[aabenoja]

I feel like it should be during the dependency graph analysis that it should determine how to best spin off and schedule tasks on their own threads. I feel that introducing any special syntax is unnecessary, as the same task definition should would regardless of parallelization.

@jnm2 As soon as I dropped my parallelization module into my project my build logs were almost useless. I think I'll take a cue from your gist and incorporate the output capturing into a buffered console writer. Still pretty satisfied with the gains, though my dependency graph analysis algorithm could definitely use some work.

[jnm2]

Okay so how about this use case:

I want to run unit tests and integration tests before building the installer, but both take a while. So what I'd like to do run tests and build the installer at the same time, but cancel building the installer as soon as any test fails. If the installer task finishes before the tests, it should wait for the tests to complete (or fail) before staging the build installer files.

[jnm2]

In pseudocode:

var runUnitTests = Task("RunUnitTests").Does(() => ...);
var integrationTestNewDatabase = Task("IntegrationTestNewDatabase").Does(() => ...);
var integrationTestUpgradeDatabase = Task("IntegrationTestUpgradeDatabase").Does(() => ...);

var buildInstaller = Task("BuildInstaller").Does(parallelDependencies => 
{
    BuildInstallerStep1();

    if (parallelDependencies.HaveFailure) return; // If any fails, let the build end
    BuildInstallerStep2();

    if (parallelDependencies.HaveFailure) return;
    BuildInstallerStep3();

    if (!parallelDependencies.WaitForSuccess()) return; // Waits for all to succeed or any to fail
    StageFiles();
});

Task("TestAndBuildInstaller")
    .DependsOn(runUnitTests)
    .DependsOn(integrationTestNewDatabase)
    .DependsOn(integrationTestUpgradeDatabase)
    .DependsOn(buildInstaller, parallelDependences: new[] {
        runUnitTests,
        integrationTestNewDatabase,
        integrationTestUpgradeDatabase
    });

[aabenoja]

Task("stage-installer")
  .IsDependentOn("unit-tests")
  .IsDependentOn("integration-tests")
  .IsDependentOn("build-installer");

Task("ci")
  .IsDependentOn("unit-tests")
  .IsDependentOn("integration-tests")
  .IsDependentOn("build-installer")
  .IsDependentOn("stage-installer");

We have a similar setup on the project I dropped my module in. unit-tests, integration-tests and build-installer have their own dependencies setup. In the end, these three tasks will all run in parallel and the staging tasks just waits until these are done. If any of these tasks fail the exception bubbles up and the whole build fails; stage-installer gets skipped completely.

[jnm2]

@aabenoja Much cleaner! However, no early exit for "build-installer" which is one of the longest parts for me. They are intrinsically different: I want "build-installer" to exit as soon as any of the tests fail, but I want all test runs to complete even if any of them fail. Could we do this?

It's a little weird getting my head around the fact that Tasks would execute in two stages: 1. IsDependentOn tasks are executed in parallel (if possible) 2. Does(() => ...) runs afterwards, not in parallel with the dependencies. So if you want to force something to not be parallel, you have to create an additional task and separate the non-parallelizable part into a Does section. Seems solid enough though.

[aabenoja]

Actually, build-installer would exit early if it takes significantly longer than either of those test tasks and if either of them fail. I recently fixed an issue where the error wouldn't kill the other task chains. What it doesn't do, however, is continue to run the test tasks regardless of the failure of the others. I think that is something better addressed as a configuration of the individual tasks themselves to continue executing regardless of the state of the cancellation token.

Task("unit-tests")
  .IgnoreCancellation();

Task("integration-tests")
  .IgnoreCancellation();

While maybe not ideal, but at least this way the whole build will still fail and we get the full results of these two tasks.

It makes sense to me that Does(() =>) waits on its dependencies. I wouldn't want to run my unit-tests before I compile.

[jnm2]

I'm thinking cancellation should be opt-in. Thread.Abort has issues for this time of thing and really isn't recommended. I'd be more in favor of passing a CancellationToken and having real cleanup, as in, running processes get killed.

It makes sense to me that Does(() =>) waits on its dependencies. I wouldn't want to run my unit-tests before I compile.

So along those lines, it makes even more sense to me if the dependencies are not parallel unless opted in. Seems the safest approach.

[aabenoja]

I'd be more in favor of passing a CancellationToken and having a real cleanup

Which I am. I've been generating task chains for each dependency, sharing tasks where necessary, and passing around a single CancellationToken across all tasks.

So along those lines, it makes even more sense to me if the dependencies are not parallel unless opted in. Seems the safest approach.

Task("nuget-restore").Does(() => {});

Task("compile")
  .IsDependentOn("nuget-restore")
  .Does(() => {});

Task("unit-tests")
  .IsDependentOn("compile")
  .Does(() => {});

Task("integration-tests")
  .IsDependentOn("compile")
  .Does(() => {});

Task("tests")
  .IsDependentOn("unit-tests")
  .IsDependentOn("integration-tests");

To go more in-depth on what I've done, build.ps1 -t unit-tests will execute "nuget-restore" -> "compile" -> "unit-tests" all sequentially. Same with build.ps1 -t integration-tests. The tests task will run restore and compile sequentially (as expected) and then run unit-tests and integration-tests in parallel.

The whole idea is to execute in parallel where it makes sense. It's possible to throw .IsDependentOn("nuget-restore") onto the integration-tests task. As stated before each task is wrapped in its own TPL Task and shared where it makes sense. Even though integration-tests would await on both tasks to complete that is no different than before.

[aabenoja]

Another example is a mvc web app. Before we create a deployable zip to drop into iis we want to ensure javascript is compiled, javascript tests are passing, our dlls are generated, and xunit tests (both unit and integration) are run. We shouldn't care what order these things and their dependencies occur, just that they happen, no different than when setting up the task without parallelization in mind. If anything, it makes more sense to me to use a cli flag run all tasks sequentially or in parallel. Having a weird mix of parallel and not parallel tasks seems like there's an issue with what is understood as a task's "dependencies."

[jnm2]

Having a weird mix of parallel and not parallel tasks seems like there's an issue with what is understood as a task's "dependencies."

That's certainly an opinionated way for Cake to go, but it might be a good thing.

[jnm2]

Counterexample:

Task("Clean")
    .Does(() => { });

// Priority #1: You want to be able to restore without cleaning
Task("Restore")
    .Does(() => { });

// Priority #2: You want to be able to build without cleaning or restoring
Task("Build")
    .Does(() => { });

// Priority #3: But never pack without cleaning
// Problem: how do you keep all three dependencies from attempting to run at the same time while preserving priorities 1 and 2?
Task("Pack")
    .IsDependentOn("Clean")
    .IsDependentOn("Restore")
    .IsDependentOn("Build")
    .Does(() => { });

[jnm2]

So we need to be able to say:

• Run only Restore when it is called directly, but when invoked as a dependency and Clean is also a dependency, make sure it runs only after Clean

[jnm2]

Do we want to specify a potentially different parallel relationship between dependencies every time they show up in dependent tasks? It could lead to duplication and inconsistency, but the API is minimalist and perhaps the extra control per dependent over the parallelism of the dependencies will be needed someday:

Task("Build")
    .Does(() => { });

Task("TestA")
    .IsDependentOn("Build")
    .Does(() => { });

Task("TestB")
    .IsDependentOn("Build")
    .Does(() => { });

Task("BuildInstaller")
    .IsDependentOn("Build")
    .Does(() => { });

Task("AllTests")
    .IsDependentOnParallel("TestA", "TestB");

Task("TestAndBuildInstaller")
    .IsDependentOnParallel("AllTests", "BuildInstaller");

[jnm2]

So in this API, the order of the IsDependentOn calls matters. Is that a good thing? Is it a breaking change?

Task("Pack")
    .IsDependentOn("Clean")
    .IsDependentOn("Restore")
    .IsDependentOn("Build")
    .IsDependentOnParallel("X", "Y", "Z")
    .Does(() => { });

[jnm2]

@patriksvensson I'm interested in your thoughts.

[jnm2]

Another reason not to parallelize by default is that it messes with logs, and there is a solution for that but it means redirecting the standard output and error of everything you run, losing Windows console colors.

[daveaglick]

Personally, I would prefer any sort of parallelization be opt-in as you suggest. There are just too many build scripts out there to turn on something that so fundamentally changes flow without causing major headaches.

As an alternative to your API, what if we used an .AsParallel() method to indicate that a given task is free to run it's dependencies in parallel. If you have a mix of parallel and non-parallel dependent tasks, then you can create "dummy" tasks to enable the parallelism. With this API your example would look like:

Task("Build")
    .Does(() => { });

Task("TestA")
    .Does(() => { });

Task("TestB")
    .Does(() => { });

Task("BuildInstaller")
    .Does(() => { });

// New task to run tests in parallel
Task("RunTests")
    .IsDependentOn("TestA")
    .IsDependentOn("TestB")
    .AsParallel();

// This would run "Build" followed by "RunTests", which runs only the tests in parallel
Task("AllTests")
    .IsDependentOn("Build")
    .IsDependentOn("RunTests");

// New task to run tests and build installer in parallel
// Since "AllTests" is also parallel, they would be run in parallel with "BuildInstaller"
Task("ParallelTestAndBuildInstaller")
    .IsDependentOn("AllTests")
    .IsDependentOn("BuildInstaller")
    .AsParallel();

Task("TestAndBuildInstaller")
    .IsDependentOn("Build")
    .IsDependentOn("ParallelTestAndBuildInstaller");

It's a little more verbose, but to me it's clearer what's going on and better matches the existing API.

[jnm2]

Cool! That's a new angle.
It's not my favorite; I really dislike boilerplate tasks, it's a bit more verbose, and the most fundamental thing from an API point of view is that it's a modifier on the current task instead of a modifier on a relationship between tasks or dependencies. This looks like it means something it doesn't:

Task("B")
    .IsDependentOn("A")
    .Does(() =>
    {
        // ...
    })
    .AsParallel();

.AsParallelDependencies() would solve that.

[ckolumbus]

Any progress on this topic? I'd like to have such a feature to speed up large project builds.

[andymac4182]

I have used https://cakebuild.net/docs/fundamentals/asynchronous-tasks and RunTarget to get this working now. It isn't pretty but does allow multiple chains of tasks to be ran. If there was a nice way for logs that would be even better.

Task("__SplitHere")
    .Does(async () => {
        var dotNetTasks = System.Threading.Tasks.Task.Run(() => RunTarget("__DotNetTasks"));
        var clientTasks = System.Threading.Tasks.Task.Run(() => RunTarget("__ClientTasks"));
        await System.Threading.Tasks.Task.WhenAll(dotNetTasks, clientTasks);
    });

[jnm2]

@andymac4182 Here's an interim solution for the log issue that might interest you: #156 (comment)

[mabead]

I pretty much liked the suggestion of @andymac4182 to use RunTarget. I tried it out and It works fine with the exception that each call to RunTarget starts a new Cake invocation from scratch. It therefore re-executes the Setup code. Also, it don't get a clean summary of my script at the end like the following:

Task                          Duration
--------------------------------------------------
Setup                         00:00:00.0177014
Restore                       00:00:06.5128068
Build                         00:00:19.4424271
PackageLambda                 00:00:08.3597148
DockerBuild                   00:00:55.6567670
GenerateClient                00:00:36.4401349
UnitTest                      00:01:55.2787180
Publish                       Skipped
Tag                           00:00:00.0032837
--------------------------------------------------
Total:                        00:04:01.7147065

Everything invoked through RunTarget is not listed in the summary. Is there another function that I could use to execute a target without restarting cake from the beginning and still getting a decent task summary?

[dnperfors]

I am currently in the process of trying to run our tasks in parallel as well and also hitting several issues.

I did try to use the solution mentioned in #156 (comment) and this does mostly work, although we ran into issues with the Cake.BuildSystems.Module that reported duplicate tasks, which probably has something to do with the fact that RunTarget will just start a new run.

2019-04-23T09:58:16.0154688Z ##[error]Unable to process command '##vso[task.logdetail progress=61;state=InProgress;id=f1f99a24-b364-4f69-b8f4-2372c902807a;]update' successfully. Please reference documentation (http://go.microsoft.com/fwlink/?LinkId=817296)
2019-04-23T09:58:16.0162344Z ##[error]Name is required for this new timeline record.
2019-04-23T09:58:16.0676280Z ##[error]Unable to process command '##vso[task.logdetail parentid=f1f99a24-b364-4f69-b8f4-2372c902807a;starttime=4/23/2019 11:58:16 AM;progress=0;id=d279ae18-49fb-482b-9627-5c2eb20ba5fe;name=Generate API Docs;type=build;order=3;]create new timeline record' successfully. Please reference documentation (http://go.microsoft.com/fwlink/?LinkId=817296)
2019-04-23T09:58:16.0681466Z ##[error]Parent timeline record has not been created for this new timeline record.
2019-04-23T09:58:16.0702945Z ##[error]An error occurred in the event handler OnTaskSetup: An item with the same key has already been added.
2019-04-23T09:58:16.1098463Z ##[error]An error occurred in the event handler OnTaskSetup: An item with the same key has already been added.

So would it be an idea to add a RunTask which that can be run synchronously, like it would be with IsDependentOn or asynchronously. Basically the usage would look like the following example:

Task("CI")
   .IsDependentOn("Clean")
   .Does(async () =>
   {
      var frontendTask = RunTask("Frontend");
      var backendTask = RunTask("Backend");
      await Task.WaitAll(new[] { frontendTask, backendTask });
      await RunTask("Publish Website");
   });

This RunTask is mainly different from RunTarget that it wouldn't create a complete new context and doesn't show an output itself, but rather it will kind of manipulate the list of tasks that have to be run. The output of the task in the example above would look like this:

Task                            Duration
----------------------------------------------------
Clean                           00:00:14.7548598
Frontend                        00:02:12.6115053
Backend                         00:04:44.4793762
Publish Website                 00:00:18.4674779
----------------------------------------------------
Total:                          00:05:17.7017139

[Mrks83]

Any new thoughts on "how to run tasks in parallel"? Would be awesome to get a statement or roadmap for future Cake evolution.

[adaviding]

I like the suggestion .IsDependentOnParallel() suggestion above: #3534 (comment)

As a developer who is looking to use Cake: I am disappointed that this idea has not been implemented. In my case: I have a bunch of pre-build setup steps that can be executed in parallel, and I would like parallel execution to save time, where each step is a cake Task (so that I don't need to codify shenanigans).

In regards to the difficulty of organizing console output when multiple tasks are running in parallel, the suggestion above sounds nice: #3534 (comment)

[jakubsuchybio]

I guess we are not getting the parallel tasks right? 😭

[devlead]

As with any community open-source project, it's not just about getting it, someone has to develop and maintain it.