Order of 'includeConfig' and 'profiles' in nextflow.config seems to affect nested property overrides

[danoreper]

Bug report

Expected behavior and actual behavior

Hi Nextflow team,

I expected that the order in which 'includeConfig' and 'profiles' are declared in nextflow.config would have no effect on property values.
However, I think I'm seeing that 'profiles' must be declared after any 'includeConfig' statements, in order for profile overrides to work correctly. In particular, this seems to affect what I'd describe as nested properties.

Steps to reproduce the problem

Given the following two files:

nextflow.config:

profiles {
    hpc {
        process {

         process_nested.value = "hpc.config"
         process_nonest       = "hpc.config"
       
         withLabel: "labelExample" {
             memory = 400
        }
       }
    }
}

includeConfig "memory.config"

and memory.config:

process {
    process_nested.value = "memory.config"
    process_nonest       = "memory.config"

    withLabel: "labelExample" {
        memory = 4
    }
}

and using the following command line invocation:

nextflow config -profile hpc

The nested properties 'process_nested.value' and 'labelExample' fail to be overridden by the hpc profile, while 'process_nonest' is correctly overridden.

(Of note, the problematic overrides work as expected once the 'includeConfig' declaration is moved above the 'profiles' declaration)

Program output

the unexpected output is as follows:

process {
   process_nested {
      value = 'memory.config'
   }
   process_nonest = 'hpc.config'
   withLabel:labelExample {
      memory = 4
   }
}

(No .nextflow.log is generated by nextflow config)

Environment

• Nextflow version: Both 23.10.1 and 23.04.4
• Java version: Java/17.0.4
• Operating system: Linux
• Bash version: GNU bash, version 4.2.46(2)-release

Additional context

The example provided is for the nextflow config command, but I'm also seeing similar behavior for nextflow run

[bentsherman]

This will be fixed by #4744

[danoreper]

Thanks @bentsherman !