bug: Config does not support inheritance for mappers

[peterspiro]

Meltano Version

3.1.0

Python Version

3.9

Bug scope

Configuration (settings parsing, validation, etc.)

Operating System

OSX

Description

I’d like to use mapper configs that inherit from meltano-map-transformer, so that I can distribute them across multiple yaml files, which I already do for my extractors and loaders.

However, the config below produces the error:

Error: Ambiguous mapping name rename_column, found multiple matches.

even though the mapping is only declared once, in the inheriting section.

I don't get the error if I omit inheritance and instead place the mappings sub-section in the meltano.yml config section, omitting the second section.

Code

# in meltano.yml:

  mappers:
    - name: meltano-map-transformer
      pip_url: "git+https://github.com/MeltanoLabs/meltano-map-transform.git"
      executable: meltano-map-transform

# in a different file:

  mappers:
    - name: transform-columns
      inherit_from: meltano-map-transformer
      mappings:
        - name: rename_column
          config:
            stream_maps:
              stream_name:
                new_name: old_name
                old_name: null

[peterspiro]

It looks like this bug may have been introduced by #7992.

[tayloramurphy]

@pnadolny13 can you take a look at this?

[pnadolny13]

@peterspiro can you share more about what your use case is for inheriting mappers?

Mappers work a little different than the extractors/loaders in that they are referred to by their mapping name vs their plugin name, so you wouldnt ever execute the transform-columns plugin directly. You'd reference the mapping within it so having the same mapping name across mappers isnt allowed today. It seems mappings need to be globally unique. Personally I dont know the history of why it was designed that way.

I also dont think plugin inheritance is supported by mappers, potentially because inheritance is usually for reusing a plugin with multiple different configs but in the mapper case thats already the default since mappings is a list.

It sounds like you're using inheritance to define and configure the same mapper in multiple files though. Maybe more details on your use case would clarify but I tested this "hack" and it does seem to work the way you want, multiple definitions of the same mapper plugin each with a single mapping config in it so you'd be able to put these in different yaml files.

  - name: meltano-map-transformer
    variant: meltano
    pip_url: git+https://github.com/MeltanoLabs/meltano-map-transform.git
    mappings:
      - name: rename_column
        config:
          stream_maps:
            stream_name:
              new_name: old_name
              old_name: null
  - name: meltano-map-transformer
    variant: meltano
    mappings:
      - name: rename_column2
        config:
          stream_maps:
            stream_name:
              new_name: old_name2
              old_name: null

[peterspiro]

@pnadolny13 I'm actually just trying to place my mapping definitions in files other than meltano.yml, to group them with their associated extractors and loaders, which are already separated into different yaml files.

I'd be happy to use your hack for this, but it only seems to work if I place both config sections in meltano.yml. If I place the second section in a different yaml file, I still get the above error.

[peterspiro]

Also tested on meltano 3.1.0 and I get the same behavior.