Typed tag metadata

[gravesee]

To ensure decorated tag key/values conform to a specified schema, it would be great if Hamilton supported typing and validation of metadata. Perhaps a tag decorator generator that takes a TypedDict as input and returns a decorator class that supports static typing of tag inputs and also validates tag arguments using something like pydantic.

class FilmDetails(TypedDict):
    title: str
    year: int


film = typedtag(FilmDetails)

# example 1 - correct
@film(title='Casablanca', year=1942)
def movie():
    ...


# example 2 - mypy/pylance complain (typos)
@film(titel='Casablanca', yeer=1942)
def movie():
    ...


# example 3 - fails mypy/pylance, but pydantic would coerce year to int anyway
@film(title='Casablanca', year="1942")
def movie():
    ...

Related, tag merging would help teams annotate features using their own type-checked namespaces. For example,

@awards(oscars=['best writing', 'best picture', 'best directing'])
@film(title='Casablanca', year=1942)
def movie():
    ...

# tag for `movie` results in:

{
   'awards': {
        'oscars': ['best writing', 'best picture', 'best directing'],
   },
   'film': {
       'title': 'Casablanca',
       'year': 1942,
   }
}

[elijahbenizzy]

I'm a big fan of this -- and I really like your API. Its a lot cleaner than anything I've thought of. Only potential problem I can see is that its a very open way to write things. E.G. it now enables infinite decorators, which could be fully customizable. That said, this is actually entirely doable by delegating to @tag, so I'm not sure that's bad. Also, while it opens up the number of decorators people uses, it locks down the inputs to them, which increases readability and safety.

One more idea (not sure its better, just throwing it out there...):

class FilmDetails(TypedDict):
    title: str
    year: int


film = typedtag(FilmDetails)

# example 1 - correct
@tag(class_=film, title='Casablanca', year=1942)
def movie():
    ...


# example 2 - mypy/pylance complain (typos)
@tag(class_=film, titel='Casablanca', yeer=1942)
def movie():
    ...


# example 3 - fails mypy/pylance, but pydantic would coerce year to int anyway
@tag(class_=film, title='Casablanca', year="1942")
def movie():
    ...

But yeah, I think yours is nicer. Maybe the convention is that everyone uses tag_ as the name (tag_film, tag_awards, etc...).

A few more points:

1. Lists aren't currently allowed, so your example will break. With this we should open it up (only, perhaps, if people use typedtag, otherwise it should be restricting to strings).
2. We might want to restrict to some pretty basic types at first so it doesn't get out of hand (primitives + lists of primitives).
3. We also need to support [tag_outputs]
   (https://github.com/stitchfix/hamilton/blob/main/hamilton/function_modifiers/metadata.py#L122), but I think that's trivial, have to write out some examples to be 100%.
4. Custom validations might be nice (E.G. year has to be between 1900 and 2000 or something like that). In this case TypedDict isn't sophisticated enough I think -- we'd need something like pydantic (as you suggested) to do it. I'd like to keep heavy dependencies out, but it might be a nice option as an extension.
5. Might be worth thinking about longer term notions of governance for this. For example, what if we had a driver (or graph adapter/reader) that ensured that all upstream data sources had to be tagged with whether or not they contained PII? Could easily be done customely with a graph-walk, but I'd be curious if this is something you need (my hypothesis is that it could prove valuable...)

Thanks for the suggestion! Definitely excited about this one.

[elijahbenizzy]

OK, POC -- pretty straightforward. This does exactly that -- makes it easier by restricting it to primitives. #277. Nice thing is it delegates to the original tagging decorator as well -- cuts new code.

Slightly rough around the edges but if you want to give it a try @gravesee I'm happy to fix any bugs :) Its tested for some basic use cases (see test_metadata.py)

Before we'd get this out, we need:

1. Clarify best practices
2. Add a few more tests
3. Update documentation
4. Get it to work with tag_outputs
5. Ensure it breaks in all cases that it should (I think this will let unknown tags in)
6. Potentially find a way of codifying governance as you suggested above?

@skrawcz I like the way you have it centrally above... Could also validate, list, manage tags, etc...

[elijahbenizzy]

And another thought -- I'd prefer not to use pydantic/another schema generator in the core python library, but that would be a good extension to add -- e.g. a tag schema. Could also allow for functions inputting/outputting pydantic models (doable, but we could have some special cases)

[gravesee]

Related @skrawcz 's question about when validation should occur. Is the Hamilton team open to using typing generics? It would be nice to know, as a developer, that I've incorrectly entered metadata. It's a little trickier due to the restricted types allowed in typed_tag. It's not much of a burdern to run the module I'm working on currently and have the runtime validator notify me of errors.

Having the editor yell loudly that I did something wrong is a good develeoper experience in my view. Another use-case would be when a tag schema is updated and metadata needs to be updated. The current design would, I believe fail one at a time and I would need to fix the code at each failing location. Leaning on a type-checker would point all these issues out at once.

Lastly, it's unclear to me whether the current tag decorator supports enums or literals. It is often helpful to restrict the values to a subset of allowed choices. Unrelated to this discussion, but something that occured to me as I was checking out the new decorator.

[elijahbenizzy]

Yeah, nothing wrong with generics. This was a quick POC -- I think we can probably use generics to make the dev experience cleaner. That said, anything more than just types (E.G. validation functions) will not provide the safety you want (maybe pydantic with an IDE plugin could?). But, I think this might be a happy middle-ground. I'd prefer to keep it breaking on import/decorator (although we could consider a lazily evaluated set of errors...), so relying on a static type-checker for dev purposes with generics could be the right approach.

Just need to figure out the right way to use generic typing to enable an IDE to be happy -- think its pretty straightforward.

And yeah, +1 for enums (was just thinking about it last night) -- should be pretty trivial to add. By literals you mean an enum of one possibility -- e.g. the Literal type?

Will mess around with typing/generics soon.

[skrawcz]

@gravesee what's your timeline? We have a few things in flight. Would love to jump on call to understand this idea and your needs a bit more so we can figure out how to allocate our time. Want to ping me on Linkedin or via email?