Describe integration with MLflow

[astrojuanlu]

See #3541.

Description

Development notes

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

• Read the contributing guidelines
• Signed off each commit with a Developer Certificate of Origin (DCO)
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes
• Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

[astrojuanlu]

Preview: https://kedro--3856.org.readthedocs.build/en/3856/integrations/mlflow.html

[astrojuanlu]

Note to reviewers:

The idea of this page is to complement kedro-mlflow documentation, and in fact it contains several references to it. What this adds then is:

• References to MLflow in the Kedro docs so users can easily find it
• Short examples on how to create custom integrations using hooks
• Fill some gaps in kedro-mlflow

The idea is for this page to serve as brief collection of MLOps use cases, and to use this as a template for future integrations.

I'm paging @stichbury as well because I was somewhat careless with the prose in certain parts.

I confess disclose that Lilli wrote the first 2 paragraphs.

[noklam]

Is it possible to keep kedro-mlflow as its own section? The current structure is by feature, "tracking", "artifact". The rest of our docs usually start from "basic" -> "advance".

[astrojuanlu]

Is it possible to keep kedro-mlflow as its own section? The current structure is by feature, "tracking", "artifact". The rest of our docs usually start from "basic" -> "advance".

I went back and forth several times on how to structure this page. I like how I ended up making 1 section per use case but I agree there might be other ways. Awaiting for @stichbury's take on this.

[stichbury]

I went back and forth several times on how to structure this page. I like how I ended up making 1 section per use case but I agree there might be other ways. Awaiting for @stichbury's take on this.

I found the sectioning useful but agree with @noklam that we usually break into simple and advanced usage. Can I suggest we do the same here and have the structure as follows, but will leave you to decide where the simple/advanced sections fall?

Maybe something like this?

However, I'm not that attached to this and if you want to stick with what you have, I'd say that's fine, but omit the basic second level "Use cases" header and promote all the following (currently 3rd level) to 2nd level.

Header

Prerequisites

Simple use cases

Tracking Kedro pipeline runs in MLflow using Hooks

Artifact tracking in MLflow using hooks

Advanced use cases

Complete tracking of Kedro runs in MLflow using kedro-mlflow

Tracking Kedro in MLflow using the Python API

Artifact tracking in MLflow using kedro-mlflow

Model registry in MLflow using kedro-mlflow

[astrojuanlu]

My biggest gripe with this is that it seems wrong to declare custom hooks as "basic" and using kedro-mlflow, which is objectively fewer lines of code (just a pip install kedro-mlflow away), "advanced". If anything, the former are more "custom", "ad-hoc", or "homegrown", whereas the latter is more "off-the-shelf".

I can totally see how someone starts with the custom hook ("basic"), then they start making it more complex because they need more functionality, and in the end it becomes way more difficult than just pip install kedro-mlflow and let the plugin take care of it for you, assuming the plugin does more or less exactly what you want to do.

[stichbury]

I can't work on your branch so I've forked and made a PR to commit back to it #3862

Please take a look, merge what you want, and I can review again when you have the entire page in your preferred final state (see comment about sectioning above).

[astrojuanlu]

I'd say that's fine, but omit the basic second level "Use cases" header and promote all the following (currently 3rd level) to 2nd level.

We agreed to do this 👍🏼 Will make the change today

[DimedS]

Thank you, @astrojuanlu , for the excellent manual. Everything is working well. It's a great starting point for exploring Kedro+MLFlow. I've left a few minor comments.

[astrojuanlu]

Thanks for the review @Galileo-Galilei 🙏🏼 will address your comments ASAP.

[astrojuanlu]

I significantly reworked the order of the sections, but the content is largely the same. I think the flow is much nicer now - wouldn't have reached this stage without @Galileo-Galilei's insightful comments.

Please do have a look again.

[astrojuanlu]

Preview: https://kedro--3856.org.readthedocs.build/en/3856/integrations/mlflow.html

[stichbury]

Looks really great, well done 🌟 I made just 2 very minor suggestions.

[DimedS]

Fantastic work, @astrojuanlu! 🚀🚀🚀 I really like the new description flow, starting from easy use cases and progressing to more complex ones like Hooks and Session management. I left a few minor comments.