Feature eng dev mmt #157
Feature eng dev mmt #157
Conversation
…e_alphanumeric_underscore` .}}/feature_engineering/notebooks/GenerateAndWriteFeatures.py.tmpl
…ing code snippets
…ing code snippets
| @@ -103,7 +103,7 @@ raw_data = spark.read.format("delta").load(input_table_path) | |||
| # Compute the features. This is done by dynamically loading the features module. | |||
| from importlib import import_module | |||
|
|
|||
| mod = import_module(features_module) | |||
| mod = import_module(features_module) ##?? | |||
There was a problem hiding this comment.
What is this comment for?
There was a problem hiding this comment.
Good catch! Should have been omitted -- maybe I deleted after the commit -- will recommit
|
|
||
| fs = feature_store.FeatureStoreClient() | ||
|
|
||
| from databricks.feature_engineering import FeatureEngineeringClient |
There was a problem hiding this comment.
We should only do this if UC is enabled, you can see other examples where we use:
{{ if (eq .input_include_models_in_unity_catalog `no`) }}, or does this client work for non-UC cases as well?
There was a problem hiding this comment.
OK -- I think need to update with conditionals. I was wondering where they were but they need to be added!
…eering related notebooks
| @@ -116,11 +116,11 @@ features_df = compute_features_fn( | |||
| # COMMAND ---------- | |||
| # DBTITLE 1, Write computed features. | |||
|
|
|||
| {{- if (eq .input_include_models_in_unity_catalog "no") }} | |||
There was a problem hiding this comment.
Is there a reason you changed this from your last commit?
| {{- if (eq .input_include_models_in_unity_catalog "no") }} | |
| {{- if (eq .input_include_models_in_unity_catalog `no`) }} |
There was a problem hiding this comment.
There were some failed tests and I noted previous conditionals in section above related to dbutils.wigets with {{- if (eq .input_include_models_in_unity_catalog "no") }} so thought to replace... should it be 'no' instead of "no" ?
There was a problem hiding this comment.
Hmm generally we use backticks (`) instead of quotes (") or ('). You can see other examples here: https://github.com/search?q=repo%3Adatabricks%2Fmlops-stacks%20input_include_models_in_unity_catalog%20&type=code
| label="fare_amount", | ||
| exclude_columns=exclude_columns, | ||
| ) | ||
| {{end -}} |
There was a problem hiding this comment.
So the - at the end means to remove space after this line. You should try generating this notebook in your local Stacks repo by using databricks bundle init and add screenshots of both UC and non-UC to make sure they look as expected
There was a problem hiding this comment.
Will try generating -- I didn't realize I could! Thanks
Updating code with feature store (fs) to feature engineering(fe) wrt https://databricks.atlassian.net/browse/MLOPS-675