@IritaSee as discussed, here's some more background and assistance for task 3:

To add a new column to the PySpark data frame, use withColumn().
For example df_pois = df_pois.withColumn('brand_matched', match_brand_name(col('brand'))) would add a new column to df_pois called 'brand_matched' which would contain the value returned from the PySpark-UDF match_brand_name.

The UDF itself is applied for each row. So when you pass the col('brand') to the UDF it is working on all values individually (e.g., if there are 1,000 rows in that data frame, the UDF is executed 1,000 times). It effectively works on one string, e.g., 'McDonalds', and simply returns the best match from the brand name list.

Since the UDF is executed many times, we don't want to and can't read the CSV with the brand names a thousand times within the UDF. That's what 'broadcasts' in PySpark are for. They wrap a serializable value and make them available from inside the UDF. This is an example of how to use a broadcast from inside a UDF:

To make the main function more readable, you can create a static function that reads the CSV with the brand names, creates a broadcast from it, adds the two columns brand_matched and operator_matched by applying the UDF and returns the result.

The static function would be applied after the way, node, and relation data frames have been joined. It could look something like this: df_pois = Processor.match_brand_and_operator_names(df_pois)

From looking and the code it seems basically ready. What is still missing @IritaSee?

I feel not ready yet with the PR because of testing issue... I tested them all but I feel the branch did not run smoothly... I think we need to discuss about this but not for so long :") @mattigrthr

This PR is currently on hold. See: #25 (comment)

After doing some initial tests with the brand and operator name matching, it turns out that including the matching in the OSM-POI pipeline directly would increase the runtime significantly. Therefore, we have decided to store the consolidated list of brand and operator names in a separate table in Postgres, which can then be used later in transformation blocks (e.g., on a filtered set of POIs and thus drastically reduce the runtime).

Since the canvas development currently has a higher priority for the core team, this issue is up for grabs again.