[Spec] Add Iceberg Materialized View Spec

[wmoustafa]

This PR adds Iceberg materialized view spec based on the MV model of a separate Iceberg view and table.

This PR is the based on extensive community discussions in various forums [1, 2, 3, 4, 5].

A Spark implementation of this PR can be found in [6]

[1] https://docs.google.com/document/d/1zg0wQ5bVKTckf7-K_cdwF4mlRi6sixLcyEh6jErpGYY
[2] #6420
[3] https://docs.google.com/document/d/1UnhldHhe3Grz8JBngwXPA6ZZord1xMedY5ukEhZYF-A
[4] https://lists.apache.org/thread/tb3wcs7czjvjbq9y1qtr87g9s95ky5zh
[5] https://lists.apache.org/thread/l6cvrp4r1001k08cy2ypybzy2kgxpt1y
[6] #9830

[wgtmac]

Suggested change

	Iceberg Materialized views are a way to store the results of an Iceberg view to reuse the computation in subsequent queries.
	Iceberg materialized views are a way to store the results of an Iceberg view to reuse the computation in subsequent queries.

[stevenzwu]

minor suggestion

Iceberg materialized view stores the precomputed results of an Iceberg view, which can be reused in subsequent queries to avoid recomputation.

[ajantha-bhat]

is iceberg.base.table.snapshot.[UUID] is the better name for this property? which is also inline with the iceberg.child.view.version.[UUID]

[ajantha-bhat]

Also, isn't the current snapshot id enough? (asking because we have mentioned as IDs).
Maybe we can update it as "These properties store the current snapshot ID of the base tables ..."

And If the stored snapshot id is expired from the base table, we can't incrementally refresh MV? because we can't map that snapshot? In this case MV does a full refresh (expensive)? Should expire snapshots should be made MV aware so that it gives warning while expiring the referenced snapshots?

[ajantha-bhat]

Just by reading this line, I couldn't understand what are these child views? how it is created? and why it is required? maybe adding an example at Appendix section can be helpful.

[wmoustafa]

I have updated the text description to this:

If the top level materialized view is nested on top of other views, these properties store the version of the child views that the top level view is associated with at the time of materialization. Each property is prefixed with 'child.view.version.' followed by the UUID of the child view.

Please let me know if this clarifies it. I have added examples too. This is explained in Example 3.

[ajantha-bhat]

Thanks. I understood it now. But the fact that event_type_count and event_type_count has to be created before the event_analysis, makes them parent instead of child view here? Do we need to change the property name?

[wmoustafa]

I do not view it form the creation time point of view (which does not constitute a hierarchical relationship either), but rather from a view graph point of view. This graph will form a tree, where the inner views are children.

[ajantha-bhat]

OK

[ajantha-bhat]

Suggested change

	* For each child view `V` with `UUID` in `iceberg.child.view.version.[UUID]`, the version of `V` is equal to the value stored in `iceberg.child.view.version.[UUID]`.
	* For each child view `V` with `UUID` in `iceberg.child.view.version.[UUID]`, the current version ID of `V` is equal to the value stored in `iceberg.child.view.version.[UUID]`.

[ajantha-bhat]

Suggested change

	* The version of the view using the table as the storage table is equal to the value stored in `iceberg.view.version`.
	* The current version ID of the view using the table as the storage table is equal to the value stored in `iceberg.view.version`.

[ajantha-bhat]

Suggested change

	| `iceberg.view.version` | This property stores the version of the view that this storage table is associated with at the time of materialization|
	| `iceberg.view.version` | This property stores the version ID of the view that this storage table is associated with at the time of materialization.|

[ajantha-bhat]

Suggested change

	| `iceberg.child.view.version.[UUID]` | These properties store the version of the child views that the top level view is associated with at the time of materialization. Each property is prefixed with `iceberg.child.view.version.` followed by the UUID of the child view.|
	| `iceberg.child.view.version.[UUID]` | These properties store the version ID of the child views that the top level view is associated with at the time of materialization. Each property is prefixed with `iceberg.child.view.version.` followed by the UUID of the child view.|

[stevenzwu]

nit: remove however. However, such views -> Such logical views

[stevenzwu]

minor suggestion

Iceberg materialized view stores the precomputed results of an Iceberg view, which can be reused in subsequent queries to avoid recomputation.

[stevenzwu]

regarding property name iceberg.materialized.view, it is redundant to have iceberg there.

metadata field name uses hyphen -. so maybe like view-materialized and view-storage-table.

[wmoustafa]

I have removed the iceberg prefix. Not quite sure if there is a rule about about hyphen vs dot. For example, there is a table property: schema.name-mapping.default in the table spec.

[jackye1995]

+1 for Steven's suggestion, which is what I also wrote in the comment below. Take schema.name-mapping.default as an example, there might be other schema related properties in the future, thus the schema. prefix. In general . is used for grouping purpose.

[wmoustafa]

Ack. I have removed iceberg. already. I am thinking of introducing mv. now for the grouping purpose.

[stevenzwu]

UUID suffix in the property name weren't quite readable. I wondering if we should have a consolidated property for base tables, like base-tables-snapshots. the value could be a JSON string like

{ "table1" : {"uuid": "xxx", "snapshot-id": 1} , "table2" : {"uuid": "xxx", "snapshot-id": 2} }

The reason is that all base tables tracking should be updated in one shot anyway. They are also always validated together.

With JSON, we can also store additional metadata in the future.

[stevenzwu]

shouldn't the lineage to child views be defined in the view metadata (not storage table metadata)? child views refer to the parent MV by view name (not storage table name).

[wmoustafa]

I am not sure if we mean the same thing when referring to child views here. Could you review this thread and let me know if it changes the perspective here?

[stevenzwu]

why does child views' version matter for the parent view/storage table?

[wmoustafa]

I have added an Examples section. Could you check Example 3?

[ajantha-bhat]

Suggested change

	| `base.snapshot.[UUID]` | These properties store the snapshot IDs of the base tables at the time the materialized view's data was last updated. Each property is prefixed with `base.snapshot.` followed by the UUID of the base table.|
	| `base.table.snapshot.[UUID]` | These properties store the current snapshot ID of the base tables at the time the materialized view's data was last updated. Each property is prefixed with `base.table.snapshot.` followed by the UUID of the base table.|

[ajantha-bhat]

other places also we can replace base.snapshot with base.table.snapshot

[ajantha-bhat]

Suggested change

	| `view.version` | This property stores the version of the view that this storage table is associated with at the time of materialization|
	| `view.version` | This property stores the version ID of the view that this storage table is associated with at the time of materialization|

[ajantha-bhat]

Suggested change

	- `event_analysis`: `SELECT E1.event_type, E1.count, E2.region, E2.count FROM EventTypeCount E1 JOIN EventRegionCount E2 ON E1.event_type = E2.event_type`
	- `event_analysis`: `SELECT E1.event_type, E1.count, E2.region, E2.count FROM event_type_count E1 JOIN event_region_count E2 ON E1.event_type = E2.event_type`

[ajantha-bhat]

there is a chance that storage table of child views are outdated (not fresh). So, just comparing the view versions is not enough to conclude child views are fresh?

[wmoustafa]

In case child views are materialized, their storage table is not relevant to determining the freshness of the current (top level view) storage table.

is not enough to conclude child views are fresh

The objective is not to tell whether child views are fresh, but rather if the current storage table is a valid materialization of the current view tree. If the view tree changes in any way, the current table is no longer a valid materialization for the current view tree.

[wmoustafa]

I have added the above explanation to the spec text.

[szehon-ho]

Agree with @JanKaul, i think we should try to reconcile this a bit with the design doc https://docs.google.com/document/d/1UnhldHhe3Grz8JBngwXPA6ZZord1xMedY5ukEhZYF-A/ , unless there is reason not to do so.

[szehon-ho]

Do you mean, an 'associated Iceberg table'?

[szehon-ho]

Why not just 'materialized'? (view seems redundant)

Also, in @JanKaul's proposal https://docs.google.com/document/d/1UnhldHhe3Grz8JBngwXPA6ZZord1xMedY5ukEhZYF-A/edit#heading=h.a0i0qwejiore it looks like it was a single field (materialization), and it was a MV if the field is not null?

[jackye1995]

I think the consensus around Iceberg property name is that . represents levels, and - represents phrases, so I think at least it should be named as something like materialized-view.enabled, materialized-view.storage-table, etc. based on my understanding. If there is already a consensus in the Google doc, I am okay with following that.

[szehon-ho]

Same comment, in https://docs.google.com/document/d/1UnhldHhe3Grz8JBngwXPA6ZZord1xMedY5ukEhZYF-A/edit#heading=h.r4ge4rk4hbp5 by @JanKaul, it was a struct called 'lineage' that has these fields. I think that was the proposal instead of ad-hoc table properties?

[szehon-ho]

For reference it was:
On Table Snapshot Metadata:
Struct 'lineage'

v1	Field Name	Description
required	refresh-version-id	Version id of the materialized view when the refresh operation was performed.
required	source-tables	A List of source-table records.

Struct 'source-table'

v1	Field Name	Description
required	uuid	Uuid of the source table
required	identifier	A full identifier record containing catalog, namespace and table-name fields.
required	snapshot-id	Snapshot id of the source table when the last refresh operation was performed.

[bennychow]

I don't think a flat list of views or tables is going to accurately represent the lineage of a MV.

You could have a MV whose SQL references the same table at different branches or timestamps.

In the Nessie catalog, the same view on different branches could refer to child tables at different snapshots.

[wmoustafa]

You could have a MV whose SQL references the same table at different branches or timestamps.

Do we want to support that in materialized views yet? This will start requiring understanding the SQL logic and reasoning about aliases, etc, which is a high expectation from the engine code and requires some standardization too. Further, what SQL dialects support this today?

[bennychow]

Most engines out there support the time travel clause so its pretty ubiquitous now.

We may not want to support the same table at different snapshots in the MV query tree right now. But I think using a struct for each record allows us to add properties in the future to disambiguate the tables (maybe sequence number, parent view, path to root or query tree depth).

I regularly see MVs on views that are 100 of non-distinct tables/views. It happens a lot when users build semantic layers out of views.

[wmoustafa]

100 of non-disitinct tables/views is fine as long as none of them has time travel. They will all share the same snapshot ID and one record can represent them all. I think the only way to truly distinguish is the alias in the query. I do not see why adding query tree depth helps (i.e., it can still lead to collisions). Sequence number does not work either because it has to be interpreted by all engines in the same way (i.e., same sequence number uniquely identifies the same occurrence of the table across all engines).

[bennychow]

I'm not sure I understand your alias idea. Can you elaborate with an example? Thanks

[jackye1995]

+1. I think supporting time travel is required given it is now a standard ANSI syntax that can be referenced in any MV SQL. Blocking it actually requires more effort to check for time travel query and then fail MV creation.

And when you travel by time, I think you need to fix the version it travels to by recording its snapshot ID. Time travel looks at the history entries to identify the snapshot to read. The entry might be removed due to setting of write.metadata.previous-versions-max. It might also be modified by accident. So at MV read time the reader should check if the specific version is still there and matches the recorded version in MV.

[szehon-ho]

I actually think that putting info like this on snapshot is a lot easier for time-travel, as moving between snapshots is what the time travel mechanism is. I understand having the right the lineage properties on snapshot is not critical for the integrity of data, but I feel it would be a great usability improvement if it was updated.

I think that would be hard to get in sync if we use table properties (or even table metadata). Look at the many problems we have just getting right schema to be read in some of these scenarios (rollback and time travel) #5591 (still not solved), #9131, #1508, etc, because it is stored as table metadata and not on snapshot.

[szehon-ho]

Also as @JanKaul pointed out, Trino already is modeling it as snapshot properties: https://github.com/trinodb/trino/blob/212455d3e1d393f58cbc395d2b9da47ed8f23dd8/plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java#L2915

[jackye1995]

I have a similar concern with @stevenzwu in the devlist:

With the agreed model of separate view and storage table, I am wondering if a separate materialized view spec page is needed.

I think it would be nice if the table related field changes are in the table spec, and view related field changes are in the view spec for the completeness of those specs.

But I also see the point that @wmoustafa made that:

I feel it is needed so the MV spec is not scattered across the table and view spec pages.

What if we keep this MV spec page, but instead of describing the fields here, we make these pointer to the corresponding table and view spec sections? So this MV spec page contains just intro, pointers, and examples?

[bennychow]

There's something wrong with this SQL. E2 doesn't have event_type in its project list.

But these are great examples of MVs on nested views.

[bennychow]

I don't think a flat list of views or tables is going to accurately represent the lineage of a MV.

You could have a MV whose SQL references the same table at different branches or timestamps.

In the Nessie catalog, the same view on different branches could refer to child tables at different snapshots.

[wmoustafa]

What if we keep this MV spec page, but instead of describing the fields here, we make these pointer to the corresponding table and view spec sections? So this MV spec page contains just intro, pointers, and examples?

@jackye1995 How about linking this from the table and view spec? We have a separate properties page for Spark configurations: https://iceberg.apache.org/docs/1.5.0/configuration/. I feel this format is more concise for MVs, yet can be referenced from the main pages.

[jackye1995]

We have a separate properties page for Spark configurations: https://iceberg.apache.org/docs/1.5.0/configuration/.

I see, if we conclude that we want to go with the properties approach then this should be fine! We can close that discussion at the devlist.

[manuzhang]

@wmoustafa how is this linked to #10043? Also, the doc [3] doesn't exist for me.

[wmoustafa]

@wmoustafa how is this linked to #10043?

This addresses the spec aspect of #10043 using the property model. More context and discussion in this dev list thread. The thread also discusses another model that introduces new top level metadata fields, along with the pros and cons of each approach.

Also, the doc [3] doesn't exist for me.

Thanks for pointing it out. I have fixed the link.