HDFS related questions

[zhougit86]

We are trying to introduce risingwave into our production enviroment. But before start, we want to understand 2 things

1, can we use hdfs as the statebackend?
2, can we dump result to aliyun maxcompute?

[lmatz]

can we use hdfs as the statebackend?

Yes, Risingwave suppors HDFS as the state backend, please check more details at:
https://www.risingwave.dev/docs/current/risingwave-kubernetes/#deploy-a-risingwave-instance

[lmatz]

can we dump result to aliyun maxcompute?

Do you mean Write data to OSS more precisely?

If you are referring to create sink XXX and sink output to OSS, Risingwave right now does not support that.
Please correct me if I understand you wrong, thanks!

[zhougit86]

yes, I mean the max computer sink. Does risingwave support custom sink like flink do?

The first step we might plan to build some etl job, they extract from kafka to aliyun odps. As these simple tasks can give us some confidence. The user defined sink might provide a very good start to us.

[StrikeW]

Right now we don't support sink to odps, and it is not trivial to add one. Maybe you can extract from kafka and sink to a kafka topic then ingest to odps via other tools. I think aliyun should support ingest to odps via kafka.

[lmatz]

If the target destination, e.g. OSS, is S3-compatible, we can consider supporting an S3 sink to avoid the need of writing a customized sink, just an early thought.

Wonder if this user-defined sink has any other requirements, or could you elaborate on more details of the dump result part?

[zhougit86]

https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sourcessinks/

this kind of source/sink is good enough

[zhougit86]

Furthermore, I read
'RisingWave offers not only streaming ETL and streaming analytics, but also online serving capabilities through its built-in batch query engine.' in https://www.risingwave.dev/docs/current/risingwave-flink-comparison/#typical-use-cases

Does this mean, risingwave could provide same ETL function with less performance cost than flink does?

[lmatz]

risingwave could provide same ETL function with less performance cost than flink does?

Yes, Risingwave is written in Rust(versus Java) and has a native storage engine that is optimized for stream processing.

but also online serving capabilities

By saying so, we want to emphasize that the results produced by Flink can only be queried when the results are output and stored into another storage system, e.g. Redis or Cassandra or any other DBs.

While in Risingwave, the results produced by Risingwave can be directly stored in Materialized Views. So you can directly query against Materialized Views to see your latest result with a consistency guarantee. No third-party system needed.

[zhougit86]

Understand, I have 2 more questions:

1. can we support java udf
2. can we specify psql address in play mode? as I want to run RW in a linux env and use psql in a mac env

[zhougit86]

Can some master provide some of the IM tools? like feishu or dingding, I might need some help in my POC process

[StrikeW]

Hi @zhougit86 , I sent a email to you, please check your mail.

[zhougit86]

Ok, I have contact you according to your suggestion.

[wcy-fdu]

Glad to hear that you plan to try RisingWave, may I ask about your HDFS configuration? For example, whether there is namenode HA, etc.

[zhougit86]

I will have further discussion later on this topic, As this is more "product" related case.