HTTP as a source?

[tomtom215]

Hello,

Long time lurker on this awesome project :)

I was curious about the possibility of having an HTTP(S) source because that would allow us to avoid creating some middleware to convert incoming HTTP json messages via a webhook endpoint to AMPQ messages to be put in a queue or a direct DB insert function.

Ideally, we would love if RisingWave offered this connector in the same way that Apache SeaTunnel does here https://seatunnel.apache.org/docs/2.3.2/connector-v2/source/Http

[fuyufjh]

@tomtom215 Thanks for your interest!

I just read the document and I am wondering how Apache SeaTunnel avoids duplication? For example, if the file is hosted in a simple Nginx server and RisingWave might pulls it for multiple times, and every time all the data will be sent to downstream.

For comparison, RisingWave's S3 file source will save a list of file_name:read_offset in its internal state to ensure no duplicated reads and exactly-once delivery semantics. But for the HTTP Source, it seems that doesn't fit because the given file might be completely overwritten, right?

---

On the other hand, we had once discussed an idea of "POST HTTP Source": it accepts POST from users and parse and inject rows in that request. What do think of this idea? Can it fulfill your use cases?

[tomtom215]

Hey @fuyufjh thanks for the response!

As far as I can tell from the SeaTunnel documentation, they currently do not support exactly-once delivery for the HTTP source connector, but it seems that they avoid duplication from their file or kafka source by saving a read & split offset as a "StateSnapshot"

I haven't found any roadmaps or discussions about including that for SeaTunnel's HTTP source yet, but I would assume it may be possible via something like a message hash + timestamp stored as a StateSnapshot, but that is just a quick idea from the top of my head.

For our use-case, I believe it would align directly with your idea of POST HTTP source that accepts POST messages and injects/inserts that data to a SINK destination.

For more context, we have legacy systems and applications that push DB updates via JSON messages. It's close to CDC or pub/sub, but since we don't have influence over the source systems to setup something like Debezium that helps with handling the integration with other destinations, we are looking for something like an HTTP source connector -> RisingWave -> DataWarehouse or Message Queue destination

[neverchanje]

Hi @tomtom215 To be honest, I am still unclear about your requirement.

we have legacy systems and applications that push DB updates via JSON messages

If so, why don't you use Kafka in the middle and rewrite the JSON messages into the Debezium JSON format?

RisingWave can do 'RisingWave -> DataWarehouse' if the data warehouse sink is supported. So the blocker only is the CDC ingestion. It's not recommended to ingest CDC through a HTTP source, which does not provide the persistence to ensure exactly-once semantic. The ideal architecture typically consists of a Kafka so that RisingWave won't lose data during the consumption. Another option is to ingest CDC directly from the upstream database, e.g., MySQL https://docs.risingwave.com/docs/current/ingest-from-mysql-cdc/

[tomtom215]

Apologies if I wasn't able to write it out clearly before. I guess a better way of saying it is we have systems that generate server sent events by means of HTTP POSTs with either XML or JSON payloads and we would like to insert those into a SQL DB with guaranteed delivery. Since HTTP sources can't guarantee delivery on it's own, I thought Risingwave might be a good middleware for this use case. Ultimately we were trying to avoid added Kafka and Debezium complexity into the stack for this which is why I was looking in to Risingwave.

If it helps to understand it a bit better, we have since found a couple projects that have almost the exact functionality I was trying to describe here:

• https://www.benthos.dev/docs/components/inputs/http_server
• https://www.fluvio.io/docs/tutorials/data-pipeline/

[neverchanje]

Thanks for elaborating the information! I suppose what you are looking for is a combination of message queue and streaming engine. There is no need for Debezium in this case, since using MySQL as an event store is typically an overkill.

Right now RisingWave does not provide this capability but we are willing to explore the possibility. A possible solution in this case is through websocket.

                 Websocket Source 
Your Websocket Server ------> RisingWave

RisingWave is by its nature not suitable for request&reply ingestion like a MySQL or Postgres. Websocket thus is a better fit as it allows RisingWave to async pull the events.

[neverchanje]

I am not sure if your HTTP server can expose the Websocket to allow downstreams to consume?