Another implementation of shuffle server

[zuston]

I want to share another implementation of shuffle server(https://github.com/zuston/uniffle-x), which is developed by rust. BTW HDFS will be supported in this ASAP.

And this project is to reduce the memory footprint and provide better stability. If someone is interested, welcome

Attension

This project is not used in production env, please use the stable uniffle's server.

And this is my first time to use rust to build program, feel free to point out any mistakes or tell me best practice.

[zuston]

cc @advancedxy I remember we talked about this few months ago, I think you will be interested.

[advancedxy]

Really nice to see a rust impl here. Some high level questions:

1. when would HDFS support be added? And how would that be supported, since I think we don't have a rust binding for HDFS?
2. Do you any performance/microbenchmark numbers now?

I think we can include this rust impl in the uniffle repo if you want to contribute it to uniffle repo, probably in 1.0 release or after the netty impl is finished.

[zuston]

1. when would HDFS support be added? And how would that be supported, since I think we don't have a rust binding for HDFS?

Apache Opendal is a good choice. But it only supports jvm way to communicate with hdfs by libhdfs.so .

And clickhouse’s libhdfs3 is another way to be interactive with hdfs by protocol directly. If you have other ideas or design, welcome to participate.

2. Do you any performance/microbenchmark numbers now?

Not yet. But compared with uniffle server of 100g terasort, the total time is the same.

I think we can include this rust impl in the uniffle repo if you want to contribute it to uniffle repo, probably in 1.0 release or after the netty impl is finished.

Absolutely yes.

[advancedxy]

Apache Opendal is a good choice. But it only supports jvm way to communicate with hdfs by libhdfs.so.

This might be a safe choice, I believe we may choose to support hdfs through OpenDAL first.

[zuston]

Oh yes. But kerberos is another problem, which has been tracked in opendal community

[connorlwilkes]

Interesting. Keeping an eye

[zhanglistar]

Is it just rewrite uniffle using Rust? Any improvements?

[zuston]

In the current status, the rewriting is the main goal

[pegasas]

I think this may target solving the GC problem in Shuffle Server? it seems lots of experiment shows that GC may be the major bottleneck of Uniffle.
Or any further steps can we take in Rust to do better?

[zuston]

I think this may target solving the GC problem in Shuffle Server? it seems a lot of experience shows that GC may be the major bottleneck of Uniffle.

Yes. But netty has been introduced in uniffle to solve GC(will be released in 0.8 version of uniffle). Rust is another try leveraging by the good design and the grpc.

Or any further steps can we take in Rust to do better?

Feel free to propose your ideas or problems.

BTW, after 0.8, I think it's time to merge this rust project into uniffle master branch, the PPMC will push this forward. I hope more people could join this

[zuston]

It's time to contribute back to community, but I hope we have enough people to be interested in

[zuston]

tracked by #1206