SubFlow.asSources

[jroper]

I would like to work with a sub flow as a stream of sources. A use case for doing this is streaming a single big file into many small files. It's possible to do this today with prefixAndTail(0).map(_._2).concatSubstreams, however, the aesthetics of that are pretty bad, it's not obvious to the reader what is being achieved by that unless they are very familiar with the pattern already, and from a discovery point of view, when a developer thinks "I want to turn my sub flow into a flow of sources", and looks through the API to find a method that would help them do that, prefixAndTail does not stand out as the name of a method that would help them achieve that. I also can't find anything in the documentation that mentions this pattern of use for prefixAndTail.

Of course, SubFlow.to(Sink) does allow writing to multiple sinks, and hence multiple small files, but the problem with that is you lose the materialized values, so for example, if one of the sinks fail on completion, so that the file isn't saved, that signal is lost, and in most use cases, you need to know about that, and you will usually want to stop the parent stream in that case.

Here's some example code that demonstrates the use case:

val MaxFileSize = 1000000l

Flow[ByteString]
  .fold((ByteString.empty, 0l)){
    case ((last, count), bytes) =>
      (bytes, count + last.length)
  }.splitAfter {
    case (bytes, count) =>
      MaxFileSize - (count % MaxFileSize) >= bytes.length
  }.asSources
    .zipWithIndex
    .mapAsync(1) {
      case ((source, _), idx) =>
        source.to(FileIO.toPath(Paths.get(s"my-file-$idx")))
    }

[johanandren]

Looks like a nice API addition to me from a first look. Behaviour might not be obvious though, splitAfter is easy, but the other way of ending up with a SubFlow is groupBy. I guess that'd means it'd have to be a merge since it would deadlock on a new group before the previous cancels or hits max substreams.

Maybe some other more specific operator factory for this use case would make more sense and be easier to understand?