Handling NaN and INF values in EdgeBuilder and Stroker

[kobalicek]

At the moment EdgeBuilder cannot handle NaN values properly in the following cases:

• NaN is part of start/end point
• NaN is part of a quadratic/cubic curve that is being flattened

The thing is that we do not handle NaNs at BLPath level, because of the following reasons:

• Vertices can be transformed, so safe path values won't protect us from NaNs resulting from other computations
• The plan is to allow users passing their own buffers that are not allocated by Blend2D nor part of BLPath, so Blend2D can never trust user input or BLPath content for security reasons

The problem is not how to handle NaNs, it's more about the behavior when NaN is encountered at this level. The following options are available:

• Reset the whole EdgeBuffer content and return BL_ERROR_INVALID_GEOMETRY error
• Treat NaN as some safe value like [0, 0] and render a corrupted image

I would prefer the first option as corrupted image is probably not what users would want anyway, but if anyone has different ideas then I'm interested.

[Wiladams]

well, 'trust' is always a hard thing. I'd fail, and try to provide as much info as possible to find the error.

[Robert-M-Muench]

Option 1

[kobalicek]

To update this: There is some work that has been already merged and should solve basic issues with NaNs, but this is not thoroughly tested. Keeping open, we need fuzzing to make sure this works as expected.