Scale Constraints

[ryanmolden]

Problem Description

I work on the Visual Studio performance and reliability team. We have internal memory dumps that we automatically analyze looking to attribute large memory usage inside VS. In a few dumps I have recently looked at FastReport is rooting/responsible for DataColumnCollection objects that in aggregate can reach over a gig in size. In some dumps it is many (say 36) each holding reasonably sized (say 30-36 MB) DataColumnCollections. In other dumps I have seen 1-2 each holding DataColumnCollections whose size is > 1 GB.

DataColumnCollection is unfortunately not built to scale as it internally holds items in an ArrayList. Using segmented or non-contiguous data structures will be much more friendly in terms of address space when dealing with very large amounts of items.

An example of a segmented collection can be seen here: https://github.com/dotnet/roslyn/blob/main/src/Dependencies/Collections/SegmentedArray%601.cs

To Reproduce
Unclear as these reports come from dump taken due to large memory usage. The GC root chains of these items tend to look like this, if it helps any:

        <snip>
        ->  000002e6fa9cb2b0 FastReport.Controls.DataTreeView
        ->  000002e6fa9d34a8 System.Windows.Forms.TreeNodeCollection
        ->  000002e6fa9cb690 System.Windows.Forms.TreeNode
        ->  000002e6fa9f60c0 System.Windows.Forms.TreeNode[]
        ->  000002e6fa9f6168 System.Windows.Forms.TreeNode
        ->  000002e6fa9f6348 System.Windows.Forms.TreeNode[]
        ->  000002e6fa9f9e38 System.Windows.Forms.TreeNode
        ->  000002e6fa8f1fb0 FastReport.Data.TableDataSource
        ->  000002e6fa8f2130 System.Collections.ArrayList
        ->  000002e6d0d71fe0 System.Object[]
        ->  000002e6fc99d6f8 System.Data.DataRow
        ->  000002e6fc991b80 System.Data.DataColumnCollection

Expected behavior
Perhaps the large memory usage is unavoidable to express the users request, but thinking about ways to scale up to handle larger and larger requests that don't just add more and more items to a DataColumnCollection would be nice.

In large scale situations like this using contiguous datastructures is non-optimal as it requires very large LOH allocations, those stick around for a long time even if GC eligible and tend to bloat the overall memory usage of VS.

[Alekarfes]

Greetings! Without examples it's quite hard to figure out where the problem lies, could you please provide some?