[feature] Performance Test Suite

[alecsg77]

Performance is the goal, measurement is the key.

We can collect information using BenchmarkDotNet
and with some adjustment in the internal abstraction
test functionality without network latency or other external dependencies.

In the future, we can integrate the performance test in the main release pipeline, to verify improvements or criticalities.

Here below, an example of a report created using the example code in this draft pull request.
Currently, only the public indexers can be tested, and only the test action is checked.
During the warmup, all the remote request are cached in memory, to avoid any network interaction during the real test.

// * Summary *

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19042
Intel Core i7-7660U CPU 2.50GHz (Kaby Lake), 1 CPU, 4 logical and 2 physical cores
.NET Core SDK=5.0.202
[Host] : .NET Core 5.0.5 (CoreCLR 5.0.521.16609, CoreFX 5.0.521.16609), X64 RyuJIT
ShortRun : .NET Core 5.0.5 (CoreCLR 5.0.521.16609, CoreFX 5.0.521.16609), X64 RyuJIT

Job=ShortRun IterationCount=3 LaunchCount=1
WarmupCount=3

Method	indexer	Mean	Error	StdDev	Median	Gen 0	Gen 1	Gen 2	Allocated
TestIndexer	1337x	92,205.233 μs	81,074.148 μs	4,443.9476 μs	93,457.900 μs	6000.0000	1000.0000	-	26918080 B
TestIndexer	7torrents	17,271.267 μs	63,787.394 μs	3,496.4023 μs	18,621.400 μs	-	-	-	3671504 B
TestIndexer	AniLibria	11,324.167 μs	35,253.614 μs	1,932.3696 μs	12,065.500 μs	-	-	-	2804136 B
TestIndexer	Animedia	96,679.000 μs	75,049.696 μs	4,113.7271 μs	94,772.500 μs	5000.0000	1000.0000	-	33357696 B
TestIndexer	acgrip	27,605.667 μs	38,625.113 μs	2,117.1728 μs	28,504.300 μs	-	-	-	4306864 B
TestIndexer	acgsou	42,090.300 μs	62,070.177 μs	3,402.2758 μs	40,497.500 μs	1000.0000	-	-	7470480 B
TestIndexer	aniRena	112,347.300 μs	140,956.713 μs	7,726.3131 μs	110,994.700 μs	3000.0000	1000.0000	-	17922752 B
TestIndexer	anidex	NA	NA	NA	NA	-	-	-	-
TestIndexer	animeclipse	162,929.100 μs	237,655.551 μs	13,026.7027 μs	169,257.300 μs	9000.0000	-	-	31039784 B
TestIndexer	animetosho	8,355.933 μs	57,709.473 μs	3,163.2509 μs	8,475.700 μs	-	-	-	1497344 B
TestIndexer	anisource	61,180.333 μs	54,601.765 μs	2,992.9070 μs	61,484.400 μs	3000.0000	1000.0000	-	17486896 B
TestIndexer	audiobookbay	54,954.100 μs	107,112.205 μs	5,871.1814 μs	52,828.200 μs	2000.0000	1000.0000	-	12550912 B
TestIndexer	bigfangroup	102,982.967 μs	72,888.886 μs	3,995.2858 μs	104,919.000 μs	8000.0000	1000.0000	-	22490952 B
TestIndexer	bitru	91,231.033 μs	13,446.842 μs	737.0668 μs	91,638.300 μs	3000.0000	1000.0000	-	16072312 B
TestIndexer	bt4g	8,561.667 μs	39,059.984 μs	2,141.0095 μs	7,937.500 μs	-	-	-	2610184 B
TestIndexer	btdb	119,334.400 μs	271,089.927 μs	14,859.3536 μs	112,184.600 μs	3000.0000	1000.0000	-	24789800 B
TestIndexer	btdigg	1,930.100 μs	2,111.772 μs	115.7534 μs	1,873.100 μs	-	-	-	570656 B
TestIndexer	btetree	48,698.333 μs	142,917.255 μs	7,833.7770 μs	47,151.900 μs	1000.0000	-	-	8015672 B
TestIndexer	btsow	19,127.200 μs	46,446.341 μs	2,545.8807 μs	20,460.100 μs	1000.0000	-	-	5825104 B
TestIndexer	byrutor	13,946.200 μs	36,183.136 μs	1,983.3198 μs	14,126.700 μs	-	-	-	3256000 B
TestIndexer	cilipro	12,553.267 μs	45,207.226 μs	2,477.9606 μs	12,544.900 μs	1000.0000	-	-	3637672 B
TestIndexer	cinecalidad	9,475.600 μs	7,586.451 μs	415.8390 μs	9,666.400 μs	1000.0000	-	-	2943720 B
TestIndexer	comicat	74,721.833 μs	34,593.166 μs	1,896.1682 μs	73,691.500 μs	2000.0000	1000.0000	-	18709632 B
TestIndexer	concen	40,986.233 μs	28,938.129 μs	1,586.1965 μs	40,589.300 μs	2000.0000	1000.0000	-	9994136 B
TestIndexer	cpasbienclone	189,087.133 μs	384,785.948 μs	21,091.4162 μs	198,203.600 μs	37000.0000	1000.0000	-	83020144 B
TestIndexer	divxtotal	11,667.867 μs	30,496.506 μs	1,671.6164 μs	11,793.000 μs	-	-	-	3263992 B
TestIndexer	dmhy	NA	NA	NA	NA	-	-	-	-
TestIndexer	ehentai	57,977.900 μs	167,827.873 μs	9,199.2120 μs	57,652.700 μs	2000.0000	1000.0000	-	13564520 B
TestIndexer	emtrek	48,157.633 μs	47,545.334 μs	2,606.1202 μs	48,941.300 μs	1000.0000	-	-	9673456 B
TestIndexer	epublibre	NA	NA	NA	NA	-	-	-	-
TestIndexer	erai-raws	16,705.233 μs	54,397.041 μs	2,981.6854 μs	15,439.400 μs	1000.0000	-	-	4784112 B
TestIndexer	ettv	8,507.933 μs	22,272.441 μs	1,220.8276 μs	7,967.500 μs	-	-	-	3206016 B
TestIndexer	extratorrent-cd	63,794.367 μs	58,557.514 μs	3,209.7350 μs	62,089.400 μs	5000.0000	-	-	19234952 B
TestIndexer	extratorrent-it	111,238.333 μs	555,761.255 μs	30,463.1497 μs	96,904.300 μs	4000.0000	1000.0000	-	26734576 B
TestIndexer	exttorrents	631.033 μs	3,275.068 μs	179.5175 μs	541.600 μs	-	-	-	105760 B
TestIndexer	eztv	NA	NA	NA	NA	-	-	-	-
TestIndexer	filebase	64,424.233 μs	29,960.004 μs	1,642.2089 μs	63,755.500 μs	6000.0000	1000.0000	-	21641008 B
TestIndexer	firebit	94,768.067 μs	212,268.523 μs	11,635.1540 μs	96,811.400 μs	11000.0000	1000.0000	-	30131272 B
TestIndexer	focusx	40,005.433 μs	183,607.302 μs	10,064.1358 μs	45,573.400 μs	1000.0000	-	-	4933128 B
TestIndexer	frozenlayer	37,913.400 μs	27,042.577 μs	1,482.2949 μs	37,074.200 μs	4000.0000	1000.0000	-	13028192 B
TestIndexer	gamestorrents	65,107.633 μs	72,279.109 μs	3,961.8619 μs	63,046.000 μs	4000.0000	1000.0000	-	19057984 B
TestIndexer	gktorrent	147,556.767 μs	66,005.935 μs	3,618.0080 μs	148,163.100 μs	13000.0000	1000.0000	-	40398176 B
TestIndexer	glodls	129,498.000 μs	711,580.046 μs	39,004.1034 μs	117,033.800 μs	17000.0000	1000.0000	-	41083432 B
TestIndexer	gtorrent	36,293.667 μs	49,432.188 μs	2,709.5450 μs	37,568.800 μs	1000.0000	-	-	11095888 B
TestIndexer	gtorrentpro	17,437.167 μs	74,638.876 μs	4,091.2086 μs	19,545.900 μs	1000.0000	-	-	5921152 B
TestIndexer	hdhouse	17,161.900 μs	50,034.526 μs	2,742.5612 μs	18,690.000 μs	1000.0000	-	-	5885456 B
TestIndexer	ibit	23,654.167 μs	31,530.508 μs	1,728.2935 μs	24,408.000 μs	1000.0000	-	-	6897584 B
TestIndexer	idope	30,759.433 μs	41,621.786 μs	2,281.4305 μs	29,634.100 μs	1000.0000	-	-	8194064 B
TestIndexer	ilcorsaronero	NA	NA	NA	NA	-	-	-	-
TestIndexer	internetarchive	246,524.233 μs	310,333.308 μs	17,010.4158 μs	255,606.000 μs	35000.0000	1000.0000	-	91675272 B
TestIndexer	isohunt2	2,002.667 μs	12,438.333 μs	681.7870 μs	1,816.500 μs	-	-	-	413200 B
TestIndexer	itorrent	26,802.667 μs	37,801.885 μs	2,072.0489 μs	25,637.200 μs	1000.0000	-	-	7181496 B
TestIndexer	kickasstorrents-to	46,710.633 μs	59,762.982 μs	3,275.8107 μs	45,266.300 μs	2000.0000	1000.0000	-	9284944 B
TestIndexer	kickasstorrents-ws	105,459.500 μs	348,840.799 μs	19,121.1413 μs	96,416.500 μs	2000.0000	1000.0000	-	15489688 B
TestIndexer	legittorrents	70,849.767 μs	94,613.964 μs	5,186.1106 μs	68,655.600 μs	2000.0000	1000.0000	-	12582824 B
TestIndexer	lepornoinfo	21,690.267 μs	50,741.704 μs	2,781.3240 μs	21,894.400 μs	1000.0000	-	-	3913264 B
TestIndexer	limetorrents	50,686.033 μs	73,721.361 μs	4,040.9166 μs	52,383.900 μs	1000.0000	-	-	7715984 B
TestIndexer	linuxtracker	164,844.200 μs	179,179.571 μs	9,821.4369 μs	168,102.500 μs	11000.0000	4000.0000	-	27945816 B
TestIndexer	mactorrents	97,678.300 μs	178,974.761 μs	9,810.2106 μs	92,779.800 μs	3000.0000	1000.0000	-	16711944 B
TestIndexer	magnet4you	54,248.967 μs	33,792.675 μs	1,852.2905 μs	54,469.700 μs	2000.0000	1000.0000	-	9493728 B
TestIndexer	mejortorrent	122,158.367 μs	218,800.041 μs	11,993.1686 μs	120,932.000 μs	4000.0000	1000.0000	-	18481624 B
TestIndexer	mixtapetorrent	24,007.467 μs	64,886.207 μs	3,556.6319 μs	24,151.300 μs	1000.0000	-	-	5490824 B
TestIndexer	montorrent	25,447.467 μs	62,909.144 μs	3,448.2625 μs	26,655.600 μs	-	-	-	3375280 B
TestIndexer	moviesdvdr	NA	NA	NA	NA	-	-	-	-
TestIndexer	movietorrent	29,482.333 μs	51,186.817 μs	2,805.7221 μs	30,217.900 μs	1000.0000	-	-	5808736 B
TestIndexer	mypornclub	88,792.233 μs	426,071.104 μs	23,354.3949 μs	88,039.700 μs	3000.0000	1000.0000	-	1865502 B
TestIndexer	newpct	41,059.700 μs	145,619.912 μs	7,981.9187 μs	37,481.800 μs	1000.0000	-	-	7568880 B
TestIndexer	newstudio	289,112.833 μs	417,895.589 μs	22,906.2673 μs	295,447.400 μs	5000.0000	1000.0000	-	40578840 B
TestIndexer	nitro	299,151.300 μs	153,914.171 μs	8,436.5551 μs	297,307.100 μs	29000.0000	1000.0000	-	79086824 B
TestIndexer	nntt	96,409.333 μs	126,887.363 μs	6,955.1245 μs	93,611.900 μs	3000.0000	-	-	16639392 B
TestIndexer	noname-club	86,783.367 μs	95,494.658 μs	5,234.3844 μs	84,949.200 μs	8000.0000	-	-	25581968 B
TestIndexer	nyaa-pantsu	108,672.533 μs	73,983.440 μs	4,055.2820 μs	107,820.900 μs	6000.0000	1000.0000	-	22520296 B
TestIndexer	nyaasi	70,729.600 μs	112,984.517 μs	6,193.0626 μs	67,172.800 μs	2000.0000	1000.0000	-	13273112 B
TestIndexer	oncesearch	98,756.867 μs	222,166.590 μs	12,177.7005 μs	92,098.500 μs	2000.0000	1000.0000	-	12668080 B
TestIndexer	onejav	39,961.867 μs	99,465.580 μs	5,452.0441 μs	38,241.400 μs	1000.0000	-	-	7398352 B
TestIndexer	oxtorrent	34,000.267 μs	28,645.895 μs	1,570.1781 μs	33,904.800 μs	1000.0000	-	-	9368896 B
TestIndexer	parnuxi	150,389.600 μs	103,889.991 μs	5,694.5610 μs	148,670.700 μs	17000.0000	-	-	42107440 B
TestIndexer	pctorrent	128,510.000 μs	63,178.088 μs	3,463.0042 μs	130,234.900 μs	2000.0000	1000.0000	-	12683760 B
TestIndexer	piratbit	16,778.067 μs	47,302.695 μs	2,592.8203 μs	18,234.400 μs	1000.0000	-	-	4676800 B
TestIndexer	pirateiro	97,328.833 μs	171,928.426 μs	9,423.9772 μs	94,373.800 μs	2000.0000	1000.0000	-	11307096 B
TestIndexer	pornforall	NA	NA	NA	NA	-	-	-	-
TestIndexer	pornleech	48,871.000 μs	80,492.255 μs	4,412.0521 μs	49,632.400 μs	1000.0000	-	-	8681696 B
TestIndexer	pornolive	68,380.500 μs	38,566.189 μs	2,113.9429 μs	67,637.200 μs	2000.0000	1000.0000	-	11569792 B
TestIndexer	pornorip	25,332.367 μs	62,242.657 μs	3,411.7301 μs	24,624.800 μs	-	-	-	4508376 B
TestIndexer	pornotor	45,873.700 μs	57,918.773 μs	3,174.7234 μs	45,798.200 μs	2000.0000	1000.0000	-	8892848 B
TestIndexer	proporno	23,508.600 μs	68,329.383 μs	3,745.3641 μs	24,646.700 μs	1000.0000	-	-	5761856 B
TestIndexer	rapidzona	33,510.367 μs	139,523.806 μs	7,647.7706 μs	31,743.600 μs	-	-	-	4071624 B
TestIndexer	rarbg	24,014.033 μs	41,412.393 μs	2,269.9530 μs	23,759.800 μs	-	-	-	4485416 B
TestIndexer	rintornet	4,034.900 μs	10,127.646 μs	555.1304 μs	3,926.600 μs	-	-	-	1070720 B
TestIndexer	rus-media	32,101.000 μs	56,191.150 μs	3,080.0265 μs	33,379.000 μs	1000.0000	-	-	4788560 B
TestIndexer	rutor	114,830.367 μs	94,229.225 μs	5,165.0218 μs	116,932.200 μs	10000.0000	1000.0000	-	31013312 B
TestIndexer	rutracker-ru	3,553,106.167 μs	482,602.705 μs	26,453.0827 μs	3,563,829.500 μs	68000.0000	4000.0000	-	157342184 B
TestIndexer	sexypics	388,861.767 μs	1,282,626.572 μs	70,305.0905 μs	363,563.600 μs	8000.0000	1000.0000	-	26980888 B
TestIndexer	shokweb	59,063.333 μs	45,417.942 μs	2,489.5107 μs	59,243.800 μs	2000.0000	1000.0000	-	10598840 B
TestIndexer	showrss	75,567.033 μs	139,620.720 μs	7,653.0828 μs	72,026.500 μs	4000.0000	1000.0000	-	17408528 B
TestIndexer	skytorrents-to	5,021.267 μs	40,358.048 μs	2,212.1608 μs	3,794.300 μs	-	-	-	1499784 B
TestIndexer	solidtorrents	NA	NA	NA	NA	-	-	-	-
TestIndexer	sosulki	2,311.067 μs	11,023.251 μs	604.2216 μs	2,388.900 μs	-	-	-	435816 B
TestIndexer	subsplease	37,192.233 μs	45,246.228 μs	2,480.0984 μs	36,222.400 μs	1000.0000	-	-	4915760 B
TestIndexer	sukebei-pantsu	5,076.233 μs	10,609.103 μs	581.5207 μs	4,905.900 μs	-	-	-	958680 B
TestIndexer	sukebeinyaasi	67,130.467 μs	109,587.315 μs	6,006.8505 μs	65,416.200 μs	3000.0000	1000.0000	-	13685760 B
TestIndexer	thepiratebay	68,784.633 μs	73,724.950 μs	4,041.1133 μs	68,987.300 μs	2000.0000	-	-	11260000 B
TestIndexer	tntfork	1,106.367 μs	546.878 μs	29.9762 μs	1,103.900 μs	-	-	-	180296 B
TestIndexer	tokyotosho	412,351.800 μs	1,021,119.888 μs	55,971.0267 μs	427,922.400 μs	20000.0000	1000.0000	-	51613760 B
TestIndexer	torlock	359,739.900 μs	256,535.530 μs	14,061.5780 μs	362,789.400 μs	2000.0000	1000.0000	-	11864200 B
TestIndexer	toros	229,633.700 μs	590,849.117 μs	32,386.4338 μs	213,396.600 μs	5000.0000	1000.0000	-	20204896 B
TestIndexer	torrent-paradise-ml	161,801.933 μs	677,805.106 μs	37,152.7850 μs	182,486.300 μs	2000.0000	-	-	14399400 B
TestIndexer	torrent-pirat	2,698.333 μs	4,106.690 μs	225.1015 μs	2,736.200 μs	-	-	-	528432 B
TestIndexer	torrent4you	34,874.533 μs	9,895.141 μs	542.3860 μs	35,032.500 μs	1000.0000	-	-	7361720 B
TestIndexer	torrent9	122,671.333 μs	250,553.162 μs	13,733.6643 μs	125,571.800 μs	5000.0000	1000.0000	-	19788144 B
TestIndexer	torrent9clone	127,247.800 μs	275,838.941 μs	15,119.6632 μs	123,382.500 μs	13000.0000	1000.0000	-	32211800 B
TestIndexer	torrentdownloads	102,225.667 μs	68,065.473 μs	3,730.8983 μs	104,206.900 μs	11000.0000	-	-	27387064 B
TestIndexer	torrentfunk	305,521.267 μs	215,951.013 μs	11,837.0037 μs	309,651.100 μs	29000.0000	3000.0000	-	78703464 B
TestIndexer	torrentgalaxy	66,931.533 μs	151,543.879 μs	8,306.6314 μs	66,023.200 μs	2000.0000	1000.0000	-	9301840 B
TestIndexer	torrentkitty	102,377.067 μs	228,844.991 μs	12,543.7662 μs	95,194.000 μs	3000.0000	1000.0000	-	17217432 B
TestIndexer	torrentmafya	NA	NA	NA	NA	-	-	-	-
TestIndexer	torrentmax	3,014.900 μs	10,078.656 μs	552.4451 μs	2,968.000 μs	-	-	-	719656 B
TestIndexer	torrentoyunindir	97,974.500 μs	280,420.913 μs	15,370.8165 μs	92,105.200 μs	5000.0000	-	-	19322328 B
TestIndexer	torrentparadise	9,034.233 μs	69,903.773 μs	3,831.6617 μs	7,288.400 μs	-	-	-	2348448 B
TestIndexer	torrentproject	90,794.233 μs	490,783.985 μs	26,901.5263 μs	104,428.200 μs	3000.0000	1000.0000	-	12103752 B
TestIndexer	torrentproject2	NA	NA	NA	NA	-	-	-	-
TestIndexer	torrentqq	91,334.000 μs	169,124.877 μs	9,270.3052 μs	93,451.600 μs	5000.0000	1000.0000	-	20694344 B
TestIndexer	torrentscsv	NA	NA	NA	NA	-	-	-	-
TestIndexer	torrentsir	6.184 μs	17.417 μs	0.9547 μs	6.449 μs	1.2283	-	-	2576 B
TestIndexer	torrentv	104,257.233 μs	279,505.267 μs	15,320.6268 μs	98,041.600 μs	5000.0000	1000.0000	-	20722984 B
TestIndexer	torrentview	27,762.167 μs	34,249.192 μs	1,877.3138 μs	26,680.800 μs	1000.0000	-	-	5820640 B
TestIndexer	torrentwhiz	NA	NA	NA	NA	-	-	-	-
TestIndexer	trupornolabs	NA	NA	NA	NA	-	-	-	-
TestIndexer	underverse	178,967.700 μs	252,543.864 μs	13,842.7814 μs	185,483.100 μs	15000.0000	1000.0000	-	43159288 B
TestIndexer	uniondht	169,865.233 μs	100,054.070 μs	5,484.3012 μs	170,682.400 μs	4000.0000	1000.0000	-	19289568 B
TestIndexer	vsthouse	325,495.967 μs	1,642,545.871 μs	90,033.4817 μs	337,921.500 μs	4000.0000	1000.0000	-	17769432 B
TestIndexer	vsttorrents	79,932.100 μs	197,008.135 μs	10,798.6807 μs	82,102.100 μs	2000.0000	1000.0000	-	14765864 B
TestIndexer	xxxadulttorrent	11,889.567 μs	32,965.258 μs	1,806.9370 μs	12,393.000 μs	-	-	-	3437376 B
TestIndexer	xxxtor	41,936.900 μs	26,985.241 μs	1,479.1521 μs	41,688.300 μs	4000.0000	1000.0000	-	19309408 B
TestIndexer	xxxtorrents	43,768.033 μs	13,477.572 μs	738.7512 μs	43,494.700 μs	6000.0000	1000.0000	-	22647864 B
TestIndexer	yourbittorrent	48,360.800 μs	63,251.337 μs	3,467.0192 μs	46,471.600 μs	2000.0000	1000.0000	-	14422024 B
TestIndexer	yts	27,220.067 μs	18,781.378 μs	1,029.4707 μs	26,955.500 μs	3000.0000	1000.0000	-	10630368 B
TestIndexer	zetorrents	3,424.600 μs	23,446.525 μs	1,285.1832 μs	2,712.700 μs	-	-	-	1586536 B
TestIndexer	zooqle	57,945.567 μs	27,878.365 μs	1,528.1073 μs	57,138.400 μs	11000.0000	1000.0000	-	28351792 B

[ilike2burnthing]

Apologies for my ignorance, but what is being measured here? The time taken, and memory needed, for an indexer to process a prefetched results page?

This would be useful when making significant changes to the indexer, or Jackett as a whole, to see if there's any significant impact, either positive or negative?

[alecsg77]

Apologies for my ignorance, but what is being measured here? The time taken, and memory needed, for an indexer to process a prefetched results page?

Yes, correct, or better, this is my goal.

This would be useful when making significant changes to the indexer, or Jackett as a whole, to see if there's any significant impact, either positive or negative?

Yes.

[ilike2burnthing]

OK, sounds good, thanks.

@garfield69

[dreulavelle]

Did development die on this?

[ilike2burnthing]

Never really go off the ground. It was the proposal of a contributor. Looking at the activity of the user, they took almost a year break from GH after submitting this.