New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](memory) Fix prune all LRU Cache based on number #34601

Merged

yiguolei merged 2 commits into apache:master from xinyiZzz:20240509_fix_cache_gc

May 11, 2024

Contributor

xinyiZzz commented May 9, 2024

Proposed changes

For NUMBER-based LRU Cache, if usage greater than CACHE_MIN_FREE_NUMBER allows prune all.

In the past, mem_consumption was greater than CACHE_MIN_FREE_SIZE to prune all, this will cause prune be delayed.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot commented May 9, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

Contributor

github-actions bot commented May 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

a4af2f0

xinyiZzz force-pushed the 20240509_fix_cache_gc branch from c5c3a4f to a4af2f0 Compare

May 10, 2024 05:44

Contributor

github-actions bot commented May 10, 2024

clang-tidy review says "All clean, LGTM! 👍"

Contributor Author

xinyiZzz commented May 10, 2024

run buildall

1 similar comment

Contributor Author

xinyiZzz commented May 10, 2024

run buildall

Contributor

github-actions bot commented May 10, 2024

clang-tidy review says "All clean, LGTM! 👍"

Contributor Author

xinyiZzz commented May 10, 2024

run buildall

xinyiZzz force-pushed the 20240509_fix_cache_gc branch from 5669467 to a4af2f0 Compare

May 10, 2024 06:56

Contributor

github-actions bot commented May 10, 2024

clang-tidy review says "All clean, LGTM! 👍"

doris-robot commented May 10, 2024

TeamCity be ut coverage result:
Function Coverage: 35.68% (8982/25172)
Line Coverage: 27.32% (74194/271529)
Region Coverage: 26.57% (38358/144392)
Branch Coverage: 23.39% (19563/83650)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5669467105de4f422583fa449b1b4389479f8fb7_5669467105de4f422583fa449b1b4389479f8fb7/report/index.html


          Merge branch 'master' into 20240509_fix_cache_gc

f205fc8

Contributor Author

xinyiZzz commented May 11, 2024

run buildall

Contributor

github-actions bot commented May 11, 2024

clang-tidy review says "All clean, LGTM! 👍"

doris-robot commented May 11, 2024

TPC-H: Total hot run time: 41264 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f205fc8b380ea71495a487af1c85f64bd6e7d2e8, data reload: false

------ Round 1 ----------------------------------
q1	17604	4444	4249	4249
q2	2017	196	191	191
q3	10479	1244	1217	1217
q4	10205	741	807	741
q5	7490	2726	2730	2726
q6	222	144	133	133
q7	1035	589	584	584
q8	9229	2194	2134	2134
q9	9424	6771	6758	6758
q10	9292	3893	3913	3893
q11	453	248	253	248
q12	500	229	226	226
q13	18774	3218	3072	3072
q14	259	217	216	216
q15	539	471	478	471
q16	509	409	402	402
q17	985	713	790	713
q18	8444	7828	7879	7828
q19	2866	1593	1556	1556
q20	648	320	316	316
q21	5622	3306	4196	3306
q22	366	291	284	284
Total cold run time: 116962 ms
Total hot run time: 41264 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4504	4429	4443	4429
q2	368	277	280	277
q3	3168	2946	2854	2854
q4	1879	1572	1587	1572
q5	5504	5523	5524	5523
q6	217	129	128	128
q7	2311	1981	1997	1981
q8	3267	3443	3399	3399
q9	8677	8761	8719	8719
q10	3959	3789	3877	3789
q11	601	490	494	490
q12	809	617	615	615
q13	17100	3138	3237	3138
q14	311	280	281	280
q15	516	490	479	479
q16	476	431	413	413
q17	1802	1514	1528	1514
q18	7833	7571	7466	7466
q19	1675	1563	1556	1556
q20	1978	1787	1757	1757
q21	8407	4956	4753	4753
q22	622	526	478	478
Total cold run time: 75984 ms
Total hot run time: 55610 ms

doris-robot commented May 11, 2024

TPC-DS: Total hot run time: 187647 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f205fc8b380ea71495a487af1c85f64bd6e7d2e8, data reload: false

query1	912	396	342	342
query2	6467	2294	2297	2294
query3	6646	220	222	220
query4	23078	21468	21391	21391
query5	4113	420	408	408
query6	256	169	178	169
query7	4584	303	289	289
query8	241	204	198	198
query9	8660	2474	2461	2461
query10	434	247	269	247
query11	14758	14231	14386	14231
query12	147	90	94	90
query13	1678	377	389	377
query14	9894	8500	7703	7703
query15	254	167	170	167
query16	8057	272	274	272
query17	1719	619	531	531
query18	2013	273	272	272
query19	237	146	153	146
query20	91	85	85	85
query21	194	130	129	129
query22	5080	4884	4862	4862
query23	34301	33461	33566	33461
query24	9025	2877	2864	2864
query25	568	364	357	357
query26	694	156	151	151
query27	2158	326	328	326
query28	5749	2080	2073	2073
query29	880	599	597	597
query30	243	156	157	156
query31	994	763	742	742
query32	90	55	54	54
query33	605	256	254	254
query34	898	483	495	483
query35	766	713	683	683
query36	1040	919	900	900
query37	103	66	68	66
query38	2909	2788	2770	2770
query39	1628	1684	1561	1561
query40	197	128	127	127
query41	43	38	40	38
query42	104	100	99	99
query43	576	526	558	526
query44	1120	735	748	735
query45	271	257	245	245
query46	1084	708	724	708
query47	1950	1868	1877	1868
query48	384	306	293	293
query49	843	408	390	390
query50	773	404	389	389
query51	6950	6702	6863	6702
query52	101	91	93	91
query53	350	288	277	277
query54	738	439	442	439
query55	73	76	74	74
query56	240	219	220	219
query57	1217	1158	1176	1158
query58	213	194	197	194
query59	3514	3131	3088	3088
query60	245	229	232	229
query61	88	85	85	85
query62	634	478	461	461
query63	307	285	288	285
query64	8429	7382	7451	7382
query65	3161	3069	3078	3069
query66	790	340	343	340
query67	15405	15118	14999	14999
query68	4593	541	537	537
query69	498	307	305	305
query70	1165	1141	1131	1131
query71	381	269	266	266
query72	7799	2552	2343	2343
query73	699	337	338	337
query74	6547	6107	6139	6107
query75	3295	2640	2627	2627
query76	2610	950	952	950
query77	560	264	262	262
query78	10597	10159	10317	10159
query79	1651	526	528	526
query80	877	498	441	441
query81	522	213	220	213
query82	858	93	96	93
query83	233	163	164	163
query84	245	86	86	86
query85	1121	271	259	259
query86	438	312	326	312
query87	3298	3145	3141	3141
query88	3215	2448	2443	2443
query89	485	378	383	378
query90	1941	199	191	191
query91	126	97	103	97
query92	60	48	47	47
query93	1454	511	506	506
query94	1200	192	187	187
query95	403	308	303	303
query96	583	273	268	268
query97	3159	2970	3012	2970
query98	237	226	271	226
query99	1184	882	881	881
Total cold run time: 273998 ms
Total hot run time: 187647 ms

doris-robot commented May 11, 2024

TeamCity be ut coverage result:
Function Coverage: 35.65% (8981/25190)
Line Coverage: 27.32% (74244/271753)
Region Coverage: 26.55% (38372/144515)
Branch Coverage: 23.38% (19572/83726)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f205fc8b380ea71495a487af1c85f64bd6e7d2e8_f205fc8b380ea71495a487af1c85f64bd6e7d2e8/report/index.html

yiguolei approved these changes

View reviewed changes

github-actions bot added the approved label

Contributor

github-actions bot commented May 11, 2024

PR approved by at least one committer and no changes requested.

github-actions bot added the reviewed label

Contributor

github-actions bot commented May 11, 2024

PR approved by anyone and no changes requested.

zhiqiang-hhhh approved these changes

View reviewed changes

Contributor

zhiqiang-hhhh left a comment

LGTM

yiguolei merged commit dbece74 into apache:master

24 of 28 checks passed

xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request


          [fix](memory) Fix prune all LRU Cache based on number apache#34601

bd0bb98

xinyiZzz mentioned this pull request

[branch-2.1](memory) Fix prune all LRU Cache based on number #34736

Merged

yiguolei pushed a commit that referenced this pull request


          [fix](memory) Fix prune all LRU Cache based on number #34601 (#34736)

7a172a5

M1saka2003 pushed a commit to M1saka2003/doris that referenced this pull request


          [fix](memory) Fix prune all LRU Cache based on number apache#34601

66719f2

ByteYue pushed a commit to ByteYue/doris that referenced this pull request


          [fix](memory) Fix prune all LRU Cache based on number apache#34601

15bf473

M1saka2003 pushed a commit to M1saka2003/doris that referenced this pull request


          [fix](memory) Fix prune all LRU Cache based on number apache#34601

8ce9c2d

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment