Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix] Process not destroy in DiskUtils df function #34387

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

[fix] Process not destroy in DiskUtils df function #34387

wants to merge 3 commits into from
+50 −22

Conversation

Vallishp
Copy link
Contributor

@Vallishp Vallishp commented May 3, 2024 •
edited

Proposed changes

Issue Number: close #34386

  1. In DiskUtils df function
    Added Process destroy and buffer read close in finally.
    added UT to verify and cover the code change.
  2. Process not destroy in plsql Exec
    fixed and reviewed.
  3. Process not destroy DppScheduler
    fixed and reviewed

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Vallishp
Copy link
Contributor Author

Vallishp commented May 3, 2024

run buildall

@doris-robot
Copy link

TPC-DS: Total hot run time: 185551 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b73aa1790006f14762b6cec001061b8bc69f5bd4, data reload: false

query1	936	361	348	348
query2	6531	2444	2270	2270
query3	6658	201	204	201
query4	23090	21802	21892	21802
query5	3894	446	417	417
query6	277	190	168	168
query7	4566	307	298	298
query8	263	189	191	189
query9	8552	2368	2342	2342
query10	413	261	269	261
query11	15281	14682	14656	14656
query12	121	85	87	85
query13	1645	382	364	364
query14	9804	8322	6530	6530
query15	244	165	161	161
query16	8179	252	252	252
query17	1910	550	563	550
query18	2089	288	270	270
query19	323	156	150	150
query20	86	86	85	85
query21	192	126	124	124
query22	5024	4876	4881	4876
query23	34003	33337	33098	33098
query24	10658	2917	2851	2851
query25	587	354	385	354
query26	1134	143	146	143
query27	2350	307	310	307
query28	7192	1983	1982	1982
query29	834	592	598	592
query30	242	147	149	147
query31	935	737	744	737
query32	97	50	51	50
query33	732	244	238	238
query34	1058	477	472	472
query35	803	644	654	644
query36	1039	903	892	892
query37	137	65	68	65
query38	3116	3007	3004	3004
query39	1580	1547	1538	1538
query40	199	126	126	126
query41	40	38	38	38
query42	103	93	92	92
query43	568	551	530	530
query44	1218	740	735	735
query45	263	243	221	221
query46	1070	717	704	704
query47	1935	1863	1833	1833
query48	379	294	291	291
query49	832	411	402	402
query50	753	397	384	384
query51	6774	6716	6697	6697
query52	101	92	89	89
query53	348	272	277	272
query54	336	233	236	233
query55	72	71	72	71
query56	242	213	214	213
query57	1233	1144	1141	1141
query58	217	191	197	191
query59	3558	3168	2971	2971
query60	252	233	229	229
query61	91	85	85	85
query62	650	444	461	444
query63	305	274	280	274
query64	8573	7252	7233	7233
query65	3123	3078	3019	3019
query66	812	328	329	328
query67	15383	15069	14892	14892
query68	5939	534	537	534
query69	544	304	306	304
query70	1130	1148	1051	1051
query71	482	265	267	265
query72	8004	2507	2382	2382
query73	723	318	315	315
query74	6497	6080	6062	6062
query75	3905	2647	2665	2647
query76	4122	1052	933	933
query77	615	258	264	258
query78	10766	10107	10070	10070
query79	6006	511	507	507
query80	1558	435	429	429
query81	501	219	223	219
query82	1576	89	92	89
query83	198	164	160	160
query84	264	87	83	83
query85	1435	291	258	258
query86	455	300	290	290
query87	3306	3088	3090	3088
query88	5054	2411	2405	2405
query89	471	366	380	366
query90	2005	184	180	180
query91	120	99	94	94
query92	56	45	48	45
query93	5286	502	506	502
query94	1132	179	178	178
query95	389	298	304	298
query96	593	264	262	262
query97	3113	2961	2956	2956
query98	231	215	214	214
query99	1206	856	871	856
Total cold run time: 293687 ms
Total hot run time: 185551 ms

@Vallishp
Copy link
Contributor Author

Vallishp commented May 3, 2024

run buildall

@xinyiZzz xinyiZzz self-requested a review May 8, 2024 02:34
dataroaring
dataroaring previously approved these changes Jun 5, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring
Copy link
Contributor

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jun 5, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jun 5, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jun 5, 2024

PR approved by anyone and no changes requested.

xinyiZzz
xinyiZzz previously approved these changes Jun 12, 2024
View reviewed changes
Copy link
Contributor

@xinyiZzz xinyiZzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xinyiZzz
Copy link
Contributor

run buildall

@Vallishp Vallishp dismissed stale reviews from xinyiZzz and dataroaring via c93dfc9 June 12, 2024 09:47
@Vallishp
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jun 12, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 39273 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c93dfc9eb8b385a4932a7cacb598bed8d1aeb290, data reload: false

------ Round 1 ----------------------------------
q1	17617	4257	4217	4217
q2	2018	185	188	185
q3	10484	1081	1148	1081
q4	10186	720	774	720
q5	7454	2650	2605	2605
q6	219	136	136	136
q7	959	608	592	592
q8	9232	2023	2065	2023
q9	8994	6505	6454	6454
q10	8959	3703	3662	3662
q11	441	231	231	231
q12	486	220	219	219
q13	17773	2966	2976	2966
q14	264	216	223	216
q15	520	468	479	468
q16	498	377	395	377
q17	946	717	658	658
q18	7977	7389	7393	7389
q19	5865	1424	1457	1424
q20	690	296	298	296
q21	4820	3022	3898	3022
q22	381	341	332	332
Total cold run time: 116783 ms
Total hot run time: 39273 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4390	4217	4204	4204
q2	363	270	267	267
q3	3005	2786	2915	2786
q4	1957	1698	1665	1665
q5	5614	5492	5462	5462
q6	218	130	131	130
q7	2167	1767	1824	1767
q8	3247	3437	3367	3367
q9	8688	8684	8782	8684
q10	4077	3899	3725	3725
q11	580	486	486	486
q12	788	625	656	625
q13	16041	3217	3134	3134
q14	313	290	282	282
q15	529	479	508	479
q16	524	435	431	431
q17	1800	1509	1512	1509
q18	8056	7880	7959	7880
q19	1820	1620	1655	1620
q20	2153	1898	1857	1857
q21	5108	4824	4825	4824
q22	615	511	535	511
Total cold run time: 72053 ms
Total hot run time: 55695 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174057 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c93dfc9eb8b385a4932a7cacb598bed8d1aeb290, data reload: false

query1	926	377	372	372
query2	6468	2367	2344	2344
query3	6640	212	216	212
query4	19230	17508	17414	17414
query5	3582	463	452	452
query6	238	155	161	155
query7	4584	302	285	285
query8	320	293	287	287
query9	8427	2373	2339	2339
query10	583	303	273	273
query11	10606	10178	10088	10088
query12	118	86	87	86
query13	1652	373	366	366
query14	9559	7060	7795	7060
query15	250	203	196	196
query16	7773	278	274	274
query17	1814	555	542	542
query18	1988	272	275	272
query19	216	156	154	154
query20	109	85	84	84
query21	207	137	136	136
query22	4567	4625	4407	4407
query23	34147	33331	33627	33331
query24	10888	2779	2803	2779
query25	640	375	374	374
query26	1133	151	152	151
query27	2689	316	318	316
query28	7036	2026	2044	2026
query29	915	631	594	594
query30	228	145	153	145
query31	936	720	741	720
query32	91	54	53	53
query33	758	275	270	270
query34	903	455	457	455
query35	745	629	613	613
query36	1124	958	933	933
query37	153	68	71	68
query38	2970	2796	2804	2796
query39	878	825	834	825
query40	210	130	122	122
query41	59	51	51	51
query42	128	109	102	102
query43	624	551	557	551
query44	1223	727	722	722
query45	195	167	159	159
query46	1069	712	711	711
query47	1859	1778	1785	1778
query48	376	301	296	296
query49	874	403	406	403
query50	763	388	404	388
query51	6817	6696	6641	6641
query52	105	89	89	89
query53	353	289	280	280
query54	882	447	453	447
query55	73	75	71	71
query56	280	255	259	255
query57	1130	1033	1063	1033
query58	239	252	245	245
query59	3682	3325	3377	3325
query60	300	268	259	259
query61	93	89	91	89
query62	613	467	456	456
query63	316	292	286	286
query64	8823	2213	1775	1775
query65	3192	3108	3108	3108
query66	767	318	323	318
query67	15366	15179	14979	14979
query68	4524	541	544	541
query69	483	406	352	352
query70	1159	1135	1057	1057
query71	399	264	264	264
query72	7037	5564	5576	5564
query73	747	327	326	326
query74	6070	5519	5484	5484
query75	3367	2644	2650	2644
query76	2189	936	891	891
query77	454	305	305	305
query78	10429	9983	9793	9793
query79	2494	510	509	509
query80	2061	451	458	451
query81	581	224	226	224
query82	815	100	106	100
query83	308	167	166	166
query84	267	86	93	86
query85	1263	281	268	268
query86	455	310	317	310
query87	3292	3060	3073	3060
query88	3945	2439	2441	2439
query89	469	373	384	373
query90	1640	183	190	183
query91	129	110	99	99
query92	61	51	50	50
query93	1958	501	496	496
query94	1132	252	183	183
query95	390	306	309	306
query96	591	278	264	264
query97	3197	3050	3069	3050
query98	223	197	198	197
query99	1306	838	836	836
Total cold run time: 269937 ms
Total hot run time: 174057 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.16 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c93dfc9eb8b385a4932a7cacb598bed8d1aeb290, data reload: false

query1	0.05	0.03	0.04
query2	0.08	0.04	0.04
query3	0.23	0.04	0.04
query4	1.69	0.07	0.07
query5	0.51	0.50	0.49
query6	1.12	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.53	0.49	0.48
query10	0.54	0.56	0.54
query11	0.15	0.11	0.12
query12	0.14	0.11	0.12
query13	0.59	0.58	0.58
query14	0.78	0.79	0.76
query15	0.82	0.81	0.81
query16	0.37	0.35	0.38
query17	1.00	0.99	0.96
query18	0.22	0.26	0.22
query19	1.75	1.68	1.70
query20	0.01	0.02	0.01
query21	15.41	0.65	0.65
query22	3.89	8.19	1.75
query23	18.26	1.27	1.24
query24	2.09	0.24	0.21
query25	0.17	0.08	0.09
query26	0.26	0.18	0.18
query27	0.07	0.09	0.08
query28	13.30	1.02	1.00
query29	13.00	3.24	3.24
query30	0.26	0.06	0.06
query31	2.89	0.39	0.37
query32	3.27	0.47	0.46
query33	2.85	2.97	2.87
query34	17.21	4.38	4.38
query35	4.53	4.50	4.51
query36	0.65	0.46	0.47
query37	0.18	0.15	0.14
query38	0.15	0.15	0.14
query39	0.05	0.04	0.04
query40	0.17	0.15	0.14
query41	0.10	0.05	0.05
query42	0.05	0.04	0.04
query43	0.04	0.03	0.04
Total cold run time: 109.5 s
Total hot run time: 30.16 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xinyiZzz xinyiZzz xinyiZzz left review comments

@dataroaring dataroaring dataroaring left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Bug] Runtime.getRuntime().exec created process need to be destroyed
4 participants
@Vallishp @doris-robot @dataroaring @xinyiZzz