Issue 7303: Adding changes for elastic thread pool for LTS processing #7304
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Amit-Singh40 <amit.singh30@dell.com>
Signed-off-by: Amit-Singh40 <amit.singh30@dell.com>
Signed-off-by: Amit-Singh40 <amit.singh30@dell.com>
Signed-off-by: Amit-Singh40 <amit.singh30@dell.com>
Signed-off-by: Amit-Singh40 <amit.singh30@dell.com>
Signed-off-by: Amit-Singh40 <amit.singh30@dell.com>
Signed-off-by: Amit-Singh40 <amit.singh30@dell.com>
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## master #7304 +/- ##
=========================================
Coverage 85.97% 85.97%
- Complexity 16310 16313 +3
=========================================
Files 1042 1042
Lines 60994 61004 +10
Branches 6191 6191
=========================================
+ Hits 52437 52451 +14
+ Misses 5354 5353 -1
+ Partials 3203 3200 -3
☔ View full report in Codecov by Sentry. |
RaulGracia
requested changes
Nov 3, 2023
There was a problem hiding this comment.
I think that both the PR and the original issue need more description of the purpose and expectations of this change. Also, I don't see how the tests are actually verifying the correct behavior of the thread pool dynamic number of threads. Finally, I think it would be required a performance comparison of Pravega with and without this change to see if it induces any regression.
common/src/main/java/io/pravega/common/concurrent/ThreadPoolExecutorConfig.java
Show resolved
Hide resolved
| @@ -42,7 +42,9 @@ public class ServiceConfig { | |||
| public static final Property<Integer> CONTAINER_COUNT = Property.named("container.count", null, "containerCount"); | |||
| public static final Property<Integer> PARALLEL_CONTAINER_STARTS = Property.named("container.parallelStarts", 2); | |||
| public static final Property<Integer> THREAD_POOL_SIZE = Property.named("threadPool.core.size", 30, "threadPoolSize"); | |||
| public static final Property<Integer> STORAGE_THREAD_POOL_SIZE = Property.named("threadPool.storage.size", 200, "storageThreadPoolSize"); | |||
| public static final Property<Integer> STORAGE_THREAD_POOL_SIZE = Property.named("threadPool.storage.size", 50, "storageThreadPoolSize"); | |||
There was a problem hiding this comment.
You are changing default thread pool sizes without any evidence on how this is going to impact the performance of moving data to LTS.
There was a problem hiding this comment.
Raised this PR for review, meanwhile I will be running the performance.
segmentstore/server/src/main/java/io/pravega/segmentstore/server/store/ServiceConfig.java
Outdated
Show resolved
Hide resolved
| @@ -64,7 +64,7 @@ public void testGetDetectionLevel() { | |||
|
|
|||
| @Test | |||
| public void testScheduledThreadPoolLeak() { | |||
| testLeaks(factory -> (ThreadPoolScheduledExecutorService) factory.newScheduledThreadPool(1, "test", 1)); | |||
| testLeaks(factory -> (ThreadPoolScheduledExecutorService) factory.newScheduledThreadPool(new ThreadPoolExecutorConfig(1, 1), "test", 1)); | |||
There was a problem hiding this comment.
This doesn't seem to be testing the "dynamic change" of threads. I think you may need to add new tests to check if the threads in the pool are changing according to what you expect.
Signed-off-by: Amit-Singh40 <amit.singh30@dell.com>
common/src/main/java/io/pravega/common/concurrent/ThreadPoolExecutorConfig.java
Outdated
Show resolved
Hide resolved
common/src/main/java/io/pravega/common/concurrent/ExecutorServiceHelpers.java
Outdated
Show resolved
Hide resolved
common/src/main/java/io/pravega/common/concurrent/ThreadPoolExecutorConfig.java
Show resolved
Hide resolved
common/src/main/java/io/pravega/common/concurrent/ThreadPoolScheduledExecutorService.java
Outdated
Show resolved
Hide resolved
common/src/main/java/io/pravega/common/concurrent/ThreadPoolScheduledExecutorService.java
Outdated
Show resolved
Hide resolved
segmentstore/server/src/main/java/io/pravega/segmentstore/server/store/ServiceBuilder.java
Outdated
Show resolved
Hide resolved
Signed-off-by: Amit-Singh40 <amit.singh30@dell.com>
Signed-off-by: Amit-Singh40 <amit.singh30@dell.com>
Signed-off-by: Amit-Singh40 <amit.singh30@dell.com>
Signed-off-by: Amit-Singh40 <amit.singh30@dell.com>
common/src/main/java/io/pravega/common/concurrent/ThreadPoolExecutorConfig.java
Show resolved
Hide resolved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change log description
Currently we use fixed sized to thread pool for the LTS which needs to changed to elastic thread pool which can grow and shrink accordingly.
Purpose of the change
Fixes #7303
What the code does
Enables elastic thread pool for LTS.
How to verify it
NA