Adjust user memory via api #4917

ningyougang · 2020-06-09T05:28:44Z

Description

Ajust invoker's containerPool's userMemory via API

Related issue and scope

I opened an issue to propose and discuss this change (#????)

My changes affect the following components

Types of changes

Bug fix (generally a non-breaking change which closes an issue).
Enhancement or new feature (adds new functionality).
Breaking change (a bug fix or enhancement which changes existing behavior).

Checklist:

I signed an Apache CLA.
I reviewed the style guides and followed the recommendations (Travis CI will check :).
I added tests to cover my changes.
My changes require further changes to the documentation.
I updated the documentation where necessary.

style95 · 2020-09-08T11:25:29Z

common/scala/src/main/scala/org/apache/openwhisk/core/connector/Message.scala

@@ -426,3 +426,22 @@ object EventMessage extends DefaultJsonProtocol {

  def parse(msg: String) = Try(format.read(msg.parseJson))
 }
+
+case class ByteSizeMessage(userMemory: ByteSize) extends Message {


I think the name should denote its purpose more.

hm.. already changed it from ByteSizeMessage to UserMemoryMessage

style95 · 2020-09-08T11:29:24Z

docs/operation.md

+```
+Note: you can add `?limit` to specify target invokers, e.g. specify invoker0 and invoker1
+```
+curl -u ${username}:${password} -X POST http://${controllerAddress}:${controllerPort}/config/memory?limit=0:1 -d '1024 MB'


I feel this API should be based on JSON and have a better protocol.
For example, the name of the invoker may not be simple like just invoker0.

Currently, i changed it to like below

curl -u admin:admin -X POST http://xxx.xxx.xxx.xxx:10001/config/memory -d ' [{"invoker":0,"memory": "20480 MB"}, {"invoker":1,"memory": "10240 MB"}] '

style95 · 2020-09-08T11:33:29Z

core/invoker/src/main/scala/org/apache/openwhisk/core/containerpool/ContainerPool.scala

@@ -305,6 +308,13 @@ class ContainerPool(childFactory: ActorRefFactory => ActorRef,
    case RescheduleJob =>
      freePool = freePool - sender()
      busyPool = busyPool - sender()
+    case message: ByteSizeMessage =>


I hope this also has a more detailed protocol.
This might be extended for general configurations.

style95 · 2020-09-08T11:33:39Z

core/invoker/src/main/scala/org/apache/openwhisk/core/containerpool/ContainerPool.scala

@@ -74,6 +76,7 @@ class ContainerPool(childFactory: ActorRefFactory => ActorRef,
  var busyPool = immutable.Map.empty[ActorRef, ContainerData]
  var prewarmedPool = immutable.Map.empty[ActorRef, PreWarmedData]
  var prewarmStartingPool = immutable.Map.empty[ActorRef, (String, ByteSize)]
+  var latestUserMemory = poolConfig.userMemory


I hope we can avoid using var if possible.

hm.. seems the lastUserMemory need to support changable

I think we can take advantage of Option.

style95 · 2020-09-08T11:35:39Z

core/invoker/src/main/scala/org/apache/openwhisk/core/containerpool/ContainerPool.scala

+        s"user memory is reconfigured from ${latestUserMemory.toString} to ${message.userMemory.toString}")
+      latestUserMemory = message.userMemory
+    case UserMemoryQuery =>
+      sender() ! latestUserMemory.toString


How about sending the intact obejct directly and handle the type conversion in the InvokerServer layer?
I think this kind of type conversion is only related to the InvokerServer and the client.

hm..
When we want to change invoker's user memory, in this pr, need to send the request to controller, there has one benefit that can modfiy many invoker's user memory with one request only. e.g.

curl -u admin:admin -X POST http://xxx.xxx.xxx.xxx:10001/config/memory -d ' [ {"invoker":0,"memory": "20480 MB"}, {"invoker":1,"memory": "10240 MB"} {"invoker":2,"memory": "51200 MB"} ] '

if send change invoker user memory request to invokerServer, if want to modify many invokers's user memory, need to send http request many times.

Ok since this is just a ByteSize I think we can take this approach.

style95 · 2020-09-08T11:38:21Z

core/controller/src/main/scala/org/apache/openwhisk/core/controller/Controller.scala

+            entity(as[String]) { memory =>
+              try {
+                val userMemoryMessage = ByteSizeMessage(ByteSize.fromString(memory))
+                if (userMemoryMessage.userMemory.size == 0) {


I think this is not the only precondition to configure memory.
This value can be smaller than MemoryLimit.min then invokers would not be able to run any container.

Updated accordingly, the user meomry can't be less than MemoryLimit.min

style95 · 2020-09-08T11:44:11Z

core/invoker/src/main/scala/org/apache/openwhisk/core/containerpool/ContainerPool.scala

+      logging.info(
+        this,
+        s"user memory is reconfigured from ${latestUserMemory.toString} to ${message.userMemory.toString}")
+      latestUserMemory = message.userMemory


Even if controllers check the validity of this value, it would be great to cross-check the value again in the invoker layer.

And regarding the check, I feel the application should not concern the condition of the underlying host.
But on the other hand, I am not sure it would be reasonable to allow a bigger memory than the invoker host has.

I would defer this to other reviewers.

style95 · 2020-11-14T13:11:51Z

core/controller/src/main/scala/org/apache/openwhisk/core/loadBalancer/LoadBalancer.scala

+   * @param userMemory
+   * @param targetInvokers
+   */
+  def sendUserMemoryToInvoker(userMemoryMessage: UserMemoryMessage, targetInvoker: Int): Unit = {}


How about changing the name like this?

Suggested change

def sendUserMemoryToInvoker(userMemoryMessage: UserMemoryMessage, targetInvoker: Int): Unit = {}

def sendChangeRequestToInvoker(userMemoryMessage: UserMemoryMessage, targetInvoker: Int): Unit = {}

style95 · 2020-11-14T13:14:10Z

core/controller/src/main/scala/org/apache/openwhisk/core/controller/Controller.scala

+      extractCredentials {
+        case Some(BasicHttpCredentials(username, password)) =>
+          if (username == controllerCredentials.username && password == controllerCredentials.password) {
+            entity(as[String]) { memory =>


Can't we directly convert this to ConfigMemoryList?

Yes, updated.

style95 · 2020-11-14T13:34:14Z

core/controller/src/main/scala/org/apache/openwhisk/core/controller/Controller.scala

+            entity(as[String]) { memory =>
+              val configMemoryList = memory.parseJson.convertTo[ConfigMemoryList]
+
+              val existIllegalUserMemory = configMemoryList.items.exists { configMemory =>


Scalaism. I think we can change this something like this:

configMemoryList.items.find(MemoryLimit.MIN_MEMORY.compare(.memory) > 0)) match {
case Some() =>
// reject the request
case None =>
}

style95 · 2020-11-14T13:39:11Z

core/invoker/src/main/scala/org/apache/openwhisk/core/containerpool/ContainerPool.scala

@@ -74,6 +76,7 @@ class ContainerPool(childFactory: ActorRefFactory => ActorRef,
  var busyPool = immutable.Map.empty[ActorRef, ContainerData]
  var prewarmedPool = immutable.Map.empty[ActorRef, PreWarmedData]
  var prewarmStartingPool = immutable.Map.empty[ActorRef, (String, ByteSize)]
+  var latestUserMemory = poolConfig.userMemory


I think we can take advantage of Option.

style95 · 2020-11-14T13:40:55Z

core/invoker/src/main/scala/org/apache/openwhisk/core/containerpool/ContainerPool.scala

+        s"user memory is reconfigured from ${latestUserMemory.toString} to ${message.userMemory.toString}")
+      latestUserMemory = message.userMemory
+    case UserMemoryQuery =>
+      sender() ! latestUserMemory.toString


Ok since this is just a ByteSize I think we can take this approach.

style95 · 2020-11-14T13:48:24Z

core/invoker/src/main/scala/org/apache/openwhisk/core/invoker/Invoker.scala

@@ -220,9 +223,3 @@ trait InvokerServerProvider extends Spi {
  def instance(
    invoker: InvokerCore)(implicit ec: ExecutionContext, actorSystem: ActorSystem, logger: Logging): BasicRasService
 }
-
-object DefaultInvokerServer extends InvokerServerProvider {


Why is this removed?
This is being used.
https://github.com/apache/openwhisk/blob/master/common/scala/src/main/resources/reference.conf#L30

I moved it to core/invoker/src/main/scala/org/apache/openwhisk/core/invoker/DefaultInvokerServer.scala

style95 · 2020-11-14T13:52:43Z

common/scala/src/main/scala/org/apache/openwhisk/core/connector/Message.scala

+}
+
+case class ConfigMemory(invoker: Int, memory: ByteSize)
+case class ConfigMemoryList(items: List[ConfigMemory])


I believe we can directly unmarshal a list of ConfigMemory rather than haveing this case class.

style95 · 2020-11-14T13:56:19Z

common/scala/src/main/scala/org/apache/openwhisk/core/connector/Message.scala

+  def parse(msg: String) = Try(serdes.read(msg.parseJson))
+}
+
+case class ConfigMemory(invoker: Int, memory: ByteSize)


How about changing the name to InvokerConfiguration.
We might extend this to other invoker configurations as well in the future.

ningyougang mentioned this pull request Jun 9, 2020

take prewarmed container's memory as used memory #4911

Merged

21 tasks

ningyougang force-pushed the adjust_user_memory branch 4 times, most recently from 58fdb50 to 12467b7 Compare September 8, 2020 07:29

style95 reviewed Sep 8, 2020

View reviewed changes

ningyougang force-pushed the adjust_user_memory branch 2 times, most recently from f9dd602 to c7f68ff Compare October 16, 2020 01:20

style95 reviewed Nov 14, 2020

View reviewed changes

Adjust user memory via api

65086cf

ningyougang force-pushed the adjust_user_memory branch from c7f68ff to 65086cf Compare March 2, 2021 09:35

ningyougang closed this Mar 3, 2021

ningyougang reopened this Mar 3, 2021

style95 added the stale old issue which needs to validate label May 31, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adjust user memory via api #4917

Adjust user memory via api #4917

ningyougang commented Jun 9, 2020 •

edited

style95 Sep 8, 2020

ningyougang Oct 14, 2020

style95 Sep 8, 2020

ningyougang Oct 14, 2020

style95 Sep 8, 2020

style95 Sep 8, 2020

ningyougang Oct 15, 2020

style95 Nov 14, 2020

style95 Sep 8, 2020

ningyougang Oct 15, 2020

style95 Nov 14, 2020

style95 Sep 8, 2020

ningyougang Oct 14, 2020

style95 Sep 8, 2020

style95 Nov 14, 2020

ningyougang Mar 2, 2021

style95 Nov 14, 2020

ningyougang Mar 2, 2021

style95 Nov 14, 2020

style95 Nov 14, 2020

style95 Nov 14, 2020

style95 Nov 14, 2020

ningyougang Mar 2, 2021

style95 Nov 14, 2020

ningyougang Mar 2, 2021

style95 Nov 14, 2020

ningyougang Mar 2, 2021

	def sendUserMemoryToInvoker(userMemoryMessage: UserMemoryMessage, targetInvoker: Int): Unit = {}
	def sendChangeRequestToInvoker(userMemoryMessage: UserMemoryMessage, targetInvoker: Int): Unit = {}

Adjust user memory via api #4917

Are you sure you want to change the base?

Adjust user memory via api #4917

Conversation

ningyougang commented Jun 9, 2020 • edited

Description

Related issue and scope

My changes affect the following components

Types of changes

Checklist:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ningyougang commented Jun 9, 2020 •

edited