Add administrative interface to invoker and controller to reconfigure runtimes #4790

ningyougang · 2020-01-07T03:18:33Z

Sometimes, admin may want to reinitalize the runtime config, e.g. nodejs:10 prewarm container number is less, lead to cold start, in order to handle user's request as soon as possible, admin may want to reinitalize the runtime configuration to increase the nodejs:10 prewarm containers.
And admin may want to reinitalize the runtime config on some limited invokers, this patch includes below changes

config runtime via controller to all managed invokers(or some limited inovkers which included in managed inovkers)
config runtime via invoker directly
get runtime info via invoker directly
add documents
test cases

Description

Related issue and scope

I opened an issue to propose and discuss this change (#????)

My changes affect the following components

Types of changes

Bug fix (generally a non-breaking change which closes an issue).
Enhancement or new feature (adds new functionality).
Breaking change (a bug fix or enhancement which changes existing behavior).

Checklist:

I signed an Apache CLA.
I reviewed the style guides and followed the recommendations (Travis CI will check :).
I added tests to cover my changes.
My changes require further changes to the documentation.
I updated the documentation where necessary.

ningyougang · 2020-01-07T03:38:51Z

core/controller/src/main/scala/org/apache/openwhisk/core/controller/Controller.scala

+      extractCredentials {
+        case Some(BasicHttpCredentials(username, password)) =>
+          if (username == controllerUsername && password == controllerPassword) {
+            entity(as[String]) { runtime =>


Usually, entity(as[Runtimes]) may be better, but if we apply this, need to change Runtimes to support convert json to entity Runtimes, and on the other hand, runtime.json's format doesn't match Runtimes entity, need to change a lot if use entity(as[Runtimes]) to receive the data.

Fortunately，we can reuse openwhisk itself's initialize method, just convert the runtime string to Runtimes.

So here, i think pass runtime string would be ok.

I think you can do as json object instead of string to tighten this a bit more. I agree not to deserialize into a Runtimes instance.

ningyougang · 2020-01-07T03:40:38Z

core/controller/src/main/scala/org/apache/openwhisk/core/controller/Controller.scala

+                logging.error(this, s"Received invalid runtimes manifest")
+                complete(s"Received invalid runtimes manifest")
+              } else {
+                parameter('limit.?) { limit =>


Support passed limit invokers, e.g. ?limit=0:1 ( sent to invoker0, invoker1 only)
And the config runtime request can be sent to some limited invokers which included in managed invokers as well.

style95 · 2020-01-07T12:20:09Z

common/scala/src/main/resources/reference.conf

@@ -27,7 +27,7 @@ whisk.spi {
  EntitlementSpiProvider = org.apache.openwhisk.core.entitlement.LocalEntitlementProvider
  AuthenticationDirectiveProvider = org.apache.openwhisk.core.controller.BasicAuthenticationDirective
  InvokerProvider = org.apache.openwhisk.core.invoker.InvokerReactive
-  InvokerServerProvider = org.apache.openwhisk.core.invoker.DefaultInvokerServer
+  InvokerServerProvider = org.apache.openwhisk.core.invoker.InvokerServer


This should be reverted.

DefaultInvokerServer is already reverted

style95 · 2020-01-07T12:21:50Z

core/controller/src/main/scala/org/apache/openwhisk/core/controller/Controller.scala

+                        val invokerArray = targetValue.split(":")
+                        val beginIndex = invokerArray(0).toInt
+                        val finishIndex = invokerArray(1).toInt
+                        if (finishIndex < beginIndex) {


Would be great to support 0:0 case as well.
Ansible is using the same way.

Yes, already support 0:0 case.

codecov-io · 2020-01-13T09:03:33Z

Codecov Report

Merging #4790 into master will decrease coverage by 6.86%.
The diff coverage is 41.78%.

@@            Coverage Diff             @@
##           master    #4790      +/-   ##
==========================================
- Coverage   82.35%   75.49%   -6.87%     
==========================================
  Files         198      199       +1     
  Lines        9021     9123     +102     
  Branches      353      379      +26     
==========================================
- Hits         7429     6887     -542     
- Misses       1592     2236     +644

Impacted Files	Coverage Δ
...la/org/apache/openwhisk/core/invoker/Invoker.scala	`71.66% <ø> (-0.47%)`	⬇️
.../apache/openwhisk/core/controller/Controller.scala	`0% <0%> (ø)`	⬆️
...che/openwhisk/core/loadBalancer/LoadBalancer.scala	`15.38% <0%> (-1.29%)`	⬇️
...e/loadBalancer/ShardingContainerPoolBalancer.scala	`73.91% <0%> (-2.93%)`	⬇️
...rg/apache/openwhisk/core/entity/ExecManifest.scala	`97.36% <100%> (+0.14%)`	⬆️
.../scala/org/apache/openwhisk/core/WhiskConfig.scala	`94.85% <100%> (+0.03%)`	⬆️
.../org/apache/openwhisk/core/connector/Message.scala	`77.61% <37.5%> (-2.55%)`	⬇️
...e/openwhisk/core/containerpool/ContainerPool.scala	`89.04% <44.44%> (-6.61%)`	⬇️
...pache/openwhisk/core/invoker/InvokerReactive.scala	`78.62% <64.4%> (-1.03%)`	⬇️
.../openwhisk/core/invoker/DefaultInvokerServer.scala	`75% <75%> (ø)`
... and 21 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 394e9f6...12201bb. Read the comment docs.

ningyougang · 2020-01-13T09:20:08Z

ansible/roles/controller/tasks/deploy.yml

@@ -203,6 +203,9 @@
      "CONFIG_whisk_db_activationsFilterDdoc": "{{ db_whisk_activations_filter_ddoc | default() }}"
      "CONFIG_whisk_userEvents_enabled": "{{ user_events | default(false) | lower }}"

+      "CONFIG_whisk_credentials_controller_username": "{{ controller.username }}"
+      "CONFIG_whisk_credentials_controller_password": "{{ controller.password }}"
+


If generate controller/invoker credentials to container's /conf/, the Standalone Tests run failed due to lack /conf/ under the build machine

On the other hand, couchdb credentials is passed via environment variable, so controller/invoker credentials can be passed via environment variable as well.

ningyougang · 2020-01-14T08:35:11Z

Already added test cases

rabbah

The limiting of a runtime to specific invokers is related to how invokers are partitioned for black box vs non-black box invokers (based on a % of active invokers in the system). I suggest splitting out the "limit" feature into a separate PR and having a mechanism for specifying distributions of runtimes to invokers and letting the load balancer calculate the partition.

That isn't powerful enough to when it is desirable to route to a specific invoker - is that a desired feature (vs limiting to a range of invokers active in the system)?

rabbah · 2020-01-26T15:41:25Z

core/controller/src/main/scala/org/apache/openwhisk/core/controller/Controller.scala

+              val execManifest = ExecManifest.initialize(runtime)
+              if (execManifest.isFailure) {
+                logging.error(this, s"Received invalid runtimes manifest")
+                complete(s"Received invalid runtimes manifest")


this completes the https request with status code 200 - is that what was intended?

Hi, almost forgot this patch :(
I changed like below

logging.info(this, s"received invalid runtimes manifest") complete(StatusCodes.BadRequest)

rabbah · 2020-01-26T15:42:13Z

core/controller/src/main/scala/org/apache/openwhisk/core/controller/Controller.scala

+            entity(as[String]) { runtime =>
+              val execManifest = ExecManifest.initialize(runtime)
+              if (execManifest.isFailure) {
+                logging.error(this, s"Received invalid runtimes manifest")


.error is for internal errors, where here this is user input error, i think .info is better.

Already changed to .info

rabbah · 2020-01-26T15:44:18Z

core/controller/src/main/scala/org/apache/openwhisk/core/controller/Controller.scala

+                parameter('limit.?) { limit =>
+                  limit match {
+                    case Some(targetValue) =>
+                      val pattern = "\\d+:\\d"


use

Suggested change

val pattern = "\\d+:\\d"

val pattern = """\d+:\d"""

using triple quotes allows you to use a regex without escape characters.

rabbah · 2020-01-26T15:45:14Z

core/controller/src/main/scala/org/apache/openwhisk/core/controller/Controller.scala

+          if (username == controllerCredentials.username && password == controllerCredentials.password) {
+            entity(as[String]) { runtime =>
+              val execManifest = ExecManifest.initialize(runtime)
+              if (execManifest.isFailure) {


scala nit: you can use a case match here instead.

I just follow other codes using execManifest.isFailure to judge whether success or fail

rabbah · 2020-01-26T15:48:55Z

core/controller/src/main/scala/org/apache/openwhisk/core/controller/Controller.scala

+                          complete(s"finishIndex can't be less than beginIndex")
+                        } else {
+                          val targetInvokers = (beginIndex to finishIndex).toList
+                          loadBalancer.sendRuntimeToInvokers(runtime, Some(targetInvokers))


is there a later validation to check that the invoker indexing is in range?

Yes, validate it in ShardingContainerPoolLoadbalacer.scala

/** send runtime to invokers*/ override def sendRuntimeToInvokers(runtime: String, targetInvokers: Option[List[Int]]): Unit = { val runtimeMessage = RuntimeMessage(runtime) schedulingState.managedInvokers.filter { manageInvoker => targetInvokers.getOrElse(schedulingState.managedInvokers.map(_.id.instance)).contains(manageInvoker.id.instance) } foreach { invokerHealth => val topic = s"invoker${invokerHealth.id.toInt}" messageProducer.send(topic, runtimeMessage).andThen { case Success(_) => logging.info(this, s"Successfully posted runtime to topic $topic") case Failure(_) => logging.error(this, s"Failed posted runtime to topic $topic") } } }

rabbah · 2020-01-26T15:49:28Z

core/controller/src/main/scala/org/apache/openwhisk/core/controller/Controller.scala

+                  limit match {
+                    case Some(targetValue) =>
+                      val pattern = "\\d+:\\d"
+                      if (targetValue.matches(pattern)) {


scala nit: you can rewrite this either if/else and nested clauses using case matching on regex.

hm.. can you show a example? my codes like below

if (targetValue.matches(pattern)) { val invokerArray = targetValue.split(":") val beginIndex = invokerArray(0).toInt val finishIndex = invokerArray(1).toInt if (finishIndex < beginIndex) { logging.info(this, "finishIndex can't be less than beginIndex") complete(StatusCodes.BadRequest) } else { val targetInvokers = (beginIndex to finishIndex).toList loadBalancer.sendRuntimeToInvokers(runtime, Some(targetInvokers)) logging.info(this, "config runtime request is already sent to target invokers") complete(StatusCodes.BadRequest) } } else { logging.info(this, "limit value can't match [beginIndex:finishIndex]") complete(StatusCodes.BadRequest) }

rabbah · 2020-01-26T15:51:30Z

core/invoker/src/main/scala/org/apache/openwhisk/core/invoker/DefaultInvokerServer.scala

+
+  private val invokerCredentials = loadConfigOrThrow[InvokerCredentials](ConfigKeys.invokerCredentials)
+
+  override def routes(implicit transid: TransactionId): Route = {


how will new runtime images get pulled to the invoker nodes? are you assuming they get pulled either via stem-cell "run" or when an action actually runs in the future?

When send to reqeust to backend (e.g. curl -u xxx:xxx-X POST http://xxx:xxx:10001/config/runtime), it will pull new runtime image immediately and create prewarm container if the stem-cell > 0.

rabbah · 2020-01-26T15:53:54Z

tests/src/test/scala/common/WhiskProperties.java

@@ -258,10 +258,40 @@ public static int getControllerBasePort() {
        return Integer.parseInt(whiskProperties.getProperty("controller.host.basePort"));
    }

+    public static String getControllerProtocol() {


i dont think any of these changes are necessary if you read configs through pureconfig.

Yes, already deleted.

rabbah · 2020-01-26T15:55:16Z

tests/src/test/scala/org/apache/openwhisk/operation/RuntimeConfiguration.scala

+        response.status shouldBe StatusCodes.OK
+      }
+
+    Thread.sleep(5.seconds.toMillis)


Just make previous's request handled successfully, because previous is a ansyc operation.

ningyougang · 2020-02-27T09:50:37Z

@rabbah

The limiting of a runtime to specific invokers is related to how invokers are partitioned for black box vs non-black box invokers (based on a % of active invokers in the system). I suggest splitting out the "limit" feature into a separate PR and having a mechanism for specifying distributions of runtimes to invokers and letting the load balancer calculate the partition.

hm.. this patch just reconfigure the runtime.json(recreate prewarm containers according to passed runtime string), i don't understand what's mean I suggest splitting out the "limit" feature into a separate PR

That isn't powerful enough to when it is desirable to route to a specific invoker - is that a desired feature (vs limiting to a range of invokers active in the system)?

Yes, it is not powerful, so i deleted route to a specific invoker because already have limiting to a range of invokers

ningyougang · 2020-02-27T09:53:39Z

core/controller/src/main/scala/org/apache/openwhisk/core/controller/Controller.scala

+                    case None =>
+                      loadBalancer.sendRuntimeToInvokers(runtime, None)
+                      logging.info(this, "config runtime request is already sent to all managed invokers")
+                      complete(StatusCodes.Accepted)


I changed to complete(StatusCodes.Accepted), because it is a async operation.

ningyougang · 2020-02-27T09:56:49Z

core/invoker/src/main/scala/org/apache/openwhisk/core/containerpool/ContainerPool.scala

@@ -279,6 +285,23 @@ class ContainerPool(childFactory: ActorRefFactory => ActorRef,
    case RescheduleJob =>
      freePool = freePool - sender()
      busyPool = busyPool - sender()
+    case prewarmConfigList: PreWarmConfigList =>
+      laststPrewarmConfig = prewarmConfigList.list


Add a inner variable laststPrewarmConfig, just make it pointed to the passed config runtime.
Because in this patch: #4698
when received ContainerRemoved(may be prewarm container crashed), it will backfill the prewarm.

ningyougang · 2020-03-09T01:07:51Z

Rebased.

ningyougang · 2020-06-08T06:27:45Z

Rebased

ningyougang · 2020-06-08T09:27:37Z

core/invoker/src/main/scala/org/apache/openwhisk/core/containerpool/ContainerPool.scala

+            passedConfig.exec.kind == config.exec.kind && passedConfig.memoryLimit == config.memoryLimit)
+          .getOrElse(config)
+      }
+      latestPrewarmConfig = newPrewarmConfig


Support just change specify runtime config as well, e.g.
Let's assume runtimes.json include nodejs:12, python:2, swift4.1, we can change nodejs:12 runtime only via just pass nodejs:12 runtime info, for other runtimes, just use previous runtime info directly.

codecov-commenter · 2020-06-08T10:03:11Z

Codecov Report

Merging #4790 into master will decrease coverage by 6.56%.
The diff coverage is 50.00%.

@@            Coverage Diff             @@
##           master    #4790      +/-   ##
==========================================
- Coverage   83.56%   76.99%   -6.57%     
==========================================
  Files         201      202       +1     
  Lines        9515     9613      +98     
  Branches      400      415      +15     
==========================================
- Hits         7951     7402     -549     
- Misses       1564     2211     +647

Impacted Files	Coverage Δ
...che/openwhisk/core/loadBalancer/LoadBalancer.scala	`76.92% <0.00%> (-6.42%)`	⬇️
...e/loadBalancer/ShardingContainerPoolBalancer.scala	`77.71% <0.00%> (-3.08%)`	⬇️
...la/org/apache/openwhisk/core/invoker/Invoker.scala	`71.66% <ø> (-0.47%)`	⬇️
.../apache/openwhisk/core/controller/Controller.scala	`45.11% <10.00%> (-9.70%)`	⬇️
.../org/apache/openwhisk/core/connector/Message.scala	`76.11% <37.50%> (-4.04%)`	⬇️
...pache/openwhisk/core/invoker/InvokerReactive.scala	`77.86% <64.40%> (-1.79%)`	⬇️
...e/openwhisk/core/containerpool/ContainerPool.scala	`94.25% <66.66%> (-3.62%)`	⬇️
.../openwhisk/core/invoker/DefaultInvokerServer.scala	`75.00% <75.00%> (ø)`
.../scala/org/apache/openwhisk/core/WhiskConfig.scala	`95.00% <100.00%> (+0.03%)`	⬆️
...rg/apache/openwhisk/core/entity/ExecManifest.scala	`92.15% <100.00%> (+0.32%)`	⬆️
... and 22 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 395149d...4157ebc. Read the comment docs.

rabbah

I like elements of this PR but the orchestration with the invokers (where it's selective) I think can lead to a situation where the controller has a view of the supported runtimes that is not supported by all the invokers. Don't you then need to inform the scheduler/loadbalancer of this disparity so it can route requests correctly to the supporting invokers?

Or do you envision only changing the stem cell configurations, and not the content of the runtimes manifest otherwise?

rabbah · 2021-03-01T23:10:00Z

common/scala/src/main/scala/org/apache/openwhisk/common/ComponentCredentials.scala

+ */
+
+package org.apache.openwhisk.common
+


You can use akka.http.scaladsl.model.headers.BasicHttpCredentials instead of creating this class.

I see later this is created to read/load the configuration - it might still work with BasicHttpCredentials because it's case class with the same fields.

rabbah · 2021-03-01T23:13:42Z

common/scala/src/main/scala/org/apache/openwhisk/core/entity/ExecManifest.scala

+   * @param runtime
+   * @return the manifest if initialized successfully, or an failure
+   */
+  protected[core] def initialize(runtime: String): Try[Runtimes] = {


why not use the method above and set the manifestOverride to Some(...) at the call site?

Yes, you are right, updated accordingly.

rabbah · 2021-03-01T23:14:21Z

common/scala/src/main/scala/org/apache/openwhisk/core/entity/ExecManifest.scala

+  protected[core] def initialize(runtime: String): Try[Runtimes] = {
+    val rmc = loadConfigOrThrow[RuntimeManifestConfig](ConfigKeys.runtimes)
+    val mf = Try(runtime.parseJson.asJsObject).flatMap(runtimes(_, rmc))
+    var manifest: Option[Runtimes] = None


this is incorrect - the reason we do the assignment to manifest on line 57 is to set the singleton. Here, this has no effect because the variable is local.

I already changed to use existed initialize method.

rabbah · 2021-03-01T23:18:35Z

core/controller/src/main/scala/org/apache/openwhisk/core/controller/Controller.scala

+      extractCredentials {
+        case Some(BasicHttpCredentials(username, password)) =>
+          if (username == controllerUsername && password == controllerPassword) {
+            entity(as[String]) { runtime =>


I think you can do as json object instead of string to tighten this a bit more. I agree not to deserialize into a Runtimes instance.

rabbah · 2021-03-01T23:20:28Z

core/controller/src/main/scala/org/apache/openwhisk/core/controller/Controller.scala

+    (path("config" / "runtime") & post) {
+      extractCredentials {
+        case Some(BasicHttpCredentials(username, password)) =>
+          if (username == controllerCredentials.username && password == controllerCredentials.password) {


you can use as authentication directive which will automatically check the credentials and reject the request. This is OK just making an observation.

codecov-io · 2021-03-02T09:04:36Z

Codecov Report

Merging #4790 (edefa8a) into master (d8cf172) will decrease coverage by 7.66%.
The diff coverage is 20.80%.

@@            Coverage Diff             @@
##           master    #4790      +/-   ##
==========================================
- Coverage   81.81%   74.15%   -7.67%     
==========================================
  Files         204      212       +8     
  Lines        9950    10319     +369     
  Branches      447      435      -12     
==========================================
- Hits         8141     7652     -489     
- Misses       1809     2667     +858

Impacted Files	Coverage Δ
.../org/apache/openwhisk/core/connector/Message.scala	`71.32% <0.00%> (-2.12%)`	⬇️
...che/openwhisk/core/loadBalancer/LoadBalancer.scala	`71.42% <0.00%> (-5.50%)`	⬇️
...e/loadBalancer/ShardingContainerPoolBalancer.scala	`77.71% <0.00%> (-3.08%)`	⬇️
...la/org/apache/openwhisk/core/invoker/Invoker.scala	`69.35% <ø> (-0.49%)`	⬇️
.../apache/openwhisk/core/controller/Controller.scala	`45.52% <12.90%> (-9.29%)`	⬇️
...e/openwhisk/core/containerpool/ContainerPool.scala	`87.45% <16.12%> (-9.36%)`	⬇️
...pache/openwhisk/core/invoker/InvokerReactive.scala	`52.63% <26.47%> (-27.37%)`	⬇️
.../openwhisk/core/invoker/DefaultInvokerServer.scala	`44.44% <44.44%> (ø)`
.../scala/org/apache/openwhisk/core/WhiskConfig.scala	`95.51% <100.00%> (+0.11%)`	⬆️
...core/database/cosmosdb/RxObservableImplicits.scala	`0.00% <0.00%> (-100.00%)`	⬇️
... and 42 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d8cf172...edefa8a. Read the comment docs.

style95 · 2023-05-31T01:08:20Z

@ningyougang I want to check if you want to continue this PR.

ningyougang commented Jan 7, 2020

View reviewed changes

style95 reviewed Jan 7, 2020

View reviewed changes

ningyougang closed this Jan 13, 2020

ningyougang reopened this Jan 13, 2020

ningyougang force-pushed the config_runtime branch 3 times, most recently from 5be2eab to 8573912 Compare January 13, 2020 08:24

ningyougang commented Jan 13, 2020

View reviewed changes

ningyougang force-pushed the config_runtime branch 2 times, most recently from 556b5bd to 1ec0720 Compare January 14, 2020 07:10

ningyougang changed the title ~~[WIP]Config runtime~~ Config runtime Jan 14, 2020

rabbah reviewed Jan 26, 2020

View reviewed changes

rabbah changed the title ~~Config runtime~~ Add administrative interfafe to invoker and controller to reconfigure runtimes Jan 26, 2020

rabbah changed the title ~~Add administrative interfafe to invoker and controller to reconfigure runtimes~~ Add administrative interface to invoker and controller to reconfigure runtimes Jan 26, 2020

ningyougang force-pushed the config_runtime branch 3 times, most recently from 81caa3f to 3e0603d Compare February 27, 2020 09:32

ningyougang commented Feb 27, 2020

View reviewed changes

ningyougang force-pushed the config_runtime branch 6 times, most recently from be5a6e5 to 6750ea9 Compare February 29, 2020 08:15

ningyougang force-pushed the config_runtime branch 2 times, most recently from fc63c07 to 12201bb Compare March 1, 2020 01:54

ningyougang force-pushed the config_runtime branch from 12201bb to c245908 Compare March 9, 2020 01:04

ningyougang force-pushed the config_runtime branch 4 times, most recently from 6733631 to 30a3e64 Compare March 23, 2020 08:41

ningyougang mentioned this pull request Mar 31, 2020

Change prewarm container #4225

Closed

19 tasks

ningyougang force-pushed the config_runtime branch from 30a3e64 to 842baa1 Compare June 8, 2020 06:27

ningyougang closed this Jun 8, 2020

ningyougang reopened this Jun 8, 2020

ningyougang commented Jun 8, 2020

View reviewed changes

ningyougang mentioned this pull request Jun 9, 2020

take prewarmed container's memory as used memory #4911

Merged

21 tasks

ningyougang force-pushed the config_runtime branch 2 times, most recently from a5c2986 to 214f8e4 Compare September 8, 2020 01:20

rabbah reviewed Mar 1, 2021

View reviewed changes

ningyougang force-pushed the config_runtime branch 2 times, most recently from a7be1d4 to 695bb14 Compare March 2, 2021 08:17

Config runtime

edefa8a

ningyougang force-pushed the config_runtime branch from 695bb14 to edefa8a Compare March 2, 2021 08:24

style95 added the stale old issue which needs to validate label May 31, 2023


		private val invokerCredentials = loadConfigOrThrow[InvokerCredentials](ConfigKeys.invokerCredentials)

		override def routes(implicit transid: TransactionId): Route = {

		*/

		package org.apache.openwhisk.common

Add administrative interface to invoker and controller to reconfigure runtimes #4790

Are you sure you want to change the base?

Add administrative interface to invoker and controller to reconfigure runtimes #4790

Conversation

ningyougang commented Jan 7, 2020 • edited

Description

Related issue and scope

My changes affect the following components

Types of changes

Checklist:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ningyougang Jan 7, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Jan 13, 2020 • edited

Codecov Report

Choose a reason for hiding this comment

ningyougang commented Jan 14, 2020

rabbah left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ningyougang Feb 27, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ningyougang Feb 27, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ningyougang commented Feb 27, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ningyougang commented Mar 9, 2020

ningyougang commented Jun 8, 2020

ningyougang Jun 8, 2020 • edited

Choose a reason for hiding this comment

codecov-commenter commented Jun 8, 2020 • edited

Codecov Report

rabbah left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Mar 2, 2021 • edited

Codecov Report

style95 commented May 31, 2023

ningyougang commented Jan 7, 2020 •

edited

ningyougang Jan 7, 2020 •

edited

codecov-io commented Jan 13, 2020 •

edited

ningyougang Feb 27, 2020 •

edited

ningyougang Feb 27, 2020 •

edited

ningyougang commented Feb 27, 2020 •

edited

ningyougang Jun 8, 2020 •

edited

codecov-commenter commented Jun 8, 2020 •

edited

codecov-io commented Mar 2, 2021 •

edited