Logs and data lost in rulechain

[MartinCouadeau]

I'm encountering a problem within the RuleChains. Occasionally, logs seem to disappear when transitioning from one node to another, despite the out message being a succes and having no errors. While in most instances, the data continues to flow and reaches its intended destination even if the logs arent present, there are rare occurrences where the data is entirely lost from one node to another.

example:

• i have this rulechain

• this is the log of node n°1

last log was yesterday at 6:42 am

• log of node n°2

last log was yesterday at 2:12 pm but that event doesnt show in the first node

a more clear example would be:

• i have this other rulechain

• this is the log of the first node

• this is the log of the second node

in this case no event reaches the log of the second node but all the data is being saved correctly, the debug mode in the second node was enabled since that node was created

Has anyone else faced a similar issue like this? I'm keen to understand if this is a common error and if there are any known solutions or workarounds or if its and error or misinterpretation from my part. Any insights or experiences shared would be greatly appreciated. Thank you!

[trikimiki]

Hi!
What kind of deployment you are using?

[devdoob]

@trikimiki I am also facing this issue.

I have Postgres behind, deployed on EC2

[trikimiki]

By default, there is a rate limit for debug node events:

  rule:
  <...>
    chain:
      # Errors for particular actors are persisted once per specified amount of milliseconds
      error_persist_frequency: "${ACTORS_RULE_CHAIN_ERROR_FREQUENCY:3000}"
      debug_mode_rate_limits_per_tenant:
        # Enable/Disable the rate limit of persisted debug events for all rule nodes per tenant
        enabled: "${ACTORS_RULE_CHAIN_DEBUG_MODE_RATE_LIMITS_PER_TENANT_ENABLED:true}"
        # The value of DEBUG mode rate limit. By default, no more then 50 thousand events per hour
        configuration: "${ACTORS_RULE_CHAIN_DEBUG_MODE_RATE_LIMITS_PER_TENANT_CONFIGURATION:50000:3600}"
   <...>

If you enabled the debug node on every other rule node, you could have reached this limit, so events were discarded in your case. If the event was not saved, the data is still being processed.

As about lost data - there are many reasons this could have happened. If you cannot investigate debug events, you can still try checking API Usage stats for Rule Engine (timeouts/failures/exceptions), or backend log files.