Consul registry performance improvements #928
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi, it's me again trying to improve route update performance for our absurd number of routes. 馃榿
There's two major changes here:
The first is adding "filtering" so that only Consul Health Checks with the applicable Service Tag Prefix (and serf/maintenance checks) are considered during the rest of the process. In our Production environments this brings down the number of checks that Fabio has to look at from ~40k down to ~12k.
The second change is simplifying the$O(n^5)$ process to $O(n^2)$ . Each of the "helper" functions (
passingServices
function from ancountChecks
isAgentCritical
isNodeInMaintenance
isServiceInMaintenance
) did their own loop over each of the Consul Health Checks which added up in time very quickly.Both of these changes drastically reduced the amount of time spent processing responses from Consul. In a test using real Consul data but not serving traffic we saw a reduction from around 90 seconds to less than 10 seconds. We've been testing Fabio with these changes in a non-production environment for about a week now, and have not detected any issues.
(The increase in
makeConfig
shown on the graph is due to how I measured it. It includes time waiting to write on the channel back to the main Fabio goroutine. This wasn't an issue before because processing the Consul data took longer than the time needed to build the actual route table (~10s) but the scales have now flipped. Building the Route Table takes longer than processing the data from Consul.)I also had profiling running during my tests. Here's stock Fabio:
After adding the Check "filtering":
After simplifying
passingServices
:I'm very interested in any feedback about these changes and will do what I can to help satisfy any concerns. Thanks!