Allow Stats Integration to take bundles of Experiments #2367
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lukesonnet
changed the title
|
Your preview environment pr-2367-bttf has been deployed. Preview environment endpoints are available at: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
This PR rewrites the stats integration to allow for multiple bundles of
DataForStatsEngines to be passed, each with an id, to the stats engine, to be returned with results alongside that id. The goal is to eventually allow for easier batch creation of multiple snapshots or snapshot analyses when needed (e.g. for impact analysis).Previously the stats engine could take a
snapshotand an array ofanalysesfor that snapshot to be computed for each metric.Now, the stats engine takes an an array of
{snapshots, analyses}so that you can compute several analyses (each set of analyses potentially different from one another) for any number of snapshots (within or across experiments).Details
It creates matching input and output objects in
stats.d.tsand in python for an object with an ID and the payload (either data in or results out).We create a helper that just does one experiment at a time to keep the current snapshot creation flow largely untouched beyond the actual python we're executing.
I also create
createMultipleSnapshotAnalysisandpostSnapshotsWithScaledImpactAnalysisfor how this new functionality gets used to kick off multiple jobs at once (e.g. the batch update for the experiment impact analysis).This PR also adds default running scaled and absolute difference types to each stats engine call, since this should add minimal python runtime and will help reduce future burden when loading the dashboard for Impact Analysis.