You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Usually, during experiments, we can face customers who perform enormous amounts of transactions. Such outliers lead to bias and can change experiment results. To fix this issue, we have filtered out such clients based on the total amount of transactions during the experiment. Currently, Growthbook supports capping for revenue and count. Unfortunately, it does not work for binomial metrics.
I tried to use count metric with capping with pretty simple binomial metric SQL adjustments:
SELECTe.value.string_value as ecentria_id,
user_pseudo_id,
TIMESTAMP_MICROS(event_timestamp) astimestamp,
1as value -- <--------- this line was addedFROM
....
However, capping does not work as expected. It filters out rows that have a value higher than the specific threshold. While I'm looking into filtering out all rows that have a relation to the client that contributes an enormous amount of transactions.
Having a feature that helps filter out clients based on a threshold (absolute or percentile) by aggregated values over all time of the experiment, can help to increase test sensitivity by decreasing variation.
We definitely can workaround it with custom SQL, but unfortunately, it requires hardcoded logic in each metric.
Related Issues
TBD
The text was updated successfully, but these errors were encountered:
Description of Feature
Usually, during experiments, we can face customers who perform enormous amounts of transactions. Such outliers lead to bias and can change experiment results. To fix this issue, we have filtered out such clients based on the total amount of transactions during the experiment. Currently, Growthbook supports capping for revenue and count. Unfortunately, it does not work for binomial metrics.
I tried to use count metric with capping with pretty simple binomial metric SQL adjustments:
However, capping does not work as expected. It filters out rows that have a
value
higher than the specific threshold. While I'm looking into filtering out all rows that have a relation to the client that contributes an enormous amount of transactions.Having a feature that helps filter out clients based on a threshold (absolute or percentile) by aggregated values over all time of the experiment, can help to increase test sensitivity by decreasing variation.
We definitely can workaround it with custom SQL, but unfortunately, it requires hardcoded logic in each metric.
Related Issues
TBD
The text was updated successfully, but these errors were encountered: