Data quality is especially important when it comes to A\/B testing. A\/B testing allows us to evaluate a feature from different perspectives and make informed decisions. Yet unreliable data can lead to erroneous analysis and poor decision-making that ultimately harms the user experience or other key metrics. Fortunately, there are tools and methodologies that we can use to ensure that we are relying on quality data to produce trustworthy A\/B testing analyses. <\/span>\u00a0<\/span><\/p>\n In this blog post, we\u00a0will\u00a0explore these resources in more detail and answer\u00a0the following\u00a0questions:<\/span>\u00a0<\/span><\/p>\n A\/B testing results are only reliable and actionable if they are based on complete and accurate data. But how do we know that the data that we are using is reliable? The following are several signs of data quality issues in A\/B testing to watch out for. <\/span>\u00a0<\/span><\/p>\n SRM (Sample Ratio Mismatch)<\/span><\/a>\u00a0means that the observed traffic allocated to treatment and control variants\u00a0are\u00a0different\u00a0from\u00a0\u00a0the\u00a0ways that they are configured.\u00a0When\u00a0this occurs, it can lead to\u00a0selection bias.\u00a0<\/span>Analyses that have SRMs are generally viewed as untrustworthy and should not be used to\u00a0make a decision<\/span> (opens in new tab)<\/span><\/a>\u00a0.\u00a0There are many scenarios\u00a0where\u00a0data quality issues\u00a0can\u00a0lead to an SRM. For example, the\u00a0incomplete logging of treatment\/control assignment information in telemetry, or\u00a0the\u00a0unbalanced telemetry loss for treatment\u00a0versus\u00a0control.<\/span>\u00a0<\/span><\/p>\n A\/B metrics should be Sensitive, Trustworthy, Efficient, Debuggable, and Interpretable (STEDI)<\/span><\/a>\u00a0\u00a0to\u00a0help\u00a0the experimenter understand the treatment effect and interpret the metric movements\u00a0that are\u00a0observed.\u00a0Poor data quality can\u00a0negatively\u00a0impact\u00a0metric sensitivity and\u00a0trustworthiness.<\/span>\u00a0<\/span><\/p>\n Sensitivity:<\/span><\/b>\u00a0If a data field has a very high missing rate, then metrics\u00a0that\u00a0are conditional calculations based on this\u00a0data field\u00a0will have a very low sample size. In this scenario, the hypothesis tests on these metrics will\u00a0lack power.\u00a0<\/span>\u00a0<\/span><\/p>\n Outliers\u00a0are\u00a0another\u00a0common\u00a0issue\u00a0that can negatively\u00a0impact\u00a0\u00a0sensitivity.\u00a0Outliers\u00a0increase the variance and small numbers of severe outliers can\u00a0introduce huge noise\u00a0that can\u00a0move metrics.<\/span>\u00a0<\/span><\/p>\n Trustworthiness:<\/span><\/b> Metrics that utilize missing data may have incorrect statistics and untrustworthy hypothesis test results. And it can be more concerning when the missing rate is not balanced between treatment and control. There are several reasons why data may be missing. For instance, failed processing or incomplete data extraction. In the context of retention analysis, missing data can make it look as if a customer churned when they did not, causing the retention rate metric to appear lower.<\/span>\u00a0<\/span><\/p>\n When\u00a0an in-depth\u00a0analysis\u00a0is needed\u00a0for\u00a0various\u00a0segments\u00a0of the population<\/span><\/a>\u00a0and when the fields\u00a0used for segmentation\u00a0are\u00a0inaccurate\u00a0or\u00a0have a high\u00a0missing\u00a0rate, results based on these segments can be misleading.<\/span>\u00a0<\/span><\/p>\n Statistical analysis relies on\u00a0complete data.<\/span>\u00a0In software products,\u00a0it is common to observe delays between data generation and data readiness for analysis. For example, if\u00a0we need\u00a0input from multiple data sources to test a hypothesis,\u00a0the time to\u00a0wait for the availability of all the data could be substantial. Delayed analysis may lead to delayed\u00a0decision-making.\u00a0Consequently,\u00a0we may not detect or rectify\u00a0unexpected regressions in product experience as promptly as desired. To mitigate this, well-designed data pipelines that meet the required Service Level Agreements for its availability are critical.\u00a0\u00a0<\/span>\u00a0<\/span><\/p>\n Figure 1: common data quality dimensions<\/p>\n As shown in\u00a0Figure\u00a01\u00a0above, common data quality dimensions include completeness, uniqueness, timeliness, validity, integrity, consistency, relevance, compliance,\u00a0and\u00a0retention.<\/span>\u00a0<\/span><\/p>\n We have seen many instances\u00a0where\u00a0data quality\u00a0varies across A\/B testing scenarios. For instance,\u00a0a bug in a new feature\u00a0that increases\u00a0invalid values. These instances highlight the importance of\u00a0data quality\u00a0metrics\u00a0in the A\/B testing process.<\/span>\u00a0<\/span><\/p>\n We\u00a0can use the\u00a0following\u00a0checklist\u00a0to\u00a0set\u00a0up additional metrics to measure data quality\u00a0in\u00a0our\u00a0analysis:<\/span>\u00a0<\/span><\/p>\n In addition to the above list,\u00a0we\u00a0should of course ensure that\u00a0we\u00a0are complying with data policy including retention periods and privacy requirements.\u00a0<\/span>\u00a0<\/span><\/p>\n Now let\u2019s take a look at several data fields we should pay extra attention to before analyzing A\/B testing results.<\/span>\u00a0<\/span><\/p>\n Randomization\u00a0unit is the unit on which we randomly assign treatment and control. Common randomization units include users, sessions, pages, impressions, devices, cookies, documents, etc.\u00a0A data quality check\u00a0must\u00a0help\u00a0to\u00a0ensure that the\u00a0data\u00a0belonging to\u00a0the same randomization unit\u00a0are\u00a0linked\u00a0in an accurate and compliant\u00a0manner.\u00a0<\/span>\u00a0<\/span><\/p>\n It is important to always check for missing rates and patterns of the columns which we consider as randomization units. Missing data can appear as nulls, empty strings, zeros, infinity, or default values. Missing data can come from a specific data source due to logging issues or may be introduced in the data cooking pipeline. For example, we may detect missing data from a certain build version or from one data source which has partial delays. In some instances, we may analyze A\/B testing data on multiple aggregation levels (aggregation levels are likely represented by the same columns of randomization units), such as sessions and users (a user can have multiple sessions). We should then check whether all the aggregation level columns are present.<\/span>\u00a0<\/span><\/p>\n Treatment assignment information is necessary to differentiate the data between treatment and control to compare the efficacy of the tested feature.\u00a0A column containing treatment assignment information should always be present in the data\u00a0that is\u00a0used for analysis.\u00a0And it\u2019s better\u00a0to\u00a0ensure\u00a0that\u00a0the treatment assignment information\u00a0follows\u00a0a consistent format, both within the same data source and across multiple data sources if there is\u00a0a\u00a0need to join them together.<\/span>\u00a0<\/span><\/p>\n Sometimes,\u00a0we can add\u00a0additional information to treatment assignment\u00a0information. We can then\u00a0use\u00a0it\u00a0to separate different traffic.\u00a0For example,\u00a0when\u00a0we expand an\u00a0experiment to another audience,\u00a0we can add\u00a0the audience itself to filter the data between groups in the analysis.<\/span>\u00a0<\/span><\/p>\n In addition, treatment assignment should be consistent\u00a0for the randomization unit (i.e,\u00a0users)\u00a0through the\u00a0analysis.\u00a0For\u00a0instance,\u00a0if we assign\u00a0a user\u00a0to a\u00a0treatment\u00a0group\u00a0she\u00a0should not appear in\u00a0the\u00a0control group.<\/span>\u00a0<\/span><\/p>\n When running an A\/B test, we need to keep track of the time when an event of interest occurs. So, there should always be a column representing timestamps of the events. Timestamps play an important role in filtering events that are included in an analysis and in deciding the sequence of events for a triggered analysis (A\/B tests that are analyzed on a subset of the assigned population \u2013\u00a0<\/span>a triggered A\/B analysis<\/span><\/a>).\u00a0We\u00a0should also check for missing rates and filter out values such as a default datetime 1900-01-01 and datetime in the future.<\/span>\u00a0<\/span><\/p>\n It is important to\u00a0carefully consider\u00a0selecting\u00a0an appropriate time to support the\u00a0aforementioned use\u00a0cases, preferably in the context of the\u00a0product. Typically, there are at least two timestamps to choose from. These include\u00a0the time the activity occurred and the time\u00a0when\u00a0the\u00a0corresponding log\u00a0is received. For web-based server-side\u00a0A\/B tests, the time difference is likely to be insignificant and the activity ordering remains unchanged.\u00a0However, for\u00a0client-side\u00a0A\/B tests, these two timestamps can be quite\u00a0different.\u00a0Deciding which one to use will be a subjective trade-off taken by the product team.\u00a0From previous experience,\u00a0client returned timestamps are often notoriously noisy. We\u00a0would\u00a0highly\u00a0recommend\u00a0using\u00a0timestamps when\u00a0receiving\u00a0the log rather than the client-logged timestamps.\u00a0Below is an example\u00a0of how device recorded\u00a0event time (client time) can differ\u00a0drastically\u00a0from the\u00a0correct event time.<\/span>\u00a0<\/span><\/p>\n Figure 2: Distribution of the event time with device count<\/p>\n We\u00a0frequently slice\u00a0data\u00a0into subgroups for analysis.\u00a0We call these\u00a0subgroups\u00a0segments\u00a0and\u00a0we\u00a0define them at\u00a0various levels of aggregation.\u00a0Segments can be on\u00a0row-level or\u00a0if applicable\u00a0and\u00a0desired,\u00a0on\u00a0randomization unit\u00a0level.\u00a0For example,\u00a0if our\u00a0A\/B test is randomized on user level,\u00a0we\u00a0can have a date segment on\u00a0row-level\u00a0as well as\u00a0a\u00a0user age group segment on user level.\u00a0Some common segments include\u00a0date, market, browser, application version, etc.<\/span>\u00a0<\/span><\/p>\n Columns used as segments should be\u00a0with\u00a0the right format,\u00a0with acceptable missing rates\u00a0and of reasonable cardinality.\u00a0A heuristic reasonable range\u00a0of cardinality\u00a0is 2 to 10. One with a constant value provides no additional insight. One with hundreds of distinct values\u00a0may\u00a0result in\u00a0a broad range of issues such as\u00a0slow computation, low statistical power, and\/or loading or responsiveness issues with visualization.\u00a0Such\u00a0segments\u00a0can\u00a0create\u00a0even\u00a0a larger\u00a0number of subgroups,\u00a0especially if\u00a0we pair it\u00a0with another segment with high cardinality. If the segment column naturally comes with high cardinality (e.g., language), a commonly adopted practice is to truncate the list to the Top N with the highest frequencies and cast the rest to \u201cOthers\u201d.<\/span>\u00a0<\/span><\/p>\n Data quality changes as the product and data sources evolve over time. Adding new logs, accommodating new data sources, following new requirements for data governance, will all potentially impact data quality. There are multiple tools that we can use to help keep track of data quality measurements.<\/span>\u00a0<\/span><\/p>\n Below is a\u00a0real example\u00a0that illustrates how\u00a0the team\u00a0uses the first 3 tools\u00a0for monitoring.\u00a0The\u00a0dashboard\u00a0(the blue line)\u00a0shows the\u00a0value change\u00a0of a data quality metric\u00a0across time. And\u00a0the\u00a0red points highlight the anomalies\u00a0that\u00a0will trigger\u00a0the alerts\u00a0to the product team.\u00a0To ensure\u00a0the data quality of subgroups, the team also\u00a0sets\u00a0up the\u00a0same\u00a0dashboard\u00a0for segments such as\u00a0market\u00a0and\u00a0OS version.<\/span>\u00a0<\/span><\/p>\n <\/p>\n Figure 3 : Continuous data quality monitoring<\/p>\n High-<\/span>q<\/span>uality data is<\/span>\u00a0a<\/span>\u00a0fundamental building b<\/span>lock<\/span>\u00a0for trustworthy\u00a0<\/span>A\/B testing<\/span>\u00a0results.\u00a0<\/span>Data quality issues\u00a0<\/span>can\u00a0<\/span>produce<\/span> unreliable A\/B testing <\/span>results<\/span>. This can\u00a0<\/span>negatively impact\u00a0<\/span>A\/B testing<\/span>\u00a0analysis<\/span>\u00a0and decision<\/span>–<\/span>making.<\/span>\u00a0<\/span>By utilizing\u00a0<\/span>t<\/span>he checklists and the\u00a0<\/span>requirements for critical data fields<\/span>,\u00a0<\/span>we<\/span><\/span>\u00a0<\/span><\/span><\/span>get ready<\/span>\u00a0to implement checks on critical components of\u00a0<\/span>A\/B testing<\/span>\u00a0data.\u00a0<\/span>When\u00a0<\/span>we\u00a0<\/span>set up the data quality check monitors,\u00a0<\/span>we\u00a0<\/span>always\u00a0<\/span>adopt\u00a0<\/span>the four\u00a0<\/span>tools for continuous monitoring<\/span>\u00a0as mentioned above<\/span>.<\/span>\u00a0We hope\u00a0<\/span>this blog post\u00a0<\/span>can\u00a0<\/span>help\u00a0<\/span>you\u00a0<\/span>produce<\/span>\u00a0<\/span>the best-<\/span>qualified<\/span>\u00a0build<\/span>ing<\/span>\u00a0b<\/span>locks<\/span>\u00a0for your A\/B testing.\u00a0<\/span><\/span>\u00a0<\/span><\/p>\n <\/p>\n – Platina Liu, Wen Qin, Hao Ai, Jing Jin, Microsoft Experimentation Platform<\/em><\/p>\n <\/p>\n [1] A. Fabijan et al., \u201cDiagnosing Sample Ratio Mismatch in A\/B Testing.\u201d https:\/\/www.microsoft.com\/en-us\/research\/group\/experimentation-platform-exp\/articles\/diagnosing-sample-ratio-mismatch-in-a-b-testing\/<\/span>\u00a0<\/span><\/p>\n [2] R.\u00a0Kohavi, D. Tang, and Y. Xu, Trustworthy Online Controlled Experiments: A Practical Guide to A\/B Testing. Cambridge University Press, 2020.<\/span>\u00a0<\/span><\/p>\n [3] W.\u00a0Machmouchi, S. Gupta, and R. Zhang, \u201cPatterns of Trustworthy Experimentation: During-Experiment Stage.\u201d https:\/\/www.microsoft.com\/en-us\/research\/group\/experimentation-platform-exp\/articles\/patterns-of-trustworthy-experimentation-during-experiment-stage\/<\/span>\u00a0<\/span><\/p>\n [4]\u00a0Janhavie, \u201cData Quality.\u201d https:\/\/medium.com\/datacrat\/data-quality-dc4018fc443<\/span>\u00a0<\/span><\/p>\n [5] Lean-Data, \u201cHere is how to start with data quality.\u201d https:\/\/www.lean-data.nl\/data-quality\/here-is-how-to-start-with-data-quality\/<\/span>\u00a0<\/span><\/p>\n [6] T. Crook, B. Frasca, R.\u00a0Kohavi, and R. Longbotham, \u201cSeven pitfalls to avoid when running controlled experiments on the web,\u201d in Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining – KDD \u201909, Paris, France, 2009, p. 1105.\u00a0doi: 10.1145\/1557019.1557139.<\/span>\u00a0<\/span><\/p>\n","protected":false},"excerpt":{"rendered":" Introduction Data\u00a0plays a vital role in\u00a0the lifecycle of\u00a0every single product in the technology industry.\u00a0With\u00a0data,\u00a0we can generate insights\u00a0to improve products\u00a0and provide a superior customer experience. Yet insights are only actionable if they are derived from trustworthy data. For example, a navigation app relies on accurate location detection data to provide directions to users.\u00a0\u00a0 Data quality is […]<\/p>\n","protected":false},"author":39168,"featured_media":793637,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-content-parent":651963,"msr_hide_image_in_river":0,"footnotes":""},"research-area":[],"msr-locale":[268875],"msr-post-option":[],"class_list":["post-774094","msr-blog-post","type-msr-blog-post","status-publish","has-post-thumbnail","hentry","msr-locale-en_us"],"msr_assoc_parent":{"id":651963,"type":"group"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-blog-post\/774094","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-blog-post"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-blog-post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/39168"}],"version-history":[{"count":20,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-blog-post\/774094\/revisions"}],"predecessor-version":[{"id":794012,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-blog-post\/774094\/revisions\/794012"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/793637"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=774094"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=774094"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=774094"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=774094"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n
How can data quality impact our A\/B testing results?<\/h2>\n
Sample Ratio Mismatch<\/h3>\n
STEDI\u00a0of\u00a0metrics<\/h3>\n
Latency in analysis result delivery\u00a0and delayed\u00a0decision-making<\/h3>\n
![\"diagram,](\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2021\/09\/Data-quality-dimensions-1024x712.png\")
<\/p>\n<\/h2>\n
What are the critical data quality requirements for A\/B testing?<\/h2>\n
\n
Randomization\u00a0unit<\/h3>\n
Treatment\u00a0assignment<\/h3>\n
Timestamp<\/h3>\n
![\"Distribution](\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2021\/11\/Device-Count-Distribution-300x209.png\")
<\/p>\n<\/h3>\n
Segments<\/h3>\n
How can we continuously monitor data quality?<\/h2>\n
\n
![\"Data](\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2021\/11\/data-quality-monitoring.png\")
<\/p>\nSummary<\/h2>\n
References:<\/h2>\n