How to Develop an Effective Data Collection Method for Root Cause Analysis

©Data collection_ Are you doing it right_ yes or no/Shutterstock.com

Introduction

Initiating a root cause analysis can be derailed by the unexpected discovery of insufficient data. This is a frequent challenge encountered by nascent analysts, aligning with the adage that a significant portion of an analyst's time is devoted to data preparation rather than analytical interpretation. This phenomenon is particularly pronounced when a critical metric, such as conversion rate, experiences an abrupt decline, creating a heightened sense of urgency. This article presents a pragmatic framework for establishing an effective data collection methodology specifically tailored to root cause analysis within the domains of product, web, and marketing analytics. The primary objective is to empower analysts to identify and acquire the necessary data from the outset, thereby mitigating the risk of unproductive time expenditure.

Method of Data Collection

The majority of analytics teams maintain dashboards for the ongoing monitoring of key performance indicators (KPIs). These dashboards typically leverage pre-existing data pipelines. While it may be tempting to utilize this readily available data for root cause analysis (i.e., determining the factors contributing to a metric's deviation), it is crucial to recognize that such data is primarily designed for tracking purposes and may not provide the granular detail necessary for in-depth investigations. Consequently, the collection of supplementary data often becomes imperative.

When embarking on a root cause analysis, the need to gather novel data frequently arises. A straightforward approach to this process entails the following steps:

• Define Data Requirements: In the initial stage, it is essential to articulate the specific information required to comprehend the underlying issue. Formulate precise questions that demand answers. For instance, if a decline in conversion rates is observed, it may be necessary to acquire data pertaining to user behavior both preceding and subsequent to the observed change.

• Develop a Collection Plan: Subsequently, determine the most effective methodology for data acquisition. This may involve the utilization of existing tools, the implementation of new tracking mechanisms, the administration of surveys, or the exploration of alternative data sources. It is crucial to approach this step with specificity.

• Identify Data Sources: Pinpoint the precise origin of the required data. Potential sources may encompass website databases, application logs, marketing platforms, or other relevant repositories.

• Assign Responsibilities: Lastly, designate individuals responsible for the data collection process. This proactive measure serves to minimize confusion and ensure the timely completion of data acquisition activities.

To facilitate the planning of data requirements for your root cause analysis, please refer to the data collection worksheet provided below.

Data Collection Goal:

Data Collection Guide Questions	Data for Questions	List data for questions
Who did what, and when?	You can find this data in the task and activity records.
How is this different from before?	Collect data on how the process is supposed to work and how it actually works
What could have been done to prevent this?	Data on barriers to success can reveal prevention opportunities.
What events preceded this outcome?	Collect data on what happened and what the effects were.
Who knows what happen?	Witness/Participant Data
Why did they do that?	Contextual Factors

How to Collect Data

For each data point you need, describe how you will get it. Be specific about the tools or methods you will use (e.g., "Use Google Analytics to check pageviews," "Review server logs for error messages," "Send out a user survey").

Data Sources

For each data point you need, list where you will find it. Be specific about the tools, databases, or systems you will use (e.g., "Google Analytics," "website server logs," "marketing automation platform," "user survey database").

A Real-World Example

Consider a scenario where a newly appointed growth analyst is tasked with augmenting sign-up rates for a monthly data analytics coaching platform. Upon observing a decline in website sign-ups, the analyst conducts preliminary investigations to confirm the veracity of this observation. Subsequently, the analyst proceeds to employ a data collection plan to ascertain the underlying causes of this decline.

Data Collection Goal: Gather data to explain the sign-up rate drop.

Data Collection Guide Questions	Data for Questions	List data for questions
Who did what, and when?	You can find this data in the task and activity records.	Data on User Behavior (What was done): Pageviews & Navigation: Navigation paths, entry/exit pages, Pageviews per Step, Unique Pageviews. Time on Page: time spent, time spent on specific fields, time to submit. Form Interactions: Form starts, submissions, completion/abandonment field-specific data, autofill use, number of attempts, field value changes. User Engagement: Scroll depth, mouse movement Data on User (Whom): How users arrived at the sign-up page (e.g., search engine, social media) Device used to access the sign-up page (e.g., desktop, mobile). Browser used for sign-up (e.g., Chrome, Safari). User's location when accessing the sign-up page (e.g., US, UK). First-time or repeat visit to sign-up page. Data on Timestamp (When) on sign-up.
How is this different from before?	Collect data on how the process is supposed to work and how it actually works	Collect Data on Ideal Process(How is Supposed to Work):  Document the ideal user flow (steps, pages). Note expected time and conversion rates for each step. Record system requirements. Collect Data on Actual Performance (What's Happening): Watch user sessions or analyze navigation data to see their actual path. Measure the time and completion rate at each stage of the process. Check system performance (load times, errors).
What could have been done to prevent this?	Data on barriers to success can reveal prevention opportunities.	Potential data on potential barriers users faced during the sign-up process: Form Complexity, Mobile Experience, Instructions and Clarity, Password Requirements, Progress Indicators, Website Errors/Bugs, Loading Times, Browser/Device Incompatibility, Privacy/Security Concerns, Trust/Credibility, Perceived Value, Competitor Actions, Market Trends/Seasonality
What events preceded this outcome?	Collect data on what happened and what the effects were.	Website Changes: Event: Redesign/updates. Result: Increased bounce rate, longer form times. Event: New form. Result: Increased abandonment, specific errors. Event: Maintenance/downtime. Result: Lost sign-ups, negative mentions. Marketing Changes: Event: New campaign. Result: Initial traffic spike, low conversions. Event: Campaign changes. Result: Decreased click-throughs changed traffic sources. Event: Promo ended. Result: Expected, but potentially steeper, drop. External Factors: Event: Competitor launch. Result: Decreased traffic. Event: Negative news. Result: Decreased interest/sign-ups. Event: Search algorithm change. Result: Drop in organic traffic. Technical Issues: Event: Website errors/bugs. Result: Increased support tickets, negative reviews. Event: Slow load times. Result: Increased bounce rate, shorter sessions.
Who knows what happen?	Witness/Participant Data	Key Stakeholders: Marketing, Web Dev, Product, Customer Support, Sales (if applicable), Users.
Why did they do that?	Contextual Factors	Survey Data: Open-ended responses regarding motivations, frustrations, and reasons for abandonment. User Testing Data: Observations of user interactions, noting usability issues and areas of confusion. Support Logs: Records of user problems and questions related to sign-up. Social Media Data: Mentions and discussions related to the sign-up process and brand sentiment.

How to Collect Data

• Analytics Platforms: Use tools like Google Analytics, marketing/ad platforms, and social media analytics to track user behavior, campaign performance, and website traffic.

• User Research: Conduct surveys, user testing, and monitor user feedback (support tickets, feedback forms, social media).

• System Monitoring: Review system logs, error reports, and performance monitoring tools to assess technical performance.

• Business Records: Analyze sales/financial data, project management tools, and meeting minutes for internal context.

• External Research: Consult market research reports, industry benchmarks, and economic data for external context.

Data Sources

• Website: Analytics (Google Analytics, heatmaps, session recordings, server logs).

• Marketing: Automation/ad platforms, social media, CRM.

• User Feedback: Surveys, user testing, support tickets, feedback forms, social media.

• Technical: System logs, error/performance monitoring, version control.

• Business: Sales/financial data, project management tools, meeting minutes.

• External: Market research, benchmarks, economic data.

• Primarily interviews/discussions, supplemented by: Meeting minutes, communication records.

This example demonstrates a data collection planning model for root cause analysis in digital analytics. It can also be adapted for marketing, product analytics, and other digital metrics.

Application Across Disciplines

A diverse array of professions across various sectors employ data collection methodologies to effectively identify the root causes of encountered problems. Notable examples include:

• Product Development: Product Managers, User Experience (UX) Researchers, Quality Assurance (QA) Engineers, and Operations Managers.

• Marketing: Marketing Analysts, Marketing Managers, and Digital Marketing Specialists.

• Web Development & Operations: Web Analysts, Developers, Engineers, and Site Reliability Engineers (SREs).

• Data Science: Data Scientists and Machine Learning Engineers.

• Cross-disciplinary Applications: Business Analysts, Data Analysts, Operations Research Analysts, Project Managers, Process Improvement Specialists, and Quality Control Specialists.

Critical Considerations for Data Collection

Several key considerations must be borne in mind during the data collection process:

• Temporal Dimension: Close attention should be paid to the temporal sequence of events. Analyzing the chronological occurrence of events facilitates the identification of temporal patterns and the assessment of changes over time.

• Contextual Analysis: A comprehensive understanding of the broader context surrounding the event is crucial. Analyzing the situational factors surrounding the event can unveil hidden causal relationships that may not be readily apparent from the data alone.

• Exploration Beyond Conventional Sources: Data collection should not be limited to traditional databases. Investigating alternative sources such as spreadsheets, charts, logs, and even video recordings may yield valuable insights and uncover crucial clues.

• Human Interaction: Conducting interviews with individuals directly impacted by the event can provide invaluable qualitative insights that may not be readily discernible from quantitative data. It is advisable to conduct these interviews promptly following the occurrence of the event to ensure the accuracy and freshness of the information.

• Historical Data Integrity: When utilizing historical data, it is imperative to thoroughly investigate its storage methods and consult with individuals responsible for its management. This diligent approach will help to mitigate the risk of data misinterpretation and ensure the reliability of subsequent analyses.

Conclusion

In summary, a focused data collection strategy, encompassing defined requirements, source identification, and clear responsibilities, is crucial for accurate and efficient root cause analysis across diverse applications.

Data Analytics Training Resources

Analysts Builder

Master key analytics tools. Analysts Build provides in-depth training in SQL, Python, and Tableau, along with resources for career advancement. Use code ABNEW20OFF for 20% off. Details: https://www.analystbuilder.com/?via=amara