Week 5 — Market Research & Digital Analytics¶

Course Outcome CO3 — Design and execute market research to support e-commerce decisions; collect, analyze, and interpret digital analytics data to optimize performance across the customer journey.

Learning Objectives¶

By the end of this week you should be able to:

• [x] Distinguish primary and secondary research methods in digital commerce contexts
• [x] Design a survey instrument appropriate for an e-commerce research question
• [x] Explain A/B testing methodology, including statistical significance, sample size, and common pitfalls
• [x] Interpret core Google Analytics 4 metrics: sessions, users, bounce rate, engagement rate, conversion rate, and funnel analysis
• [x] Construct and explain UTM parameters for campaign tracking
• [x] Describe the capabilities and use cases of heatmap and session recording tools (Hotjar, Microsoft Clarity)
• [x] Apply basic keyword research methodology and interpret SEO metrics using industry tools
• [x] Identify and use social listening and competitive intelligence techniques
• [x] Calculate and interpret Net Promoter Score (NPS) and Customer Satisfaction Score (CSAT)
• [x] Explain cohort analysis and interpret a cohort retention table
• [x] Compare attribution models (first-touch, last-touch, linear, time-decay, data-driven) and their strategic implications

1. Primary and Secondary Research in Digital Commerce¶

1.1 The Research Hierarchy¶

All market research begins by distinguishing what you already know from what you need to find out, and how you'll find it.

Secondary research uses data that already exists — collected by someone else for a different primary purpose. Primary research generates new data specifically for your research question.

1.2 Secondary Research Sources for E-Commerce¶

Industry reports and market data: - Statista: Aggregates statistics from 22,500+ sources; e-commerce market sizing, benchmarks - eMarketer / Insider Intelligence: US and global e-commerce forecasts; paid subscription but often cited freely - Nielsen / NielsenIQ: Consumer panel data; retail market share; behavioral tracking - Forrester Research: B2B and B2C digital commerce research; enterprise technology adoption - McKinsey Global Institute: Macro-level digital economy analysis

Government data sources: - U.S. Census Bureau: Quarterly E-Commerce Report; Annual Retail Trade Survey - Bureau of Labor Statistics (BLS): Consumer Expenditure Survey; industry employment data - Federal Trade Commission (FTC): Consumer complaint data; enforcement trends - OECD Digital Economy Outlook: International comparative data

Platform-provided data: - Google Trends: Search volume trends; geographic interest patterns; keyword seasonality - Facebook Audience Insights: Demographic and interest profiles of Facebook users - Amazon Bestseller Lists: Product category demand signals - Shopify Partners: Annual merchant and consumer behavior reports

1.3 Primary Research Methods in E-Commerce¶

Qualitative methods (generate insights, hypotheses):

Quantitative methods (measure scale, patterns, statistical relationships):

• Surveys / questionnaires: See Section 1.4
• A/B testing: See Section 2
• Analytics instrumentation: See Section 3
• Conjoint analysis: Statistical technique to identify which product attributes consumers value most by analyzing tradeoff choices
• Eye-tracking studies: Hardware-based measurement of visual attention on page elements

1.4 Survey Design for E-Commerce¶

A well-designed survey is a powerful primary research tool. A poorly designed survey produces data that misleads decision-making.

Core survey design principles:

1. Clarity: Every question should be interpretable in exactly one way
2. Neutrality: Avoid leading questions ("How much did you enjoy our checkout experience?" presupposes enjoyment)
3. Single topic per question: Double-barreled questions ("Was the shipping fast and affordable?") cannot be analyzed cleanly
4. Appropriate scales: Likert scales (1–5 or 1–7) for attitude measurement; ensure scales are balanced with equal positive and negative options
5. Order effects: Put demographic questions last (reduces dropoff); put easy questions first to build completion momentum
6. Length: Completion rates drop precipitously after 7–10 minutes; target 3–5 minutes for unsolicited surveys

E-commerce specific survey types:

2. A/B Testing Methodology¶

2.1 What Is A/B Testing?¶

A/B testing (also called split testing) is a controlled experiment in which a randomly selected portion of users sees version A (the control, typically the existing experience) and another randomly selected portion sees version B (the treatment, a modified version). By comparing outcomes between groups, you can measure the causal effect of the change.

All Visitors
    │
    ├──── 50% → Version A (Control)  ──→ Measure: Conversion Rate A
    │
    └──── 50% → Version B (Treatment) ──→ Measure: Conversion Rate B

    If B > A by a statistically significant amount → Ship version B

2.2 Why A/B Testing Matters¶

A/B testing is the gold standard for optimization because it establishes causation rather than just correlation. Without it, you might observe that customers who see a particular page element convert more — but you can't tell whether the element caused the conversion or whether those customers were already more likely to convert for unrelated reasons.

2.3 Statistical Significance¶

The most misunderstood concept in A/B testing is statistical significance — the probability that an observed difference between A and B reflects a true difference rather than random chance.

Key concepts:

p-value: The probability of observing the measured difference (or larger) 
         if the null hypothesis (no true difference) were true.

p < 0.05: Less than 5% probability the result is due to chance
          → "Statistically significant at the 95% confidence level"

p < 0.01: Less than 1% probability → 99% confidence

Confidence Interval (CI): The range within which the true value
likely falls with a specified probability.
"Version B converts 3.2% better than A (95% CI: 1.1% – 5.3%)"

Sample size and test duration:

One of the most common A/B testing mistakes is stopping the test too early when early results look good (or bad). Early fluctuations are normal; you need sufficient data to distinguish signal from noise.

# Sample size estimation (simplified)
# Target: detect a 5% relative lift (e.g., 2.0% → 2.1% conversion rate)
# At 80% power and 95% significance

# Rule of thumb calculation:
# n = 16 × σ² / δ²
# Where σ² = baseline variance, δ = minimum detectable effect

# For conversion rates, using a calculator (e.g., Evan Miller's):
# Baseline: 2.0% conversion rate
# Minimum detectable effect: 10% relative lift = 0.2 percentage points
# Required sample per variant: ~38,000 visitors

# Always run tests for at least 2 full business cycle weeks
# to account for day-of-week effects

2.4 Common A/B Testing Pitfalls¶

2.5 What to Test in E-Commerce¶

High-impact areas for A/B testing, ranked by typical uplift potential:

1. Checkout flow: Reducing steps, removing form fields, offering guest checkout (often 10–20%+ conversion uplift)
2. Free shipping threshold: $25 vs. $35 vs. $50 vs. free (significant impact on AOV)
3. Product page layout: Hero image size, review placement, CTA button text/color/position
4. Homepage value proposition: Different headline copy, hero imagery
5. Email subject lines: Major impact on open rates (typically test batches before full send)
6. Pricing display: Charm pricing vs. round, bundle presentation, crossed-out original prices
7. Trust signals placement: Where trust seals, reviews, and guarantees appear

3. Google Analytics 4: Key Metrics and Features¶

3.1 GA4 vs. Universal Analytics¶

Google replaced Universal Analytics (UA) with Google Analytics 4 (GA4) as the default product in July 2023. GA4 represents a fundamental redesign:

3.2 Core GA4 Metrics¶

Step 1: Homepage view         100% (baseline)
Step 2: Product page view      45%  (-55% dropped)
Step 3: Add to cart            18%  (-27% dropped)
Step 4: Begin checkout         12%  (-6% dropped)
Step 5: Complete purchase       3%  (-9% dropped)

Overall conversion rate: 3%
Biggest drop: Homepage → Product page (opportunity: improve homepage CTA)

3.3 UTM Parameters and Campaign Tracking¶

UTM parameters (Urchin Tracking Module, from Google's acquisition of Urchin in 2005) are tags appended to URLs that tell GA4 where traffic came from.

The five UTM parameters:

Example tagged URL:

https://www.yourstore.com/sale?utm_source=facebook&utm_medium=cpc
&utm_campaign=spring_sale_2025&utm_content=carousel_ad_v2

Best practices: - Create a UTM naming convention document and enforce it organization-wide (inconsistent naming creates messy data: Facebook, facebook, and FB appear as three different sources) - Always tag paid traffic; never tag organic or direct traffic (GA4 handles these automatically) - Use a UTM builder tool (Google's Campaign URL Builder, or UTM.io) to avoid typos - Tag all email links, social posts, and paid ad destination URLs

4. Heatmaps and Session Recordings¶

4.1 What Heatmap Tools Reveal¶

Heatmap tools visually represent user behavior on a page using color gradients (hot = high activity, cool = low activity) for:

• Click heatmaps: Where users click (including "rage clicks" — repeated frustrated clicking on non-interactive elements)
• Move heatmaps: Where users move their mouse (correlated with visual attention)
• Scroll heatmaps: How far down the page users scroll before leaving

Session recordings capture anonymized video replays of individual user sessions, showing exactly what the user did — every click, scroll, mouse movement, and form interaction.

4.2 Hotjar¶

Hotjar (hotjar.com) is the most widely used behavior analytics platform:

• Pricing: Free tier (35 daily sessions); paid from $32/month (100 daily sessions)
• Features: Heatmaps, session recordings, surveys (on-site popups and polls), feedback widgets

E-commerce use cases for Hotjar:

4.3 Microsoft Clarity¶

Microsoft Clarity (clarity.microsoft.com) is a free behavior analytics tool (no session limits):

• Free: Unlimited sessions, heatmaps, recordings
• Unique features:
• Dead clicks: Clicks that produce no action (elements that look clickable but aren't)
• Rage clicks: Multiple rapid clicks indicating frustration
• Excessive scrolling: Users scrolling up and down repeatedly, suggesting confusion
• Quick backs: User navigated to a page then immediately left, suggesting the content wasn't what they expected

5. Keyword Research and SEO Basics¶

5.1 Why SEO Matters for E-Commerce¶

Search Engine Optimization (SEO) is the practice of optimizing web content to rank higher in organic (non-paid) search results. Organic search is one of the highest-quality and lowest-cost acquisition channels for e-commerce:

• 53% of all website traffic comes from organic search (BrightEdge, 2023)
• The top 3 organic results capture 54% of all clicks on a search results page
• Organic traffic has a 2–5× higher conversion rate than paid traffic (lower intent mismatch)
• Once established, organic rankings provide traffic without per-click costs

5.2 Keyword Research Process¶

Keyword research identifies the specific search queries your target customers use, enabling you to create content that matches their intent.

Step 1: Seed keyword generation Start with broad terms related to your products and business. For an online running shoe store: - "running shoes" - "trail running shoes"
- "women's marathon shoes"

Step 2: Expand using research tools

Step 3: Evaluate keywords

5.3 Search Intent Framework¶

Understanding search intent is the most important keyword evaluation dimension:

6. Social Listening and Competitive Intelligence¶

6.1 Social Listening¶

Social listening is the practice of monitoring digital channels for mentions of your brand, competitors, products, and relevant industry topics.

What social listening reveals: - Customer sentiment: How do customers feel about your products and service? - Competitive intelligence: What are customers saying about your competitors' weaknesses? - Emerging trends: What topics are gaining traction in your category? - Crisis detection: Identifying PR crises early before they escalate - Influencer identification: Who talks about your category with authentic authority?

Social listening tools:

Conducting a social listening audit:

1. Define keywords: Brand name + misspellings, product names, competitors, category terms
2. Set up monitoring across: Twitter/X, Reddit, Instagram, TikTok, Facebook, YouTube comments, review sites (Trustpilot, G2, Amazon reviews)
3. Track sentiment over time: Positive/negative/neutral ratios
4. Create response protocols: Who responds? In what timeframe? With what tone?

6.2 Competitive Intelligence Techniques¶

Competitive intelligence (CI) is the systematic process of gathering, analyzing, and using information about competitors.

7. Customer Satisfaction Metrics¶

7.1 Net Promoter Score (NPS)¶

Net Promoter Score (NPS), developed by Fred Reichheld at Bain & Company (2003), measures customer loyalty through a single question:

"On a scale of 0–10, how likely are you to recommend [Company/Product] to a friend or colleague?"

Scoring methodology:

Responses 0-6 = Detractors (unhappy customers who may damage the brand)
Responses 7-8 = Passives (satisfied but not enthusiastic)
Responses 9-10 = Promoters (loyal enthusiasts who recommend actively)

NPS = % Promoters − % Detractors
Range: −100 to +100

NPS benchmarks by industry (Qualtrics XM Institute, 2023):

NPS follow-up questions dramatically increase the actionability of the metric: - "What is the primary reason for your score?" (open text) - "What could we do to improve your experience?" (open text for detractors)

7.2 Customer Satisfaction Score (CSAT)¶

CSAT measures satisfaction with a specific interaction or transaction (rather than overall loyalty like NPS).

Standard CSAT question:

"How satisfied were you with [specific interaction/product]?" Response scale: Very Satisfied / Satisfied / Neutral / Dissatisfied / Very Dissatisfied

Calculation:

CSAT = (Number of Positive Responses / Total Responses) × 100

Where "positive" = Satisfied + Very Satisfied responses

CSAT use cases in e-commerce:

NPS vs. CSAT comparison:

8. Cohort Analysis¶

8.1 What Is Cohort Analysis?¶

Cohort analysis groups users by a shared characteristic — typically the time period when they first purchased or registered — and tracks their behavior over subsequent periods. It is the most powerful tool for understanding customer retention and the long-term impact of acquisition channels and campaigns.

Why cohort analysis matters: Aggregate metrics can be misleading. If your month-over-month revenue is flat, you can't tell from the aggregate whether (a) customer quality is constant, (b) customers are churning faster but you're acquiring more to compensate, or (c) something else entirely. Cohort analysis separates these explanations.

8.2 Cohort Retention Table¶

          Month 0  Month 1  Month 2  Month 3  Month 4  Month 5
Jan cohort  100%     45%      28%      22%      18%      16%
Feb cohort  100%     42%      26%      21%      17%       —
Mar cohort  100%     48%      30%      24%       —        —
Apr cohort  100%     51%      33%       —        —        —
May cohort  100%     53%       —        —        —        —

How to read the table: 100% of customers in each cohort are active in Month 0 (their first purchase month). By Month 1, the January cohort has 45% retention — 55% of January customers did not make a second purchase in February. By Month 5, only 16% remain active.

Key observations from this table: - Recent cohorts (Apr, May) show higher Month 1 retention (51–53% vs. 42–45%) — suggesting that a product change or marketing improvement is working - The biggest drop is Month 0 → Month 1 in all cohorts, which is typical; this is where the biggest retention opportunity lies (onboarding optimization)

8.3 Interpreting and Acting on Cohort Data¶

9. Funnel Visualization and Attribution Modeling¶

9.1 Funnel Visualization¶

Funnel visualization maps the percentage of users who complete each step in a defined conversion path. In GA4, this is done through the Funnel Exploration report.

A full e-commerce funnel:

                    Visitors       100,000
                        │
                        ▼
              Category page visits   48,000  (48%)
                        │
                        ▼
               Product page views    22,000  (22%)
                        │
                        ▼
                  Add to cart         7,000   (7%)
                        │
                        ▼
              Begin checkout          4,200   (4.2%)
                        │
                        ▼
          Enter payment information   2,800   (2.8%)
                        │
                        ▼
              Purchase complete       2,100   (2.1%)

Reading the funnel: The biggest percentage drop determines where to focus optimization effort. In this example, the Category → Product page transition loses 26 percentage points (54% of category visitors don't click into a product). This suggests either poor product imagery in category view, irrelevant product mix, or bad search intent match.

9.2 Attribution Modeling¶

Attribution modeling determines how to assign credit for a conversion across the multiple touchpoints a customer encountered before purchasing.

The challenge: A customer may: 1. See a Facebook ad → not purchase 2. Google the brand organically → not purchase 3. Receive an email → not purchase 4. Click a Google Shopping ad → purchase

Who gets credit? All four channels contributed. Attribution models answer this question differently.

9.3 Attribution in Practice¶

Choosing a model by decision type:

Key Vocabulary¶

Review Questions¶

Further Reading¶

Summary¶

Market research and digital analytics are the evidence base upon which all e-commerce strategy rests. The shift from intuition-driven to data-driven decision-making — enabled by the rich behavioral data that digital commerce generates — is one of the most significant competitive advantages available to modern businesses.

But data creates its own risks: the temptation to over-optimize for measurable metrics ("what gets measured gets managed") can cause under-investment in brand, customer relationships, and long-term equity that don't show up immediately in conversion rate dashboards. The most effective digital marketers combine quantitative rigor — A/B testing, cohort analysis, attribution modeling — with qualitative understanding of customer needs, motivations, and experiences. Neither alone is sufficient.

As you build your analytical toolkit, cultivate the habit of asking "why" before reaching for the "optimize" lever. The numbers tell you what is happening; your research skills help you understand why — and that understanding is what enables genuinely good decisions.

← Week 4 — Consumer Behavior & Digital Psychology | Course Index