Product Experiment · CRM return journey
What happens after a customer comes back?
Customers returning after CRM outreach spent just 20 to 30 seconds in the app. I recommended testing the return experience while CRM kept email and push outreach consistent.
Read the case
Audience and assignment
I clustered customers by recency, frequency and engagement using K-means. One previously active group had gone 30 days without opening the app. On their first eligible return, we randomized customers 50/50 to the existing personalized experience or the same experience with a Spotlight prompt. Each customer kept their assignment.
Readout and decision
Valid-lead conversion reached about 2.3% against a 1.5% baseline. The six-week test was designed to detect a one-percentage-point lift. I treated the observed 0.8-point difference as directional. Regular-listing click-through fell, so Product and Analytics reviewed browsing and quality measures as exposure expanded to 80%, then 100% of eligible returners. The test supported a rollout decision for the bundled experience. It could not isolate personalization or the effect of outreach.
Product Experiment · Search ranking
Would ranking for lead intent improve search?
A listing-level comparison showed no clear increase in lead submission as click-through rose. That gave us a reason to test a different ranking objective and a reason to keep watching clicks.
Read the case
Model and experiment
I defined the valid-lead target and designed the user-level randomized A/B test. ML owned training and deployment. Historical listing exposures and lead outcomes produced one propensity score per active listing within its market. With Engineering, I checked that impression, click and lead events reflected the order customers actually saw.
Readout and decision
After four weeks, click-through was down and lead submission trended up. The lead effect was statistically inconclusive. The click decline stayed within the agreed tolerance and page views per customer held steady. I recommended keeping the reversible change with 30-, 60- and 90-day value reviews. Product and Marketing retained the ranking and continued iterating.
Causal Inference · SEM incrementality
Would more paid search create demand we would otherwise miss?
New-construction visitors were progressing from browsing to inquiries. Before recommending more non-branded search spend, I wanted to know how many additional leads it would generate across channels, including organic.
Read the case
Choose the market as the unit of intervention
I built a ZIP-by-week panel and matched roughly 100 ZIP codes into 50 pairs using historical leads, traffic and available inventory. I checked balance and pre-test trends, then randomized one ZIP in each pair to receive about 20% more spend. The other continued at its usual level.
Estimate incremental demand
My primary estimate was a preregistered difference-in-differences comparison of total leads. I used CausalImpact as a robustness check and examined spend delivery, organic displacement and lead quality before interpreting the result.
Change the mix, then review
The estimated lift was positive but statistically inconclusive. I recommended shifting the existing search mix toward non-branded terms while monitoring cost, quality and market capacity. Marketing changed the allocation. A larger total budget would require stronger evidence of incremental return.
Modeling · Marketplace value
When more customers reach out, who benefits?
A customer contacting an agent signals intent. The agent still has to decide where to spend time. I helped our teams evaluate both perspectives before expanding a change that generated more inquiries.
Read the case
Measure each side of the exchange
On the customer side, I examined whether people progressed from browsing to a valid inquiry. On the agent side, I estimated the value of the opportunities those inquiries created. In Snowflake, I aggregated opportunity value across each experiment group and divided by everyone assigned, including customers who never submitted an inquiry. That kept the comparison meaningful even when the change affected who reached out.
Use disagreement to guide the decision
I compared each outcome with its own control baseline. If inquiries increased while expected value per assigned customer fell, my recommendation was to examine which listings gained exposure before expanding the change. Marketing used the agent-side measure to refine targeting. An inquiry showed customer intent, and estimated opportunity value helped assess the agent’s prospects. Neither established that a transaction closed.
Embracing AI · Analytics assistant
Which parts of a KPI investigation should an assistant own?
Analysts repeatedly rebuilt the same weekly comparisons before investigating a change. I built the reusable measurement layer and the rules for when a person should take a closer look.
Read the case
Data and review workflow
Versioned dbt models defined the grain, time window and denominator for each KPI. Historical ranges flagged unusual movement. Freshness and tracking checks came first. The assistant drafted a summary with the observation, plausible explanations, next check and responsible team. Analysts reviewed root-cause claims and business recommendations.
Pilot and evidence
We reconciled outputs with BI dashboards and Amplitude, then compared assistant drafts with manual analyst readouts across verticals. Noisy alerts and missing context informed threshold and routing revisions. Usage logs showed roughly 80% to 90% of the analytics team had tried the workflow. Pilot comparisons and team feedback suggested around 30% less preparation and coordination effort. That figure was an operational estimate.