Evaluate an A/B test of showing bank website URLs in a sign-in widget

BankBridge makes Connect, a widget that lets people sign in to developer apps with their banking credentials. A user opens Connect, searches for their bank, selects it, and logs in; at that last step (the handoff) the data is passed to the developer without the user’s bank credentials ever being exposed. The flow can fail at any step, and a user who hits a failure has to start over.

The team is considering showing each bank’s website URL next to its name in the search results, and wants to know whether that changes conversion: the rate at which users successfully open Connect, search for their bank, select it and log in. To find out, it ran an A/B test. Treatment A is the current Connect; in treatment B, users see the bank URLs in the search results.

The data

• users — one row per user. client_id is the developer app the user came from; institution_id is the user’s bank.
• sessions — one row per Connect session; user_id links to users.id. A user can have more than one session.
  • product: what the developer app requested from the user’s bank, auth or connect.
  • screen_size: the device screen size the session ran on.
  • treatment_type: A (current widget) or B (bank URLs shown in search results).
• events — one row per step attempted in a session; session_id links to sessions.id.
  • event_name: open, search, select or handoff (the final login step).
  • event_status: success or failure.
  • timestamp: when the step happened.

Your task

1. Define conversion precisely. Say what the unit is (session or user), what counts as a success at each step, how you treat retries after a failure and users with several sessions, and why your definition is the one that matters for BankBridge and its developers.
2. Before reading the results, check whether the test can be trusted. Look at how users and sessions were assigned to A and B (balance across developer apps, banks, products and screen sizes; whether any user saw both versions), at the event sequences themselves (steps out of order, missing steps, duplicates), and at anything else in the data that concerns you. Say how each issue affects the analysis and what you did about it.
3. What effect did showing the URLs have? Estimate the change in overall conversion and at each step of the funnel, with a measure of uncertainty that fits the unit of analysis. Explain where in the flow you would expect the URLs to matter, and whether the data agrees.
4. Do the effect and the funnel itself vary by segment, for example by product, screen_size, developer app or bank? Describe the patterns or relationships you find, and say which differences you trust and which could be noise from slicing the data many ways.
5. What do you recommend: ship the URLs, drop them, or keep testing? Justify it, say what result would change your mind, and describe what you would measure or change in a follow-up test.

What to deliver

A short report for the Connect product manager that makes the case for your recommendation, with an appendix showing the queries or code behind each number.

All company names in this question are fictional.