Inclusive Search — Kate Crotty

Quick summary

We ran a six-week study for a global video-sharing platform to define search inclusion in terms its product and engineering teams could build against. Evidence from 220 users across 600+ search journeys drove a global change to the search model, shipped to over two billion users, and product, design, and engineering now share the framework and taxonomy we built.

Read this study for

Designing for viewers and creators

Research that changes models

Search inclusion framework and taxonomy

Workshop and diary activities

The engagement

We were engaged by the culture team of a global video-sharing platform to define search inclusion in terms its product and engineering teams could build against.

While there was a general organizational commitment to DE&I, teams on the ground lacked the tools to operationalize it. With no way to test algorithms or measure search bias, they couldn’t prevent users from getting inconsistent search results.

Our goals:

1Give product and engineering a clear, usable taxonomy of search inclusion and search exclusion they could build, test, and experiment against.

2Design a research framework grounded in the perspectives of marginalized users by directly recruiting and engaging with intersectional communities.

The study in numbers

Study duration

Diary panel

Diary missions

Indexed search journeys

The panel

Intersectional communities including: Black, AAPI, Latino/a, Queer, and Trans/Non-Binary.

Our timeline

Prepare & frame Discover & track Co-create & test Synthesize & map

W1 W2 W4 W6

User panel

Recruit & screen Online diary · N=220 · 47 missions

Live sessions

3 focus groups (n=15) & IDIs

Creators & synthesis

Creator residency Taxonomy & library

The methods

Online diary study

N=220 · four weeks · 47 missions / short activities

Focus groups & interviews

3 groups (n=15) + 1:1 in-depth interviews

Creator residency

Co-creation workshop with platform creators / influencers

Method 01 — Online diary study

Over four weeks, 220 participants completed missions on a secure diary platform that allowed them to take pictures, videos, and engage in online worksheets, capturing behavior in real time.

Week 1 — DefineWhat belonging and othering feel like online, in metaphor and collage
Week 2 — NoticeLog organic moments of inclusion and exclusion
Week 3 — SearchRun set queries on the platform and submit screen recordings, autofills, and reactions to the results
Week 4 — SortCard-sort the search journey by emotional friction and impact

Method 02 — Focus groups & interviews

Three 90-minute groups (n=15) and 1:1 interviews validated the diary findings in a high-trust, in-group space where people could share vulnerable narratives about online marginalization.

Format3 moderated focus groups · multiple 1:1 IDIs

Method 03 — Creator residency

A half-day co-creation workshop with platform stakeholders and US creators to test our emerging frameworks.

What we set out to answer

What search inclusion means, in terms a product and engineering team can actually build.

What search exclusion looks like when it happens.

How a person’s inclusion needs shift depending on what and why they are searching.

The creator residency

We had the opportunity to plan and facilitate a workshop with the platform’s creator residency, a group of content creators / influencers they engage on a regular basis to test new ideas and features. We used the time together to test our emerging frameworks and taxonomy against real creator experience and map where the platform should step in.

A remote creator residency working session with creators and the research team

Activity 1

Creators shared their own search experiences

Activity 2

Creators ran real searches and logged what failed

Activity 3

We audited how those failures affected creators

Activity 4

Creators mapped fixes onto the search journey

Our deliverables

Query story library600+ real search journeys, each indexed by demographics, query type, met and unmet needs, and exclusion type.
Search inclusion taxonomyA standardized way to score a results page against the six needs, so teams can measure inclusion.
Search inclusion frameworkThe dimensions that comprise an equitable, inclusive search experience.

Part four

Our impact

Our insights drove a global model change, which the platform integrated directly into its core search architecture and shipped as a public product update to over two billion users.

The search organization now shares a standardized taxonomy and insights repository for search inclusion and exclusion.

Lessons

Consider in-group moderation for sensitive research

To engage with transgender participants, we partnered with a woman-owned research vendor and had one of their transgender moderators lead one of the focus groups. This shared lived experience established an immediate sense of safety, unlocking deep, candid insights that one of us likely wouldn’t have reached.

Test with the whole ecosystem

A lot of what we learned came from testing with the whole ecosystem: creators and viewers. A change that helps viewers can hurt creators, and you only really see those trade-offs when you have every perspective in front of you at once. Some of our richest findings came straight out of the tension between them.

The core team

Program LeadDomain direction and delivery
Polycultural SMEResearch materials, guides and outputs guidance
Equity-Centered Research Lead my roleStudy design, method architecture, moderation, synthesis, client delivery
Inclusive Design LeadEquity framing and visual design, workshop development