Senior Software Engineer, Test

Job description

Teamwork makes the stream work. Roku is changing how the world watches TV Roku is the #1 TV streaming platform in the U.S., Canada, and Mexico, and we've set our sights on powering every television in the world. Roku pioneered streaming to the TV. Our mission is to be the TV streaming platform that connects the entire TV ecosystem. We connect consumers to the content they love, enable content publishers to build and monetize large audiences, and provide advertisers unique capabilities to engage consumers. From your first day at Roku, you'll make a valuable - and valued - contribution. We're a fast-growing public company where no one is a bystander. We offer you the opportunity to delight millions of TV streamers around the world while gaining meaningful experience across a variety of disciplines. What does the team work on? Our Vertical Search team's mission is to deliver the best TV search experience: fast, comprehensive, and reliably relevant. For REA Vertical Search, that means helping users discover the right content across typed, voice, and assistant-driven experiences while supporting a growing set of subscription services and Roku Channel discovery surfaces. We partner with search engineers, ML engineers, product managers, and data partners to define quality strategy for relevance-sensitive experiences, build scalable automation and evaluation frameworks, and apply agentic automation to keep pace with expanding catalog breadth and use cases. What is the role? We are looking for a Lead, Search Relevance Quality to work closely with the Roku Vertical Search Team to define, measure, and improve Search and Assistant relevance quality. This person will lead quality strategy for relevance-sensitive experiences, build scalable automation and evaluation frameworks, and apply agentic automation to keep pace with expanding catalog breadth and use cases. The role requires balancing automated quality systems with pragmatic human-in-the-loop review where needed until confidence and coverage are high. Quality for Vertical Search will be measured with clear, trusted metrics rather than inferred only from downstream engagement signals. Search and Assistant launches will have reusable relevance-evaluation coverage, stronger release confidence, and faster debugging loops. The team can scale quality across a growing number of subscription services and Roku Channel experiences without linear growth in manual effort. How will I use AI at Roku? At Roku, AI agents do most of the keystroke-level coding. Engineers act as the technical leads for those agents, setting context, planning work, verifying outputs, and recovering when the agent goes off course. If you are excited about working this way, we want to talk to you. At Roku, we're embracing AI as a powerful tool to amplify human creativity, accelerate innovation, and deliver better results for our customers. Across teams, we're exploring how AI can enhance our work and we're only looking for people who are curious, adaptable, and excited to grow alongside these technologies. What are the responsibilities of the role? • p]:inline" data-streamdown="list-item">Define the quality strategy for Vertical Search and Assistant relevance across Roku Premium subscription services, Roku Channel search journeys, and related discovery experiences • p]:inline" data-streamdown="list-item">Build scalable frameworks to measure search quality across query understanding, retrieval, ranking, result presentation, and assistant response quality • p]:inline" data-streamdown="list-item">Create automated evaluation pipelines for offline, pre-launch, and continuous post-launch quality monitoring using modern test infrastructure and agentic workflows • p]:inline" data-streamdown="list-item">Partner with search engineers, ML engineers, product managers, and data partners to define KPI hypotheses, guardrails, and learning plans for quality-related launches and experiments • p]:inline" data-streamdown="list-item">Develop quality signals and gold-standard datasets for vertical search, including subscription-service discovery, entity matching, ranking relevance, and assistant answer quality • p]:inline" data-streamdown="list-item">Use agentic automation to generate, maintain, and expand test coverage and evaluation assets while ensuring governance, traceability, and reliability of the resulting workflows • p]:inline" data-streamdown="list-item">Introduce the right mix of manual review and automation for ambiguous or high-impact experiences until the automated system is proven and trusted • p