AI Evaluation Program Manager

WHO WE ARE: At Twelve Labs, we are pioneering the development of cutting-edge multimodal foundation models that have the ability to comprehend videos just like humans do. Our models have redefined the standards in video-language modeling, empowering us with more intuitive and far-reaching capabilities, and fundamentally transforming the way we interact with and analyze various forms of media. With a remarkable $107 million in Seed and Series A funding, our company is backed by top-tier venture capital firms such as NVIDIA’s NVentures, NEA, Radical Ventures, and Index Ventures, and prominent AI visionaries and founders such as Fei-Fei Li, Silvio Savarese, Alexandr Wang and more. Headquartered in San Francisco, with an influential APAC presence in Seoul, our global footprint underscores our commitment to driving worldwide innovation. We are a global company that values the uniqueness of each person’s journey. It is the differences in our cultural, educational, and life experiences that allow us to constantly challenge the status quo. We are looking for individuals who are motivated by our mission and eager to make an impact as we push the bounds of technology to transform the world. Join us as we revolutionize video understanding and multimodal AI. ABOUT THE ROLE: You will be a vital member of our ML Data Team – which leads the full spectrum of video-language data preparation and model evaluation. This role comes with high ownership and includes responsibilities such as defining dataset needs and requirements in consultation with our research and product teams; designing and building data pipelines; and driving our post-training model evaluation strategy. You will also be responsible for automating as much of the repetitive partnership, annotation, and quality evaluation work as possible. A desire to work cross functionally and to build relationships is critical for success in this position. YOU WILL: - Model Evaluation: Design and build robust model evaluation frameworks, automating repetitive processes and maintaining a balanced approach to efficiency and depth in obtaining evaluation metrics and feedback. - Portfolio Monitoring: Manage resource allocation and timelines, adjusting direction flexibly based on real-time information across all data streams in your product vertical. - External Partner Collaboration: Enhance dataset and process quality through seamless collaboration with vendors and outsourcing partners. - Data Quality & Tooling Advancement: Establish labeling guidelines, monitor data quality, and improve tools and infrastructure to build a sustainable data operations framework. - Internal Collaboration: Partner with Engineering and AI Model teams to align on top priority data needs, design tools such as analytical reports and dashboards, and clearly communicate project progress. YOU MAY BE A GOOD FIT IF YOU HAVE: - 5+ years of experience working in an AI focused data operations organization. - A proven track record designing and executing large scale data or evaluation projects, including gathering, labeling, and post-processing data. - The ability to analyze messy and complex data, identify overarching patterns, and distill your findings into crisp annotation guidelines or model quality reports. - Proficiency with Python, LLMs, or other popular industry tools for automation. - Excellent communication and project management skills, and the ability to support several projects simultaneously. - A foundational understanding of and interest in LLMs/VLMs and multimodal AI. - Conviction that data is the key ingredient for the performance and assessment of AI models. YOU’LL STAND OUT IF YOU HAVE: - Experience in data collection and labeling for multimodal language models. - Experience in red teaming, localization testing, or other evaluation focused fields. - Experience working with research scientists and engineers. - Expertise or interest in video-centric domains, such as sports, advertising, and content creation. TECH STACK: - Development & Analysis: Python (primarily pandas, Jupyter, etc.) - Data Management & Visualization: Amazon S3, Various data visualization tools (framework-agnostic) - Project Management Tools: Linear, Notion Even if there are a few checkboxes that aren’t ticked through your prior experience, we still encourage you to apply! If you are a 0-1 achiever, a ferocious learner, and a kind and fun team player who motivates others, you will find a home at TwelveLabs. We are a global company that values the uniqueness of each person’s journey. It is the differences in our cultural, educational, and life experiences that allow us to constantly challenge the status quo. We are looking for individuals who are motivated by our mission and eager to make an impact as we push the bounds of technology to transform the world. Join us as we revolutionize video understanding and multimodal AI. BENEFITS AND PERKS: 🤝 An open and inclusive culture and work environment. 🧑‍💻 Work closely with a collaborative, mission-driven team on cutting-edge AI technology. 🦷 Full health, dental, and vision benefits. ✈️ Flexible PTO and parental leave policy. Office closed the week of Christmas and New Years.