SEAL Research Scientist/ Research Engineer

Job Description

Scale is seeking a talented Research Scientist/ Research Engineer to join their Safety, Evaluations and Alignment Lab (SEAL) in San Francisco. The role involves shaping the landscape for safety and transparency in the AI industry by building robust evaluation products and tackling research problems in evaluation and red teaming.

The role involves:

• Designing and implementing robust evaluation benchmarks.
• Measuring and improving eval reproducibility and reliability.
• Researching and implementing cutting-edge model-assisted red teaming methods.
• Developing state-of-the-art LLM-based automated rating systems using the Scale ecosystem for customized training data.
• Developing rater assist techniques such as critique modeling for improving rating quality.

Requirements:

• Commitment to promoting safe, secure, and trustworthy AI deployments.
• Practical experience conducting technical research collaboratively, with proficiency in frameworks like Pytorch, Jax, or Tensorflow.
• Adept at interpreting research literature and turning new ideas into prototypes.
• A track record of published research in machine learning, particularly in generative AI.
• At least three years of experience addressing sophisticated ML problems, whether in research or product development.
• Strong written and verbal communication skills.
• Experience with open-source LLM fine-tuning or bespoke LLM fine-tuning projects using Pytorch/Jax (Nice to have).
• Experience in crafting evaluations or a background in data science roles related to LLM technologies (Nice to have).
• Experience working with a cloud technology stack (e.g., AWS or GCP) and developing machine learning models in a cloud environment (Nice to have).

What Scale offers:

• Comprehensive health, dental and vision coverage.
• Retirement benefits.
• A learning and development stipend.
• Generous PTO.
• Commuter stipend (potentially).