Search - Workchat - Principal Data Scientist

Job Description

Elastic is seeking a Principal Data Scientist to join its Search Data Science team. This role focuses on developing and integrating statistical tools and machine learning models within the Search domain, supporting semantic search, RAG, agentic search, and chat applications. The ideal candidate will collaborate closely with Product teams to drive innovation and prototype AI/ML-driven Search experiences, with an emphasis on bringing new ideas to production quickly.

The Principal Data Scientist will be responsible for research and development to improve semantic search using proprietary and customized open-source models. They will develop techniques and models for query and document understanding, implement RAG and LLM-driven search experiences, and create tooling to help customers design successful end-to-end RAG systems. The role also involves investigating modern agentic search aspects, including reasoning engines and prompt engineering techniques.

What This Role Involves:

• Exploring, selecting, and benchmarking open source and Elastic proprietary models.
• Implementing RAG and other LLM-based search experiences.
• Designing evaluation protocols for semantic search, tool selection, and generation in LLM-based search experiences.
• Keeping up-to-date with the most significant recent developments in the field of NLP and information retrieval.
• Engaging with the NLP and information retrieval communities.
• Collaborating with cross-functional teams of data scientists, engineers, and product managers.
• Promoting knowledge sharing and collaboration in a distributed team.

Requirements:

• 8+ years of confirmed experience building and applying NLP to production use cases.
• 8+ years of professional software development experience in Python.
• Experience in Generative AI, Retrieval Augmented Generation, and information retrieval.
• Experience with libraries and frameworks such as PyTorch, transformers, and Pandas.
• Experience using collaborative notebook-based workflows (e.g., Jupyter) for prototyping and knowledge sharing.
• Expertise in AI/ML quality evaluation and improvement.
• Self-motivated, collaborative style, open communicator, experience in a distributed team.
• Good attention to detail and highly organized.
• Real passion for data, analysis, and achieving excellence.
• Experience with Elasticsearch is useful.
• An academic background in the domain is also a plus.

What Elastic Offers:

• Competitive pay based on the work you do.
• Health coverage for you and your family in many locations.
• Ability to craft your calendar with flexible locations and schedules for many roles.
• Generous number of vacation days each year.
• Matching up to $2000 for financial donations and service.
• Up to 40 hours each year to use toward volunteer projects you love.
• Minimum of 16 weeks of parental leave.