AI Engineer & Researcher - CUDA/GPU Kernel

Job Description

xAI is seeking an AI Engineer & Researcher - CUDA/GPU Kernel to join their team. The ideal candidate will focus on developing and improving low-level CUDA kernel optimizations for state-of-the-art inference and training software stack. This person will be responsible for profiling, debugging, and optimizing single and multi-GPU operations, understanding GPU memory hierarchy, implementing deep learning methods in CUDA kernels, and innovating new ideas. The role is based in the Bay Area [San Francisco and Palo Alto]. Candidates are expected to be located near the Bay Area or open to relocation.

Responsibilities Include:

• Developing and improving low-level CUDA kernel optimizations
• Profiling, debugging, and optimizing single and multi-GPU operations
• Understanding GPU memory hierarchy and computation capabilities
• Implementing deep learning methods in low-level CUDA kernels
• Innovating new ideas to optimize GPU performance

Requirements:

• Experience building high-performance GeMM CUDA kernels
• Experience implementing features for attention kernels
• Comfortable writing both forward and backward kernels
• Optimizing for memory-bound and compute-bound operations
• Familiarity with optimizing inference and training workloads
• Experience integrating custom-written kernels into JAX/XLA using pybind

• Opportunity to work on challenging engineering problems, CUDA kernel optimizations