Performance Engineer, Inference

About this job

As a Performance Engineer for Inference at Sarvam AI, you will join the Performance Engineering team based in Bengaluru to own the production serving path for large distributed models. Sarvam AI develops diverse model families, including LLMs, Mixture-of-Experts, ASR, TTS, and multimodal models, running on advanced hardware fleets. In this role, your normal day involves integrating model and kernel artifacts into a running multi-node serving stack, modifying serving frameworks at the source code level, building and training custom speculator models, and collaborating closely with architecture, model, and SRE teams. You will be responsible for driving and defending critical system metrics such as TTFT, TPOT, throughput, GPU utilization, and cost efficiency.

Responsibilities

• Own Sarvam's production serving path end-to-end for large distributed models.
• Read and modify source code in frameworks like SGLang, vLLM, NVIDIA Dynamo, or TensorRT-LLM.
• Operate and extend distributed-serving stacks covering disaggregated prefill-decode and KV cache transfer.
• Integrate artifacts from model and kernel teams into a multi-node, multi-tenant serving environment.
• Build, train, and tune custom speculators and draft models against live serving distributions.
• Generate and defend performance metrics including latency, throughput, and cost per million tokens.
• Participate in cross-team work spanning architecture co-design, kernels, modeling, and SRE on-call ownership.

Requirements

• 5+ years of experience in ML systems with 2+ years focused on production-scale inference serving.
• Experience serving 100B+ parameter models using tensor, pipeline, or expert parallelism across multiple nodes.
• Source-level fluency in at least one of SGLang, vLLM, Dynamo, or TensorRT-LLM, with reading familiarity for the rest.
• Practical experience running and extending distributed serving stacks and disaggregated prefill-decode architectures.
• Competency in building and training speculative decoding solutions, including draft models and acceptance-rate tuning.
• Solid understanding of KV cache internals, NCCL primitives, and multi-tenant serving isolation techniques.
• Profiling fluency using Nsight Systems, framework tracing, and perf tools along with C++ and CUDA reading skills.

Good to have

• Upstream code contributions to major LLM serving frameworks on non-trivial code paths.
• Direct production experience operating Dynamo or llm-d at scale, or maintaining a forked runtime.
• Published or shipped work involving trained draft models and speculative decoding techniques.
• Experience with Mixture-of-Experts serving at scale, long-context windows, or multilingual workloads.

Who should apply

This role suits experienced ML systems and performance engineers with a strong background in production-scale inference serving, distributed infrastructure, and framework-level code modifications. Candidates should have a proven track record of handling massive model deployments and optimizing key GPU metrics. Freshers cannot apply as the position specifically mandates 5+ years of relevant industry experience.

Frequently asked questions