Software Engineer, Ray Data

About this job

At Anyscale, the mission is to democratize distributed computing through Ray, an open-source project designed to create a scalable machine learning ecosystem utilized by major companies worldwide. The company builds a platform allowing developers and data scientists to scale machine learning applications easily from a laptop to a cluster. The Ray Data team focuses on developing and maintaining the Ray Datasets library, a Python-based tool built on Apache Arrow and Ray Core that handles critical production use cases like large-scale data compaction and end-to-end ML pipelines. In this Software Engineer role, a normal day involves building, optimizing, and scaling Ray's data processing capabilities. Team members work on improving dataset performance at large scale, integrating with ML training and data sources, building stability and stress-testing infrastructure, and developing open-source software to simplify distributed programming. Engineers also evaluate architectural improvements, enhance testing processes, and communicate their work through talks, tutorials, and blog posts.

Responsibilities

• Develop high quality open source software to simplify distributed programming using Ray
• Identify, implement, and evaluate architectural improvements to Ray core and Datasets
• Improve the testing process for Ray to ensure smooth software releases
• Communicate work to a broader audience through talks, tutorials, and blog posts
• Work on performance optimization of Ray Datasets at large scale
• Integrate Ray with ML training and various data sources
• Build stability and stress testing infrastructure

Requirements

• At least 5 years of relevant work experience
• Solid background in algorithms, data structures, and system design
• Experience in building scalable and fault-tolerant distributed systems
• Experience with data processing
• Experience with database internals including Spark or Dask

Good to have

• Experience with streaming workloads such as Beam on Ray

Who should apply

This role is well-suited for experienced software engineers with a strong foundation in distributed systems, algorithms, and data processing. Candidates who have a background working with database internals, Spark, or Dask, and who enjoy building open-source software will find this position a great fit. Because the posting requires at least 5 years of relevant work experience, freshers are not eligible to apply.

Frequently asked questions