Senior Data Engineer - Data and Analytics

About this job

Pattern is hiring a Senior Data Engineer for its Data and Analytics team in Pune, India. In this full-time role, you will work on Destiny, Pattern's automated ad-tech optimization system that influences live client advertising budgets. Unlike traditional data engineering that ends at a reporting dashboard, your data pipelines directly drive outward-facing automated ad spend, making data correctness, idempotency, and strict data-quality gating essential daily priorities. A normal day involves owning and extending Airflow orchestration for complex multi-DAG pipelines, writing and tuning large-scale analytical SQL queries, supporting SageMaker-based machine learning model training and batch inference, and implementing blocking data-quality checks. You will collaborate closely with data scientists and advertising strategists, triage pipeline failures, mentor other data engineers, and ensure that every action pushed to the marketplace Ads API meets rigorous safety and accuracy standards.

Responsibilities

• Develop, deploy, and support automated, scalable batch data pipelines from various sources into the lakehouse.
• Own and extend Airflow orchestration for multi-DAG daily pipelines and 15-minute action pipelines.
• Write and tune large analytical SQL queries, multi-hundred-column joins, window functions, and incremental merges.
• Extend the feature store by adding new features and labels while preserving leakage and data-completeness conventions.
• Orchestrate model training and batch inference on SageMaker from Airflow, managing S3 and Parquet round-trips.
• Develop data auditing strategies and blocking data-quality checks in front of outward-writing systems.
• Troubleshoot pipeline failures, conduct on-call triage, and guard safety properties like idempotency and audit trails.
• Collaborate with data scientists and advertising strategists to translate business requirements into data models and pipelines, and mentor junior engineers.

Requirements

• 4+ years of overall professional experience.
• 4+ years of hands-on experience with advanced SQL and Python.
• 3+ years building production data pipelines on modern data architectures with real ownership of scheduling and backfills.
• 2+ years working with cloud data warehouses such as Snowflake, Redshift, or BigQuery.
• Production experience with a workflow orchestrator, with Airflow strongly preferred.
• Experience orchestrating ML training and batch inference from a scheduler on SageMaker or an equivalent platform.
• Working knowledge of applied machine learning fundamentals including dataset construction, leakage, evaluation metrics, and model lifecycle operations.
• Comfort with AWS including S3 and IAM, columnar file formats, and demonstrated ownership of data quality and software engineering best practices.

Good to have

• Experience with digital advertising, bidding, or auction systems like Amazon Ads or Google Ads.
• Advanced Snowflake skills including streams, tasks, stored procedures, UDFs, clustering, and performance tuning.
• Experience with time-series data, forecasting, and data-quality frameworks such as Soda, Great Expectations, or dbt tests.
• Experience with open-source and distributed data platforms like Spark, Hive, Trino/Presto, or Elasticsearch.
• Broader AWS cloud experience including SNS, SQS, Lambda, Glue, and ECR.

Who should apply

This role suits experienced data engineers with a strong background in building production-grade data pipelines, advanced SQL, and workflow orchestrators like Airflow. Candidates should have a working grasp of applied machine learning operations, cloud data warehouses, and AWS environments. Because this position involves mentoring team members and guarding critical outward-writing ad systems, strong communication and problem-solving skills are essential. Please note that this is a senior position requiring professional experience; freshers are not eligible to apply.

Frequently asked questions