VP - Senior Data Engineer (Python/Snowflake/Apache Airflow)

Job description

Locations: **Mumbai, India** ## **Job description** **About this role** **Responsibilities** - Actively participate in Agile ceremonies, including requirements refinement, sprint planning, effort estimation, sprint reviews, and retrospectives, to support successful delivery of platform capabilities. - Participate in technical design discussions, analyze requirements, and translate complex business needs into scalable data platform solutions. - Design, develop, and optimize core enterprise data platform capabilities, including data acquisition, ingestion, transformation, workflow orchestration, and reusable processing frameworks, leveraging Python, SQL, cloud-native technologies, and cloud data warehouse platforms to support large-scale, high-performance data processing. - Build and optimize large-scale data pipelines supporting data ingestion, transformation, validation, and distribution across enterprise cloud environments. - Implement metadata-driven processing frameworks and reusable engineering patterns that improve automation, standardization, scalability, and reuse. - Build and enhance platform capabilities related to metadata management, lineage, dependency tracking, data quality, governance, and workflow control. - Incorporate Agentic AI capabilities into platform services to enhance automation, intelligent decision-making, developer productivity, and operational efficiency. - Partner with cross-functional teams to support collaborative development, system integration, and end-to-end testing across platform components and dependent systems. - Develop and maintain system, integration, regression, and test automation frameworks to ensure platform quality, reliability, and production readiness. - Support deployment and release activities, ensuring successful rollout of platform enhancements and seamless integration with dependent systems and downstream consumers. - Contribute to engineering standards, architectural patterns, operational best practices, and framework conventions that improve platform maintainability, consistency, scalability, and supportability. - Provide L2/L3 production support for data platforms and products, perform root cause analysis, and implement corrective actions to ensure platform stability and operational excellence. - Drive continuous improvements in platform performance, scalability, resilience, observability, and operational efficiency across data workflows and framework components. - Lead Proof of Concepts (POCs) and technology evaluations to assess emerging technologies, validate architectural approaches, and identify opportunities for platform modernization and innovation. - Support product documentation by providing technical expertise, implementation details, and content validation to ensure accuracy, completeness, and alignment with platform capabilities. - Conduct live product demonstrations and participate in user workshops to showcase platform capabilities, gather feedback, validate solutions, and support platform adoption **Required Qualifications** - Bachelor’s degree in Computer Science, Information Systems, or a related technical field. - 10–14 years of experience in Data Engineering, Data Platform Engineering, or a related software engineering role. - Hands-on experience with Apache Airflow for orchestrating complex, dependency-driven data pipelines and workflows. - Strong proficiency in Python, with hands-on experience developing scalable, production-grade data pipelines and frameworks. - Hands-on experience designing and building ETL/ELT frameworks, reusable data pipeline components, and workflow-driven data systems. - Hands-on experience with Snowflake or equivalent cloud-native analytical data platforms. - Strong hands-on experience writing and optimizing complex SQL, including stored procedures, UDFs, and performance-critical queries. - Proven experience in query optimization, performance tuning, and workload management for large-scale data environments. - Extensive hands-on experience with dbt, including data modeling, transformation, testing, documentation, and deployment within modern data platforms. - Proven track record of improving the performance, reliability, scalability, and maintainability of enterprise data platforms and pipelines. - Hands-on experience with streaming technologies (e.g., Snowpipe, Kafka) and messaging platforms for building scalable, real-time data processing solutions. - Hands-on experience building solutions in cloud-native and distributed environments, leveraging object storage, cloud data services, and major cloud platforms such as Azure, AWS, or GCP. - Exposure to Apache Iceberg, leveraging schema evolution, partition evolution, and time-travel capabilities to build scalable and reliable data lake solutions. - Experience with SDK and API-based development, enabling seamless integration of data platforms, cloud services, and enterprise applications through secure and scalable interfaces. - Hands-on experience designing and developing cloud-native data platforms, including Data Lake and Data Warehouse architectures, using modern data engineering best practices. - Exposure to AI-native development tools such as Windsurf, Cursor, and Antigravity, leveraging AI-assisted coding, testing, and automation to improve engineering productivity and software delivery. - Understanding of Agentic AI architectures, including Skills, Model Context Protocol (MCP), Vector Databases, Retrieval-Augmented Generation (RAG), and AI Agent orchestration concepts. - Experience working with Docker, Kubernetes, and containerized deployment architectures in modern engineering environments. - Strong understanding of software engineering principles, scalable system design, and modern development practices, including Agile methodologies, Git-based source control, CI/CD pipelines, code reviews, and production-grade development standards. - Experience implementing and managing metadata, lineag