Azure & AWS - Architect

Job description

**Job Summary:** We are seeking a visionary and hands-on **Data Engineering Lead** with **8+ years of experience** to spearhead the design and execution of our global data strategy, modernization initiatives and newer projects (E2E set ups). In this leadership role, you will not only build and manage scalable data pipelines but also provide technical mentorship to a cross-functional team of Data Scientists, MLOps Engineers, and Application Engineers. You will act as the primary architect for our hybrid cloud environment, ensuring robust, high-quality data solutions across Microsoft **Azure and AWS**. **Key Responsibilities:** - **Strategic Architecture**: Lead the end-to-end design and implementation of scalable data pipelines across Azure and AWS platforms. - **Technical Oversight**: Supervise the development of batch and near real-time streaming solutions, ensuring they meet rigorous business demands. - **Engine Innovation**: Drive the optimization of rule-based Spark processing engines, leveraging SQL DB for dynamic, high-performance execution. - **Advanced Analytics**: Oversee the creation of complex KQL queries in Azure Data Explorer (ADX) to facilitate advanced anomaly detection and telemetry monitoring. - **Cross-Functional Collaboration**: Serve as the technical bridge between stakeholders and engineering teams to translate complex business needs into reliable data sets. - **Operational Excellence**: Accountable for pipeline performance, cost-efficiency, data integrity, and the resolution of high-level technical blockers. **Required Skills & Qualifications:** The Lead must possess deep expertise in the following areas: **1. Data Processing & Languages** - **Languages:** Mastery of **Python** for workflow logic, **SQL** for complex data management (Synapse, Redshift, SQL DB), and **KQL** for deep log analytics. - **Big Data:** Advanced proficiency in **Spark/Databricks** and **Structured Streaming** for large-scale, fault-tolerant data ingestion. **2. Multi-Cloud Ecosystems** - **Azure Services:** Expert-level knowledge of **ADLS**, **Azure Data Factory (ADF)**, **Synapse Analytics** (Pipelines, Spark Pools, Serverless SQL), **Azure Functions**, **Event Hubs**, and **Cosmos DB**. - **AWS Services:** Extensive experience with **S3**, **Redshift**, **Glue**, **EMR**, **Athena**, and **Kinesis**. - **Orchestration & Messaging:** Ability to architect event-driven triggers using **SQS**, **SNS**, and **Logic Apps**. **3. Specialized Databases** - Deep understanding of **Timestream** for sensor/log data and **DynamoDB** for low-latency metadata management. **Soft Skills & Leadership Qualities** - **Visionary Design:** Ability to architect solutions that balance immediate scalability with long-term cost-performance ratios. - **Technology Evangelism:** Proactively evaluate and recommend emerging tools to maintain a competitive technical edge. - **Data Modeling Mastery:** Expertise in designing logical and physical models for both analytical and transactional workloads. - **Communication:** Exceptional ability to communicate technical roadmaps to non-technical executive stakeholders. - **Ownership:** A "lead-by-example" mindset with the ability to work independently while fostering a culture of accountability. **Education & Experience:** - Bachelors or Master’s degree in Computer Science, Engineering, or a related field. - **8+ years** in Data Engineering with at least **2+ years** in a lead or supervisory capacity.