AWS Data Engineer with API & Integration expertise

Job description

Tredence is a global analytics services and solutions company our clients use to bridge the gap between insight delivery and value realization. We provide actionable solutions by combining deep data science expertise with business context to drive meaningful digital transformation. **Role - AWS Data Engineer with API & Integration expertise** **Experience: 6~9 years** **Preferred - Data Engineering Background** **Location - Bangalore, Chennai, Pune, Gurgaon, Kolkata** **Required Skills - AWS Glue, Lambda, Step Function, Realtime batch data, API's paginations and tokes** Job Overview We are seeking a skilled **AWS Data Engineer** with deep expertise in API integrations to join our team at Tredence Analytics. You will design, build, and optimize scalable data pipelines handling both real-time and batch data workloads for enterprise clients. Key Responsibilities - **Pipeline Development**: Build robust batch and real-time data pipelines using AWS native services. - **API Integration**: Design data ingestion patterns from third-party APIs, handling complex authentication and pagination. - **Performance Tuning**: Optimize complex SQL queries and PySpark jobs for massive datasets. - **Storage Architecture**: Manage and optimize data storage layers using cloud warehouses and modern open table formats. - **Orchestration**: Direct end-to-end data workflows and microservices execution. Mandatory Technical Skills 1. Core AWS Data Services - **AWS Glue**: Hands-on experience building, managing, and monitoring enterprise ETL/ELT pipelines. - **AWS Lambda**: Expertise in writing efficient, serverless code for data transformation and event-driven tasks. - **AWS Step Functions**: Experience orchestrating complex, multi-step data workflows and error handling. - **Amazon Kinesis**: Proven track record of capturing, processing, and storing real-time streaming data. 2. Data Processing & Processing Paradigms - **Batch & Real-time**: Practical experience balancing and executing both real-time streaming and high-volume batch architectures. - **PySpark**: Advanced development skills for distributed data processing. - **Query Optimization**: Deep understanding of tuning performance in both relational SQL and PySpark environments. 3. Advanced Storage & Analytics - **Amazon Redshift**: In-depth knowledge of warehousing, distribution keys, sort keys, and performance tuning. - **Amazon Athena**: Profiling and querying ad-hoc data directly in data lakes. - **Apache Iceberg**: Hands-on experience with open table formats, time travel, schema evolution, and hidden partitioning. 4. API Expertise - **API Mechanics**: Comprehensive understanding of RESTful/GraphQL API structures, request/response cycles, and HTTP status codes. - **Pagination**: Proven ability to handle varied API pagination strategies (e.g., cursor, offset, page-based). - **Security & Tokens**: Practical experience managing diverse token mechanisms (e.g., OAuth 2.0, Bearer tokens, JWT, API keys, refreshing expired tokens). Required Qualifications & Experience - **Experience**: **6 to 9 years** of dedicated experience in data engineering and cloud architectures. - **Education**: **BE / BTech** degree in Computer Science, Information Technology, or a related engineering field. - Strong problem-solving skills. - Clear technical communication. **Required Skills** Have good understanding and hands-on AWS Glue, Step function, Lambda, Kinesis. Worked on Realtime as well as batch data. Experience in handling query optimizations in sql and pyspark. Understanding APIs how it works paginations, types of tokens. depth knowledge of Redshift as well as open table formats like iceberg and athena.