Software Engineer - ETL Developer with Python

Job description

We are seeking a highly skilled and motivated Data Engineer to join our dynamic team. The ideal candidate will possess advanced technical expertise in Python programming, relational databases, and ETL processes, with a strong foundation in data modeling, transformation, and processing of both structured and unstructured data. This role will also play a pivotal part in expanding our data engineering capabilities to incorporate agentic AI solutions, driving automation and intelligence across our data workflows. **Key Responsibilities:** - Design, develop, and maintain scalable and robust data pipelines and ETL systems using Python and SQLAlchemy. - Transform and process large datasets from diverse sources, ensuring data integrity, quality, and consistency. - Optimize and build efficient SQL queries for data retrieval and manipulation; manage and interact with relational databases (MSSQL, PostgreSQL, Oracle, etc.). - Collaborate with cross-functional teams to understand data requirements and deliver innovative solutions. - Implement data models and structures to support business intelligence, analytics, and machine learning initiatives. - Leverage advanced Python programming techniques and libraries (e.g., Pandas, Numpy) to solve complex data challenges. - Architect and implement agentic AI solutions to automate data processes, enhance pipeline intelligence, and support next-generation analytics. - Work with AWS cloud services related to data engineering ( AWS Batch, S3, Lambda, etc.). - Stay abreast of emerging trends in data engineering and AI and proactively recommend improvements. - Integrate real-time data streaming solutions (e.g., Kafka, Kinesis) into data pipelines. **Technical Skills:** **Python Programming Language:** - Level: Advanced - Key Concepts: Multi-threading, Multi-processing, Regular Expressions, Exception Handling, Generators, Decorators, Context Managers, Asyncio, Type Hinting, and best practices for scalable code. - Libraries: Pandas, Numpy, SQLAlchemy, etc. **Data Modelling and Data Transformation:** - Level: Advanced - Key Areas: ETL, processing structured and unstructured data, data quality, and integrity. **Relational Databases:** - Level: Advanced - Key Areas: Query Optimization, Query Building, Experience with ORMs like SQLAlchemy, Exposure to databases such as MSSQL, PostgreSQL, Oracle, etc. **Agentic AI / Automation:** - Exposure to or interest in agentic AI frameworks and tools (e.g., LangChain, OpenAI APIs, or similar). - Experience integrating AI/ML models into data pipelines is a plus. **Cloud Technologies:** - Good experience with AWS data engineering services (Athena, AWS Batch, S3, Lambda, etc.). **Real-Time Streaming:** - Experience with real-time data streaming platforms (Kafka, Kinesis, etc.) is a plus. **Programming Paradigms:** - Functional and Object-Oriented Programming (OOPS): Intermediate **Problem Solving:** - Strong analytical and problem-solving skills for feature development and automation. **Qualifications:** - Bachelor’s or master's degree in computer science, Engineering, or a related field. - Proven experience as a Data Engineer or in a similar role. - Experience with agentic AI or automation frameworks is a plus. - Excellent communication and teamwork abilities. - Ability to work in a fast-paced, agile environment. **What’s in it for you:** - Opportunities for professional growth and development. - Exposure to a talented technical team, providing an environment that encourages learning and skill enhancement. - Collaborative and inclusive work environment. **Who are we?** At Gartner, Inc. (NYSE:IT), we guide the leaders who shape the world. Our mission relies on expert analysis and bold ideas to deliver actionable, objective business and technology insights, helping enterprise leaders and their teams succeed with their mission-critical priorities. Since our founding in 1979, we’ve grown to 20,000 associates globally who support over 13,000 client enterprises in ~90 countries and territories. We do important, interesting and substantive work that matters. That’s why we hire associates with the intellectual curiosity, energy and drive to want to make a difference. The bar is unapologetically high. So is the impact you can have here. **What makes Gartner a great place to work?** Our vast, virtually untapped market potential offers limitless opportunities – opportunities that may not even exist right now – for you to grow professionally and flourish personally. How far you go is driven by your passion and performance. We hire remarkable people who collaborate and win as a team. Together, our singular, unifying goal is to deliver results for our clients. Our teams are inclusive and composed of individuals from different geographies, cultures, religions, ethnicities, races, genders, sexual orientations, abilities and generations. We invest in great leaders who bring out the best in you and the company, enabling us to multiply our impact and results. This is why, year after year, we are recognized worldwide as a great place to work. **Gartner is the world authority on AI** At Gartner, you’ll join a company at the very center of the AI revolution. Gartner has proactive, objective guidance throughout clients’ AI journeys. We set the standard for how organizations leverage artificial intelligence to drive meaningful impact. You’ll have access to unmatched resources, expertise, and technology, and play a key role in helping Gartner and our clients innovate and grow as we leverage AI to transform business and technology landscapes. It’s an exciting time to be at Gartner, with limitless opportunities to make a real impact, grow your skills, and build a lasting, meaningful career in a field that’s reshaping the way we operate. If you’re passionate about AI and want to be part of a team that’s guiding the leaders who shape the world, Gartner is the place for you. **What