Senior Software Engineer(Python, Spark, Iceberg, Hive)

Job description

**Required Qualifications:** - **4+ years of Soft**ware Engineering experience, or equivalent demonstrated through one or a combination of the following: work experience, training, military experience, education **Desired Qualifications:** - **Technical Experience (4+ years)** - ***Hands-on engineering experience using***: - **Python, Spark, Iceberg, Hive** - **Dremio** or virtualization/federation/semantic layer tools - **Any major cloud** – Azure or GCP - **ETL tools**: Ab Initio (preferred), Informatica, DataStage - **Databases**: Oracle, MS SQL, Teradata - **Orchestration**: Autosys, Airflow (preferred) - Strong SQL development and tuning skills - Proven experience supporting and optimizing **large enterprise-scale data environments**. - Working with **modern Data Warehousing, Data Lakes, and Lakehouse architectures**. - Monitor, tune, and troubleshoot large-scale distributed systems. - Diagnose memory, performance, partitioning, and query-related inefficiencies across compute engines. - Ensure system reliability, scalability, and high throughput in enterprise data environments. - Expertise in **designing / implementing modern data platforms** with: - Standardization of patterns, frameworks, and reusable components - Automated code-generation, CI/CD and DevOps best practices - Cloud-native and open-table formats (e.g., Iceberg) - Hands-on experience in **GenAI, Agentic AI, and LLM adoption**, such as: - Building RAG architectures, vector stores, embedding pipelines - Integrating data systems with enterprise AI platforms - Using LLMs to improve data quality, metadata extraction, or automation of data engineering tasks - Familiarity with data governance and security frameworks, including privacy, compliance, lineage, and access control. - Strong communication skills and collaborate with cross-technical / functional teams. - Ability to work in an agile, fast-paced environment while delivering high-quality results. - Passion for innovation, continuous learning, and adopting modern practices including AI-driven development.