Python wen Scrapping Engineer

Job description

1. **Roles and Responsibilities :** 2. Design and develop scalable web scraping workflows using Python and Scrapy. 3. Extract structured data from static and dynamic websites with high accuracy. 4. Implement robust solutions for handling pagination, sessions, retries, throttling, duplicate records, and missing data. 5. Utilize Selenium or Playwright for browser automation when necessary. 6. Integrate with enterprise systems using SOAP APIs. 7. Deploy automation workloads using Docker and operate/support scraping workloads in Zyte. 8. Implement logging, error handling, traceability, and root-cause analysis. 9. Maintain code using Git/GitHub best practices and support data validation, reconciliation, and production issue resolution. 10. **Job Requirements :** 11. Proficiency in Python programming language. 12. Experience with Jenkins, Docker, Kubernetes, web crawling, web scraping, and Beautiful Soup. 13. Minimum of 5 years of experience in software development. 14. Strong understanding of software development principles and methodologies. 15. Excellent problem-solving skills and attention to detail. 16. Ability to work collaboratively in a team environment. 18. Must-Have Technical Skills Python: Build scraping scripts, automation utilities, validation checks, logging, retries, and error-handling logic. Scrapy: Create scalable spiders, manage pagination, pipelines, middlewares, throttling, and structured outputs. BeautifulSoup: Parse HTML/XML pages and extract clean data from irregular or nested page structures. Selenium: Automate browser-based workflows where UI interaction or verification is required. Playwright: Handle modern JavaScript-heavy websites with reliable browser automation. SOAP API: Integrate with enterprise services to send, receive, and validate structured payloads. Docker: Package scraping and automation workloads for consistent execution across environments. Git/GitHub: Manage source code through branches, commits, pull requests, reviews, and version control. Logging and Error Handling: Track failures, retries, exceptions, and production issues clearly. Production Debugging: Investigate broken crawls, selector failures, blocked requests, data mismatches, and job failures. Good-to-Have Technical Skills SQL: Validate scraped or loaded data, reconcile records, and support troubleshooting through queries. Jenkins: Run scheduled jobs, CI/CD pipelines, automated tests, and deployment checks. Zyte: Deploy, monitor, and operate production-grade web scraping workloads. Microsoft Power Automate: Trigger workflows, notifications, approvals, and operational routing. Microsoft Power Apps: Build lightweight internal apps for tracking requests, runs, and exceptions. NLP: Support text extraction, classification, normalization, and content validation use cases. Machine Learning: Apply basic anomaly detection or pattern-based validation to improve data quality. Node.js: Build lightweight services, APIs, or integration utilities when needed. Payment Gateway Integrations: Work with payment APIs, webhooks, transaction payloads, and reconciliation flows.