Senior Platform Engineer

Job description

**Job Summary** We are seeking a highly skilled **Vector Developer** to design, implement, and optimize high-performance data pipelines using **Vector** . This role focuses on migrating and managing log and metric pipelines from platforms such as **Splunk** , enabling efficient data collection, transformation, and routing to downstream systems including **Datadog** and other analytics platforms. The ideal candidate will have strong expertise in **observability systems, log processing, and pipeline optimization** , with a focus on delivering reliable, scalable, and cost-efficient data pipelines. **Key Responsibilities** - Design, build, and maintain high-performance data pipelines using **Vector** (Sources → Transforms → Sinks). - Migrate existing log and metric pipelines from **Splunk** and similar platforms to **Vector** . - Develop data transformations using **Vector Remap Language (VRL)** . - Filter, enrich, and efficiently route logs and metrics to downstream platforms such as **Datadog** and analytics systems. - Handle complex ingestion scenarios, including oversized records, malformed data, and high-throughput workloads. - Optimize pipelines for low latency, high throughput, and reduced infrastructure and log ingestion costs. - Configure buffering, retries, and backpressure handling to ensure reliable and resilient data processing. - Monitor, troubleshoot, and resolve pipeline failures, data inconsistencies, and performance bottlenecks. - Collaborate with Data Engineering and Platform teams to establish and enforce data contracts and pipeline best practices. **Qualifications & Experience** - Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field (or equivalent practical experience). - 3+ years of experience in Data Engineering, Platform Engineering, DevOps, or Observability Engineering. - Hands-on experience building and managing log or telemetry data pipelines. - Strong experience with **Vector** , **Vector Remap Language (VRL)** , or similar log processing technologies. - Experience migrating log pipelines from **Splunk** or comparable observability platforms. - Strong understanding of observability, logging, monitoring, and telemetry architectures. - Experience optimizing data pipelines for scalability, reliability, and cost efficiency. - Excellent troubleshooting, debugging, and problem-solving skills. - Strong communication and collaboration skills with cross-functional teams. **Preferred Skills & Domain Expertise** - Experience with **Datadog** , **Open Telemetry** , **Fluent Bit** , **Fluentd** , **Logstash** , or similar observability tools. - Knowledge of **Rust** programming language. - Experience working in **Cybersecurity** , SIEM, or Security Analytics environments. - Familiarity with cloud platforms (AWS, Azure, or GCP). - Experience with containerized environments using **Docker** and **Kubernetes** . - Understanding of high-throughput distributed systems and streaming data architectures. Why Join Us? - Opportunity to build and modernize enterprise-scale observability and logging platforms. - Work on cutting-edge technologies focused on high-performance data ingestion and pipeline optimization. - Collaborative environment with Data Engineering, Platform, and Infrastructure teams. - A culture that values innovation, automation, reliability, and continuous learning. We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, sex, gender, gender expression, sexual orientation, age, marital status, veteran status, or disability status.