Senior Data Engineer

Senior Data EngineerTechnologyMiami, FL (Hybrid)$150,000 - $170,000 + equityThe CompanyOur client is revolutionizing operational safety in critical industries. They specialize in leveraging computer vision to provide real-time insights for the Construction industry. Their mission is to eliminate failures due to inadequate visual inspection, ensuring safety, reliability, and compliance.Role DescriptionAs a Senior Data Engineer, you will play a pivotal role in designing and optimizing scalable data pipelines that power machine learning and analytics applications. You will be responsible for architecting a robust data infrastructure, collaborating with ML engineers, and ensuring efficient storage and processing of large-scale industrial and vision data. This is an opportunity to make a significant impact on a rapidly growing team that values technical excellence and innovation.Key ResponsibilitiesArchitect and build scalable data pipelines using Dagster to transform and move data for analytics and ML.Design and optimize storage solutions for large-scale industrial and vision data, ensuring efficient retrieval and processing.Develop data ingestion frameworks for real-time and batch processing of images, video, and metadata.Work with ML engineers to structure data effectively for model training and experimentation.Deploy and manage data pipelines in Kubernetes-based environments.Automate CI/CD workflows for data infrastructure using GitLab CI/CD.Manage AWS-based data infrastructure, leveraging Terraform for Infrastructure as Code.Implement data governance best practices, ensuring data quality, lineage tracking, and metadata management.Optimize batch and real-time processing frameworks for scalability and performance.Provide technical leadership, establishing best practices and guiding future data scaling strategies.Requirements5+ years of experience in data engineering with a focus on scalable, production-grade data infrastructure.Strong expertise in Python (Pandas, PyArrow, Dask, etc.).Hands-on experience with data orchestration tools (Dagster, Prefect, or Airflow).Proficiency in Kubernetes for data pipeline orchestration.Experience deploying infrastructure via Terraform (or similar IaC tools).Cloud expertise, preferably AWS, including S3, EKS, Lambda, Glue, and RDS.Experience with streaming and event-driven architectures such as Apache Ray Core, Kafka, Kinesis, Pulsar, or Storm.Strong database skills, including SQL, NoSQL, and columnar storage (Postgres, BigQuery, ClickHouse).CI/CD expertise with GitLab for managing automated data pipeline deployments.The BenefitsAs a Senior Data Engineer, you will receive a base salary of $150,000 to $170,000 plus benefits like equity, health, dental and vision.How to ApplyTo apply, please submit your resume via the link on this page.Key WordsPython, Dagster, Kubernetes, Terraform, AWS, Data Pipelines, CI/CD, Streaming Data, Apache Ray, Kafka, Computer Vision, Machine Learning, SQL, Big Data, Data Engineering

Created: 2025-03-02