Data Engineer - Data Bricks / AWS

SOFT's client located in New York, NY ( Hybrid ) is looking for a Data Engineer - Data Bricks AWS for a long term contract assignment.WE ARE NOT ACCEPTING 3RD PARTY SOLICITATIONS FOR THIS OR ANY OF OUR JOB POSTINGS OR REQUISITIONS. ANY SUCH INQUIRIES WILL NOT BE CONSIDERED OR RESPONDED TO. WE CAN ONLY WORK WITH DIRECT APPLICANTS WHO ARE AUTHORIZED TO WORK IN THE US WITHOUT SPONSORSHIP THIS IS A HYBRID ROLE, REQUIRING 2 DAYS A WEEK AT THE CLIENT SITE IN LOWER MANHATTAN. CANDIDATES MUST BE IN THE NYNJCT TRI-STATE AREA AT THE TIME OF APPLICATION. CANDIDATES OUTSIDE OF THIS RADIUS WILL NOT BE CONSIDERED Qualifications: • Minimum of 5 years of experience in data engineering roles, with a focus on AWS and Databricks. • Highly proficient with Databricks, Spark, StarburstTrino, Python, PySpark and SQL • Hands-on experience in Gitlab with CICD. • Hands-on experience in AWS Services like S3, RDS, Lambda, SQS, SNS, MSK is required. • Strong SQL skills to perform data analysis and understanding of source data. • Experience with data pipeline orchestration tools Responsibilities:Design, develop, monitor, and maintain data pipelines in an AWS ecosystem with Databricks, Delta Lake, Python, SQL and Starburst as the technology stack. Collaborate with cross-functional teams to understand data needs and translate them into effective data pipeline solutions. • Establish data quality checks and ensure data integrity and accuracy throughout the data lifecycle. • Automate testing of the data pipelines and configure as part of CICD • Optimize data processing and query performance for large-scale datasets within AWS and Databricks environments. • Document data engineering processes, architecture, and configurations. • Troubleshooting and debugging data-related issues on the AWS Databricks platform. • Integrating Databricks with other AWS products such as SNS, SQS, and MSK. CommentsSpecial Instructions

Created: 2024-11-06