Azure Databricks DevOps Administrator

At USEReady,We empower people to succeed with data. USEReady is a data and analytics firm that provides the strategies, tools, capability and capacity that businesses need to turn their data into a competitive advantage. USEReady partners with cloud and data ecosystem leaders like Tableau, Salesforce, Snowflake and Amazon Web Services, and has been named Tableau partner of the year multiple times.Headquartered in NYC, the company has 600+ employees across offices in the U.S., Canada, Singapore and India and specializes in financial services. USEReady's deep analytics expertise, unique player/coach approach and focus on fast results makes the company a perfect partner for a cloud-first, digital world.Job Title: Azure Databricks DevOps AdministratorLocation: New York CityJob Type: Long term Contract/FTEExperienceAs an Azure Databricks DevOps Administrator, you will be responsible for managing and maintaining the Databricks platform, ensuring optimal performance and security across workspaces, clusters, and jobs. This role is crucial in overseeing user administration, managing Azure storage solutions, and implementing CI/CD pipelines using Azure DevOps. By leveraging a deep understanding of Databricks architecture and proficiency in scripting languages, the Azure DevOps administrator will automate tasks and enhance the efficiency of data operations. Strong communication skills and a commitment to high-quality support will enable effective collaboration with cross-functional teams, directly contributing to the departments mission of delivering robust data solutions and insights.Job ResponsibilitiesHands-on experience in Azure Cloud services, Networking Concepts, Security - Cloud and on-premises system, Deployment using Azure DevOps, Azure Cloud Monitoring and Cost controls and Terraform - hands-on experienceCI/CD Pipeline Management: Design, implement, and manage Continuous Integration/Continuous Deployment (CI/CD) pipelines using Azure DevOps and GitHub. Ensure the pipelines are efficient, reliable, and scalable.Infrastructure as Code (IaC): Automate the provisioning and management of infrastructure, with a focus on Azure Databricks and Azure Data Factory in a private network environment, using tools like Terraform and ARM templates.Environment Management: Create and manage development, testing, and production environments, ensuring consistency, security, and alignment with organizational requirements.Security: Implement security best practices throughout the CI/CD pipeline, including secrets management, secure code scanning, and compliance with security standards.Monitoring & Logging: Set up and maintain monitoring and logging for applications and infrastructure using Azure Monitor, Log Analytics, and related tools to ensure system reliability and performance.Automation: Identify opportunities for automation to streamline processes, reduce manual errors, and improve operational efficiency.Policy Enforcement: Establish and enforce policies such as branch policies, pull request reviews, and pipeline approvals to maintain code quality and compliance with organizational standards.Manage and maintain Azure Databricks Platform, workspaces, clusters, and jobsOversee user administration including access controls and permissionsHandle library installations, runtime management, and policy enforcementImplement/analyze cost control measures Administer Unity Catalog for data governance and securityCollaborate with data engineers, data scientists, and analysts to optimize and streamline data workflows, and analytical pipelines on the Databricks platformManage Azure storage solutions, including Blob Storage and Data Lake StorageAdminister Azure Key Vault for secure storage of secrets and keysConfigure alerts for proactive issue detection and resolutionDatabricks Lakehouse MonitoringMinimum QualificationsMinimum of Bachelors degree in Computer science, Information systems, or EngineeringExperience: 7+ years of professional experience in DevOps, with a strong focus on Azure. Knowledge of Data Platforms like Databricks, Data Factory (preferred)Proficiency in scripting languages such as Python/PySpark, PowerShell, or BashExperience in automating administrative tasks and workflowsKnowledge of security best practices and compliance requirementsExperience with ETL processes, data pipelines, and big data technologiesExperience with backup and restore procedures for Databricks and Azure servicesAbility to troubleshoot and resolve issues in a timely and efficient mannerStrong verbal and written communication skillsAbility to document processes, procedures, and configurations clearlyCommitment to providing high-quality support to internal/external users and stakeholders

Created: 2025-02-24