Databricks Administrator

Onsite at client location in Mountain View, CA (4 days per week)Long term contract with possibility of conversion (~3 years)Average hourly rate of $80/hour depending on experience and interview performanceNext Ventures financial services consulting client is seeking a Databricks Administrator with subject matter expertise in AWS services and products for a long term contract position out of their end clients office in Mountain View, CA.The end client is just beginning their Databricks journey and is seeking a seasoned administrator to leadin setting up, managing, and securing the Databricks platform within an AWS environment, ensuring it runs smoothly for the entire team.Responsibilities:Key Responsibilities:Databricks & Cloud Platform Administration:Deploy, configure, and maintain Databricks workspaces on AWS, ensuring high availability and security.Manage cluster configurations, autoscaling, and compute resources for optimal performance.Administer Unity Catalog, enforcing governance policies for structured and unstructured data assets.Security, Compliance & Data Governance:Implement Unity Catalog for centralized access control, enforcing RBAC (Role-Based Access Control), attribute-based policies, and lineage tracking.Define fine-grained permissions for tables, views, and files stored in AWS S3.Ensure compliance with banking industry regulations (e.g., SOX, PCI-DSS, GDPR) through data masking, encryption, and audit logging.Manage IAM roles and authentication mechanisms between AWS and Databricks.Performance Optimization & Monitoring:Optimize Spark job performance and cluster tuning using best practices.Monitor Databricks workspace health with CloudWatch, Datadog, or Splunk.Automate cost control measures through idle cluster termination, autoscaling policies, and query optimizations.Integration & Automation:Integrate Databricks with AWS services (S3, Glue, Redshift, Kinesis, IAM) and external databases.Automate workspace provisioning and Unity Catalog policies using Terraform or Infrastructure-as-Code (IaC).Implement CI/CD pipelines for Databricks notebooks, jobs, and configurations.Collaboration & Support:Work closely with data engineers, cloud architects, security teams, and compliance officers to align Databricks with banking security policies.Troubleshoot platform issues, user access problems, and data pipeline failures.Develop best practices and documentation for Databricks administration and Unity Catalog implementation.Qualifications & Experience:5+ years of experience administering Databricks on AWS in an enterprise environment.Hands-on experience with Unity Catalog, including fine-grained permissions, lineage tracking, and governance policies.Strong knowledge of AWS services (S3, Glue, Redshift, IAM, Lambda, Kinesis).Expertise in Spark cluster management, Databricks Jobs, and SQL analytics.Experience in cloud security frameworks and data compliance for regulated industries.Familiarity with Terraform, CloudFormation, or Infrastructure-as-Code for automation.Understanding of networking & VPC configurations for Databricks on AWS.Preferred Skills:Experience with Databricks Delta Lake, MLflow, and AI/ML workloads.Proficiency in Python, Bash, or PowerShell for scripting and automation.Knowledge of Airflow, Kafka, or other data orchestration tools.Databricks Certified Associate/Professional Administrator certification is a plus.

Created: 2025-02-20