Hadoop Data Lake Automation Engineer Job at Cloudious LLC, Charlotte, NC

cFNaU0RTK3JYaFJsdG9IOXM0SlJsRXdB
  • Cloudious LLC
  • Charlotte, NC

Job Description

Job Summary:

We are looking for a skilled and motivated Hadoop Data Lake Automation Engineer with 4 5 years of experience in automating data workflows and processes within Hadoop-based data lake environments. The ideal candidate will be responsible for building scalable automation solutions, optimizing data pipelines, and ensuring efficient data movement and transformation across platforms.

Key Responsibilities:

  • Design and implement automation solutions for data ingestion, transformation, and processing in Hadoop data lake environments.
  • Develop and maintain scalable data pipelines using tools such as Apache NiFi, Spark, Hive, and Sqoop.
  • Collaborate with data engineers, analysts, and business stakeholders to understand data requirements and deliver automation solutions.
  • Monitor and troubleshoot data workflows, ensuring reliability and performance.
  • Implement best practices for data governance, security, and metadata management.
  • Maintain documentation for data flows, automation scripts, and operational procedures.
  • Support production environments and participate in on-call rotations as needed.

Required Skills & Qualifications:

  • 3 5 years of hands-on experience in Hadoop ecosystem (HDFS, Hive, Spark, Sqoop, Oozie, etc.).
  • Strong experience in automating data lake workflows and ETL processes.
  • Proficiency in scripting languages such as Python, Shell, or Scala.
  • Experience with scheduling and orchestration tools (e.g., Apache Airflow, Control-M, AutoSys).
  • Solid understanding of data modelling, data quality, and performance optimization.
  • Familiarity with cloud platforms (AWS, Azure, GCP) and big data services.
  • Excellent problem-solving and communication skills.

Preferred Qualifications:

  • Experience with Apache NiFi or similar data flow tools.
  • Exposure to CI/CD pipelines and DevOps practices.
  • Knowledge of data cataloguing and lineage tools.

Job Tags

Similar Jobs

Gables Search Group, Inc.

Remote Senior Wealth Advisor - RIA Firm Job at Gables Search Group, Inc.

 ...Remote Senior Wealth Advisor RIA Firm Location: Remote Open to candidates nationwide Job Type: Full-time | Remote | Wealth...  ...your first year A nationally expanding Registered Investment Advisory (RIA) firm is actively hiring experienced Wealth Advisors... 

Aerotek

General Production Worker Job at Aerotek

**Job Title: General Production Worker****Job Description:**We are seeking an enthusiastic general production worker to join our team, responsible for the operation of various high-speed manufacturing equipment.**Responsibilities**+ Setup and operate manufacturing machinery... 

Kilgore College

Contract - Fitness Class Instructors - Fitness Center Job at Kilgore College

 ...Boot Camp, HITT, Pilates, and more. The Contracted Fitness Instructor will be responsible for conducting effective and safe exercise classes, as assigned. Responsibilities: Teach the exercise class according to Policy and Procedure and within the description of the... 

Cordis

Clinical Sales Specialist (Cleveland, Ohio) Job at Cordis

 ...just the place for you. We are the people behind the people who keep saving lives. Responsibilities: J ob Summary : The Clinical Sales Specialist is responsible for supporting the cardiovascular and endovascular product portfolio within a defined territory... 

Kentuckiana Oral & Maxillofacial Surgery

Surgical Assistant Job at Kentuckiana Oral & Maxillofacial Surgery

 ...Qualifications: A high school diploma or equivalent BLS Certification (or completed within 1 month of hire) Strong teamwork and independent work capabilities, demonstrating good judgment and critical thinking. Passion for ensuring safety and exceptional attention to...