Hadoop / ETL Developer and PySpark - Multi Location - Full Time Job at Lorven technologies, Dallas, TX

V1VwTUFXSW1UcHZ2R1BSZ3pWZGRrb2Fub3c9PQ==
  • Lorven technologies
  • Dallas, TX

Job Description

Hi,

Our client is looking for a Hadoop / ETL Developer and PySpark with a Full Time project Dallas, TX / Atlanta, GA / Cleveland, OH / Pittsburgh, PA . below is the detailed requirement.

Job Title : Hadoop / ETL Developer and PySpark

Location : Dallas, TX / Atlanta, GA / Cleveland, OH / Pittsburgh, PA

Duration : Full Time

Job Description:

We are seeking a highly experienced Senior Big Data & DevOps Engineer with 8+ years of professional experience in HDFS, Hive, Impala, PySpark, Python, and DevOps automation tools such as uDeploy and Jenkins. This role is responsible for managing end-to-end data operations, including HDFS table management, ETL pipeline development, multi-environment codebase governance, platform upgrades, and production support.

The ideal candidate will have strong expertise in Linux system operations, Big Data ecosystem tools, and experience with incident/change management using ServiceNow. This role plays a key part in ensuring the stability, scalability, and efficiency of enterprise data platforms while enabling seamless development-to-production workflows.

Key Responsibilities:

Big Data Platform Operations:

  • Design, manage, and optimize HDFS directories, tables, and partitioning strategies.
  • Implement and enforce data retention and lifecycle policies across large datasets.
  • Administer Hive and Impala environments, ensuring high availability, performance tuning, and security compliance.

ETL Development & Data Engineering:

  • Develop scalable ETL pipelines using PySpark, Hive, and Python.
  • Build reusable frameworks for data ingestion, transformation, and aggregation.
  • Optimize job performance through query tuning, resource management, and parallelization.

DevOps & Environment Management:

  • Maintain and promote code across DEV, QA, UAT, and PROD environments.
  • Develop and support CI/CD pipelines using Jenkins and uDeploy for automated deployments.
  • Perform environment upgrades, patching, and dependency management aligned with release schedules.

Linux & Infrastructure Operations:

  • Execute Linux administration tasks including performance tuning, disk management, and scripting (Bash/Python).
  • Troubleshoot cluster-level issues including node failures, job errors, and distributed system anomalies.

Change & Incident Management:

  • Drive incident resolution and change execution using ServiceNow workflows.
  • Conduct root cause analysis (RCA) for critical issues and implement preventive solutions.
  • Ensure compliance with ITIL processes for change, incident, and problem management.

Collaboration & Technical Leadership:

  • Partner with data engineers, developers, DevOps teams, and business analysts to ensure operational excellence.
  • Mentor junior engineers and contribute to technical leadership across the Big Data ecosystem.
  • Document operational procedures, troubleshooting guides, and architectural decisions for internal knowledge sharing.

Required Qualifications:

  • Bachelor's degree in Computer Science, Information Technology, or related field.
  • 8+ years of experience in Big Data engineering and DevOps practices.
  • Advanced proficiency in HDFS, Hive, Impala, PySpark, Python, and Linux.
  • Proven experience with CI/CD tools such as Jenkins and uDeploy.
  • Strong understanding of ETL development, orchestration, and performance optimization.
  • Experience with ServiceNow for incident/change/problem management.
  • Excellent analytical, troubleshooting, and communication skills.

Nice to Have:

  • Exposure to cloud-based Big Data platforms (AWS EMR).
  • Familiarity with containerization (Docker, Kubernetes) and infrastructure automation tools (Ansible, Terraform).

Job Tags

Full time,

Similar Jobs

Kids First Services

ABA Therapist Job at Kids First Services

 ...Company Overview Are you an ABA Therapist looking for a rewarding career where you can make a lasting impact? At Kids First, our Behavior Technicians use many of the same skillspatient care, communication, and attention to detailto support children with autism... 

Unlimit Ventures

Entrepreneur in Residence (EIR) Job at Unlimit Ventures

 ...Role Summary The Entrepreneur in Residence (EIR) will be responsible for providing guidance and support to entrepreneurs and startup companies. The EIR will work closely with the CEO and other executive team members to provide advice on business strategy, product development... 

Assisted Home Health & Hospice

Print Room Operator Job at Assisted Home Health & Hospice

 ...salary can be matched) A very generous referral bonus earn up to $600* per referral. Well empower you to reach your highest levels of personal and professional development by providing up to $6,000* tuition education reimbursement. Vacation, sick leave, and... 

Hire Velocity

Bulk Truck CDL Delivery Driver Job at Hire Velocity

 ...We're looking for full-time Bulk Truck Drivers to deliver feed to local farms and a few commercial clients. This is a local position where you'll be home every night and part of a solid, hardworking team. It's a physical job - you'll be climbing, lifting, and... 

Cortica

Pediatric Medical Administrative Assistant Job at Cortica

 ...Endeavor, and at some Cortica locations, Transcranial Magnetic Stimulation) Order and track labs, genetic tests, and imaging. Intuit and anticipate the needs of medical staff. Perform a wide variety of clerical procedures that require independent judgment,...