Job Title: Python Data Engineer

Location: Houston, TX (Onsite role)

Duration: Long term contract

 

Job Description:
We are looking for a talented Data Engineer with expertise in Python data processing. The ideal candidate will have a strong background in Python API development, parallel data processing, and distributed systems design. You will be responsible for building and maintaining systems that handle large-scale data processing tasks, ensuring high performance and scalability.

Key Responsibilities:
Python API Development:
o    Develop and maintain RESTful APIs using Python web frameworks such as FastAPI or Django.
o    Collaborate with front-end developers to integrate user-facing elements with server-side logic.

Parallel Data Processing:
o    Utilize Pandas, NumPy, and other libraries to process large datasets efficiently.
o    Implement multithreading, multiprocessing, and asynchronous programming techniques.
o    Optimize data processing pipelines to handle millions of rows with minimal latency.

Distributed Systems Design:
o    Design and implement distributed systems with a focus on scalability and reliability.
o    Understand and apply core concepts such as load balancing and task queues.
o    Use Docker to containerize applications and manage dependencies.
o    (Preferred) Experience with Kubernetes for container orchestration.

Technical Communication:
o    Clearly articulate complex technical concepts to team members and stakeholders.
o    Document system designs, processes, and code effectively.
o    Collaborate with cross-functional teams to align on project goals and deliverables.


Must-Have Qualifications:
 Experience in Python Web Frameworks:
o    Proficiency with FastAPI, Django, or similar frameworks.
O   C# coding
o    Understanding of RESTful API principles and best practices.

Docker Knowledge:
 o    Ability to create and manage Docker Files.
 o    Experience with containerization for deployment and development workflows.
 
 Systems Design Understanding:
o    Basic knowledge of load balancing, task queues, and distributed system concepts.
o    Ability to design systems that are scalable and maintainable.

Concurrent and Parallel Computing Skills:
o    Proficiency in multithreading and multiprocessing without relying solely on external libraries or frameworks.
o    Familiarity with asynchronous programming, particularly asyncIO in Python.

Communication Skills:
o    Excellent technical communication abilities.
o    Experience collaborating in team environments and conveying complex ideas clearly.


Preferred Qualifications:
Education:
o    BS or MS in Computer Science

Advanced Data Processing Tools:
o    Experience with Polars, PySpark, or similar tools.
o    Handling of large-scale data processing tasks efficiently.

Distributed Computing Experience:
o    Hands-on experience with distributed architectures in Docker.
o    Familiarity with concepts like task queuing, MapReduce, and saga patterns.

Kubernetes Experience:
o    Knowledge of container orchestration using Kubernetes.
o    Experience deploying and managing applications in a Kubernetes cluster.

Problem-Solving at Scale: 
o    Demonstrated ability to solve complex problems using parallel or distributed computing. 
o    Innovative thinking beyond single-threaded processes.