Luth Research is looking to solve the problem of managing and processing large-scale data deliveries to clients, ensuring data reliability and performance, and transforming unstructured data into structured feeds to enhance insights, efficiency, revenue, and profitability.
Requirements
- Expert proficiency in Python for data scripting, processing, and automation.
- Hands-on experience with Spark (PySpark) for distributed data processing.
- Strong command of Linux/Command Line and Hadoop for data management and operations.
- Proficiency in Postgres for database integration and querying.
- Experience utilizing LLMs for data transformation, pattern recognition, and solution development.
- Expert proficiency in data engineering tools and environments, including Python, Spark (PySpark), Linux/Command Line, Hadoop, Postgres, and LLMs.
- Familiarity with project management tools like JIRA for documentation, tracking, and ticket creation.
Responsibilities
- Design, implement, and maintain ETL pipelines and data processing workflows using Python, PySpark, and Hadoop in a Linux environment to ensure efficient transformation and delivery of large-scale data to clients.
- Monitor data pipelines, troubleshoot issues, and ensure compliance with SLAs to maintain high availability, reliability, and performance of data systems.
- Manage client deliveries of large-scale datasets, optimizing for scalability, security, and integration with Postgres databases.
- Utilize LLMs and other relevant tools to transform unstructured data into structured, actionable feeds, enhancing pattern recognition and insights for client solutions.
- Document work projects in JIRA, including tracking progress on assigned tasks and proposing/writing new tickets in collaboration with the team and management.
- Support management in data architecture planning, including system design and optimization, to support long-term growth and innovation in data solutions.
- Support management in conducting code reviews, providing technical guidance, and directing day-to-day activities of junior team members to foster team development and maintain code quality.
Other
- Commitment: Expert knowledge of market research and data engineering and data science practices, dedicated to the goals of the company while leading and promoting a positive, collaborative culture to ensure strong team relationships, overall satisfaction, and profitability.
- Accessibility: Must be able to access and function within areas key to productivity. Adequate manual dexterity to operate assigned and shared office equipment. Ability to travel on periodic overnight and extended stays and work extended hours when necessary.
- Communications: Must speak clearly and fluently to communicate on a wide range of technical and business topics. Superior reading comprehension and grammatically correct written communication skills, including the ability to document complex data processes and architectures. Must have the ability to understand and respond intelligently to business and technical correspondence. Advanced interpersonal skills to persuade, manage conflict, debate, negotiate, and mentor junior team members. Response to pressure should be calm, reasonable, and solution oriented.
- Judgment: Strong comprehension of job requirements and procedures. Can independently initiate, lead, and perform work assignments under minimal supervision that are broad, varied, and highly complex in nature. Employee must leverage educated, intuitive, and experienced judgment to make strategic decisions.
- Flexibility: Must thrive in a dynamic environment with the ability to juggle and prioritize multiple, competing tasks and demands, seeking supervisory assistance only as needed for high-level alignment.
