LMI is seeking a Lead Data Engineer to support Special Operations Command (SOCOM) by providing big data ETL support on classified data and systems to directly support client mission.
Requirements
- Expert in Python with significant on the job experience
- Significant experience with big data technology and ETL
- Experience with Airflow as an ETL workflow technology
- Advanced proficiency in SQL (especially Postgres and/or Trino) with significant on the job experience
- Experience with Kubernetes as an operating environment and Helm as infrastructure management
- Experience with AWS services and cloud architecture
- Experience with Postgres or Trino
Responsibilities
- Design, develop, and implement scalable data pipelines and ETL processes using Apache Airflow, with a focus on data for AI applications.
- Develop messaging solutions utilizing Kafka to support real-time data streaming and event-driven architectures.
- Build and maintain high-performance data retrieval solutions using ElasticSearch/OpenSearch.
- Implement and optimize Python-based data processing solutions.
- Integrate batch and streaming data processing techniques to enhance data availability and accessibility.
- Deploy and manage cloud-based infrastructure to support scalable and resilient data solutions, including AWS S3.
- Optimize data storage, retrieval, and processing efficiency, focusing on big data.
Other
- This position requires an active Top Secret SCI Clearance, for which you must be a US citizen.
- Bachelor’s degree or higher in data engineering, computer science, data science, or similar
- At least 5 years of experience in data engineering or related work
- Work closely with cross-functional teams to define data strategies and develop technical solutions aligned with mission objectives.
- Previous DOD experience
