Pyramid Systems is looking for a Data Engineer (Senior) who is passionate about bringing creative architect solutions to end customers.
Requirements
- Experience with Databricks, Structured Streaming, Delta Lake concepts, and Delta Live Tables required
- Advanced level SQL experience (Joins, Aggregation, Windowing functions, Common Table Expressions, RDBMS schema design, Postgres performance optimization)
- Expertise in Spark/Python/Databricks, Data Lake and SQL
- Create, maintain, and manage Spark Structured Steaming jobs, including using the newer Delta Live Tables and/or DBT
- Maintain and update Python-based data processing scripts executed on AWS Lambdas
- Unit tests for all the Spark, Python data processing and Lambda codes
- Experience with Docker, Jenkins, and CloudWatch
Responsibilities
- Plan, create, and maintain data architectures, ensuring alignment with business requirements
- Obtain data, formulate dataset processes, and store optimized data
- Identify problems and inefficiencies and apply solutions
- Determine tasks where manual participation can be eliminated with automation.
- Identify and optimize data bottlenecks, leveraging automation where possible
- Create and manage data lifecycle policies (retention, backups/restore, etc)
- In-depth knowledge for creating, maintaining, and managing ETL/ELT pipelines
Other
- 8+ years of IT experience focusing on enterprise data architecture and management
- Must be able to obtain a Public Trust security clearance
- MUST BE US CITIZEN
- Bachelor degree required
- Ability to thrive in a team-based environment
