Epic Games needs a Data Engineer to design, build, and maintain their data infrastructure, ensuring the reliability and efficiency of data and systems used by the Ecosystem Security Core team. This role will focus on building and maintaining data pipelines, managing AWS infrastructure for security and analytics platforms, and collaborating with engineers, product managers, and data scientists to implement scalable data services that support the Ecosystem Security mission while ensuring user privacy and combating bad actors.
Requirements
- Experience of ETL technical design, automated data quality testing, QA and documentation, data warehousing, and data modeling
- Experience with Python for interaction with Web Services (e.g. Rest and Postman)
- Experience with using AWS, Databricks, Snowflake, Elastic or other comparable large scale analytics platforms
- Experience monitoring and managing databases (we use Elasticsearch/MongoDB/PostgreSQL)
- Experience with SQL
- Experience with data versioning tools
- Experience developing and maintaining data automation infrastructure for streaming & ETL pipelines, such as Apache Airflow
Responsibilities
- Design and implement automated end-to-end streaming & ETL process to prepare data for machine learning and ad-hoc analysis, including data anonymization
- Manage and scale the tools and technologies that we use to label data, which run on AWS
- Devise database structure and technology for storing and efficiently accessing large data sets (millions of records) of different types (text, images, videos, etc.)
- Use and implement data extraction APIs
- Write and invoke custom SQL procedures
- Support data versioning strategies using automated tools, such as DVC
- Support devising strategies for labeling of new data by humans
Other
- Strong analytical background: BSc or MSc in Computer Science/Software Engineering or related subject - candidates without a degree are welcome as long as they have proven extensive hands-on experience
