Data Engineer
Data engineers optimise data processing tool-chains and data flow across functional teams. Data engineers will work in partnership with the software development, data science, GIS and wider MapAction teams on data processing activities.
Within this role you will have the opportunity to be involved in:
- Creating and maintaining optimal, performant data pipeline architecture to ensure data is sourced, enhanced and packaged
Role specific criteria
The criteria listed here are in addition to the general criteria outlined for all volunteers
Essential
- At least one year of professional experience working in the area of data engineering is required
- Programming ability in Python or other programming languages
- Shell scripting
- Knowledge and experience with utilities such as Hadoop, Spark, Hive, Storm
- Understanding of how to set-up and use Apache Airflow or similar e.g. Luigi, Oozie etc.
- Databases (NoSQL, SQL, Graph Dbs, Principles of RDBs)
- Data pipeline design skills (ETL, ELT)
Desirable
- Knowledge of cloud platforms like GCP, AWS or Azure
- Knowledgeable on Windows, Mac and Linux systems
- System administration of Linux & Windows preferable
- Thorough grounding in containerisation e.g. Docker, Kubernetes
- Complete understanding of CI/CD pipelines
- Understands continuous Integration e.g. Jenkins, GitOps or similar
- Great with source control
- Proficient with source code management tools such as Git, Bitbucket etc.
- Understanding of the Software Development Life cycle (preferably Agile)
- Ability to demonstrate grasp of “Infrastructure as Code” concepts using tools such as Puppet, Ansible, Chef or Terraform
- Ability to work in a secure environment and diligence to adhere to security principles
- Experience in understanding partner needs and requirements, in particular working with non-technical clients
- An interest in software-engineering or data-science e.g. ML / DL / AI / Statistics
- Project management experience
- Humanitarian knowledge/experience
- Adventure or professional travel experience
- Relevant international work experience
- Knowledge of languages (particularly French and Spanish)