תיאור המשרה
We are searching for an innovative and experienced Senior Big Data Engineer that will join us and be part of our data group.
As a Senior Big Data Engineer, you will:
· Be a part of a cross functional team of data, backend and DevOps engineers.
· Work closely with Data Scientists to understand needs and design for data layout and infrastructure used in data ingestion for ML and analytics use-cases.
· Lead the architecture, planning, design and development of mission-critical, diverse and large-scale data pipelines and data lakes over both public and on-prem cloud solutions.
· Establish policies for data storage and versioning over peta-bytes scale complex storage solutions of multiple data lakes.
· Drive for continuous innovation and improvement in Final’s data stack.
דרישות התפקיד
At least 5 years of experience working as a Big Data Engineer or Data Architect
· At least 3 years of experience working in python development with emphasis on data analysis tools such as numpy, pandas, polars, Jupyter notebook.
· Proven understanding in designing, developing and optimizing complex solutions that move and/or manipulate large volumes of data.
· Sound understanding of different big data file formats such as Parquet, DeltaLake, AVRO, HDF5.
· Hands-on experience with distributed query engines such as Spark and Hadoop.
· Experience with implementing open-source solutions on existing large-scale systems.
· Experience working on large scale and complex on-premises systems.
· Experience with Docker, Linux, CI/CD tools and concepts, Kubernetes.
· Experience with data pipelining tools such as Airflow, Kubeflow or similar.
· Understanding of ML concepts and processes.
· BSc / MSc degree in Computer Science/ Engineering / Mathematics or Statistics.
Advantages:
· Experience and understanding of various storage protocols (NFS, S3, CIFS)
· Experience working with data versioning tools such as DVC, LakeFS, GitLFS or similar technologies.
· Experience working with AWS data processing tools and concepts.
· Hands-on experience with DataBricks platform.
· Hands-on experience in ML frameworks and models training and implementation.
· Experience with various databases technologies, including Vertica, MongoDB, Neo4j, Postgres, influxDB
· Hands-on experience in lower-level programming languages such as C++ or RUS
Senior Big Data Engineer
הגשת מועמדות מותנית באישור נציג משאבי אנוש. אם טרם קבעת ראיון, עשה זאת דרך האזור האישי
היקף משרה
מלאה
מודל עבודה
היברידי
תחום
תוכנה
תת-תחום
Senior BigData Developer