Job Description:
Design data collection processes including textual and multi-dimensional data.
Implement data searches and access APIs, between edge computing, cloud stack and prem storage.
Develop and optimize the data and metadata storage and databases for our AI and Algorithms to access train and test data efficiently.
Analyze existing data schemes and processes, suggest and implement ways to improve them.
Analyze cloud data usage and optimize cost structures while maintaining low latency access using common and proprietary services.
Work closely with Software, AI and Algorithms teams on system integration, on-going data requests, data access and more.
Job Qualifications:
Bachelor’s degree in a quantitative field such as computer science, math, engineering, etc.
5+ years of professional experience in data engineering or related roles, including hands-on experience in designing and implementing data pipelines, using orchestration frameworks like Apache Airflow, Prefect, etc
5+ years of experience and deep understanding of SQL and NoSQL DBs (MySql, Elastic, MongoDB, Postgres).
5+ years of SW development, 2+ are Python.
Full control of version control systems (e.g. Git), containerization (e.g. Docker), and CI/CD pipelines.
Strong collaborator with teams and peers
Innovative with a growth mindset.
Advantages:
Experience with streaming technologies, unstructured data and different data types (other than text and tables).
Experience with Hadoop, Map Reduce, Spark, or other distributed computing platform.
Experience with Kubernetes based systems.
Experience with Data visualization and analysis tools such as – Grafana, ELK, Kibana, etc.
Apply for this position
Company Occupation:
High Tech
Company Size:
Small (0 - 50)