Data Engineer (Senior) Hybrid

Automation & Optimization: Implement automated data quality checks using DataBricks
Data Pipeline Development: Design, develop, and maintain complex data pipelines using Databricks and PySpark to ingest, process, and transform large datasets from multiple sources.
Data Warehousing and Modeling: Develop and maintain data models and warehouse architecture that support data analytics and business intelligence.
Technical Innovation: Stay updated on new technologies and methodologies in data engineering, ML, AI, and LLM, and drive initiatives to incorporate them into our technology stack where relevant.
Collaboration & Mentoring: Work closely with cross- functional teams, including Data Science, Product, and Software Engineering, to meet business requirements and mentor junior team members.
Cloud Infrastructure Management: Manage and optimize data infrastructure components on AWS, utilizing services such as S3 and Kinetics
Machine Learning & AI Integration: Collaborate with Data Scientists and ML Engineers to design, deploy, and optimize ML/AI models, including LLMs, to enhance business processes and drive decision- making.

Experience: 5+ years of experience in data engineering, with a strong focus on DataBricks, PySpark, and AWS.
Educational Background: Bachelor&039;s or Master&039;s degree in Computer Science, Data Engineering, or a related field.
Technical Skills
Familiarity with AI and LLM technologies, including practical experience in fine- tuning, deploying, and optimizing LLMs.
Experience working with Machine Learning frameworks (e.g., TensorFlow, PyTorch) and deployment of models in a production environment.
Advanced proficiency in Databricks and PySpark for ETL processes and data transformation.
Strong programming skills in Python; experience with Scala is a plus.
In- depth experience with AWS services, particularly S3 and Kinetics.
Proficiency in SQL and relational database design.

Soft Skills

Ability to mentor and guide junior engineers, fostering a collaborative team environment.
Excellent communication and collaboration skills, with the ability to convey complex technical concepts to non- technical stakeholders.
Proactive problem- solving abilities and a strong attention to detail.
Preferred Qualifications
Exposure to real- time data processing frameworks (e.g., Apache Kafka)

Benefits

Collaborative and inclusive work culture.
Flexible work hours and remote work options.
Competitive salary and performance- based bonuses.
Opportunities for professional growth and certifications.

Cập nhật gần nhất lúc: 2024-11-13 01:25:03