Senior Python Data Engineer (Databricks, Delta Lake)

We are seeking a highly skilled Senior Data Engineer with a strong background in Databricks, Data Lake, ElasticSearch and Database to join our team. As a Senior Data Engineer, you will play a crucial role in designing, building, and maintaining scalable data solutions to support our organization's data-driven decision-making processes. You will work closely with cross-functional teams to develop efficient data pipelines and ensure the reliability and quality of our data infrastructure

• Design, develop, and maintain data pipelines using Databricks, Delta Lake, and other relevant technologies to support data analytics and machine learning initiatives.


• Manage and optimize data storage solutions, including Data Lake, to ensure high performance and scalability.


• Work with NoSQL, SQL databases such as PostgreSQL, MongoDB to handle diverse data sets and unstructured data or structured data. 


• Implement ETL/ELT processes and data modeling solutions to transform raw data into valuable insights.


• Write complex SQL queries to manage, analyze, and manipulate data across various relational databases.


• Collaborate with data scientists, analysts, and other stakeholders to gather requirements and ensure data solutions align with business needs.


• Utilize programming languages like Python for data processing and automation of workflows.


• Monitor and troubleshoot data pipelines to ensure data quality, consistency, and availability.


• Leverage data pipeline orchestration tools such as Apache Airflow or Prefect to automate and schedule data workflows.


• Continuously improve data engineering practices by adopting new tools, techniques, and best practices in the industry.

Requirements:

• 6+ years of experience in data engineering or a related role.


• Hands-on experience with Databricks and Delta Lake.


• Proficient in SQL for querying and managing relational databases.


• Experience with NoSQL databases like MongoDB or Cassandra.


• Strong programming skills in Python for data processing.


• Familiarity with cloud platforms, linux.


• Experience with data pipeline orchestration tools such as Apache Airflow, Prefect, or similar.


• In-depth understanding of data warehousing concepts, ETL/ELT processes, and data modeling.


• Strong problem-solving skills with great attention to detail.

Preferred Qualifications:

• Experience with big data technologies like Apache Spark, Hadoop, ElasticSearch or Kafka.


• Familiarity with CI/CD practices and tools for data engineering

• Performance Bonuses
• 13th Month Salary
• Potential for International Travel for Training
• Team Building Activities
• Interaction with high level senior executives internationally
• Opportunity for growth

Cập nhật gần nhất lúc: 2024-10-25 11:25:02