Senior Data Engineer (AWS)

We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. The ideal candidate will have a strong background in designing, implementing, and optimizing scalable data pipelines and architectures on AWS. You will work closely with data scientists, analysts, and other engineering teams to build and maintain robust data solutions that drive business insights and innovation. Key Responsibilities:
Data Architecture and Design

Define and manage the architecture for real- time and batch data processing.
Design and implement scalable, secure, and efficient data pipelines and architectures on AWS

Data Pipeline Development

Develop solutions for data integration, data quality, and data governance.
Build ETL/ELT processes to ingest, transform, and store structured and unstructured data from diverse sources.

Optimization and Monitoring

Optimize data processing workflows for performance and cost- effectiveness.
Monitor data pipelines and systems, troubleshooting and resolving issues promptly.

Collaboration

Collaborate with DevOps teams to implement CI/CD for data workflows.
Partner with data scientists, analysts, and business teams to understand data needs and deliver high- quality solutions.

Security and Compliance

Ensure data solutions comply with organizational policies and industry security, privacy, and governance standards.
Implement data anonymization and encryption techniques as needed.

Innovation

Drive continuous improvement in processes, tools, and methodologies.
Stay updated with emerging AWS technologies and data engineering best practices.

Education

Bachelor’s or Master’s degree in Computer Science, Information Systems, Engineering, or a related field.

Experience

Proven experience designing and building data architectures and pipelines.
5+ years of experience as a Data Engineer or in a similar role, focusing on AWS cloud solutions.

Technical Skills

Experience with Linux and Bash commands.
Experience with data governance, security, and compliance standards.
Experience with query engines like Trino and Presto.
Strong knowledge of data modeling, data warehousing, and data lakes.
Experience with data tool stacks such as Spark, Flink, and Airflow.
Expertise in AWS services, including but not limited to S3, EMR, Lambda, and RDS.
Experience with data lake formats such as Delta or Iceberg.
Real- time data integration using Debezium and Kafka.
Containerization experience with Docker and Kubernetes (K8s).
Familiarity with CI/CD tools and infrastructure as code (e.g., Terraform).
Proficiency in SQL, Python, and Java for data processing and manipulation.

Soft Skills

Ability to work in an agile and fast- paced environment.
Excellent communication and collaboration abilities.
Strong problem- solving and analytical skills.

Nice to have

Knowledge of monitoring tools like CloudWatch, Datadog, or Prometheus.
Knowledge of machine learning workflows and integration with data pipelines.
Knowledge of modern data lakehouse platforms like Apache Doris and Pinot.

Social healthcare insurance and annual leave benefits are according to regulations.
Opportunities for training, professional skill enhancement, and career development.
A dynamic, professional, and youthful work environment that encourages creativity and personal development.

Cập nhật gần nhất lúc: 2025-01-03 05:25:02