Middle/Senior Data Engineer

• Design, build, and maintain efficient and reliable real-time data pipelines to support live sports scoring and analytics.


• Develop architectures and databases optimized for fast querying and real-time data processing on physical servers.


• Work with sports data feeds and APIs to capture, store, and normalize data for real-time usage.


• Collaborate with the frontend and analytics teams to implement features that rely on real-time data processing.


• Ensure system consistency and data integrity with high availability and performance.


• Monitor system performance, implement necessary adjustments, and respond to system-wide issues in a timely manner.


• Manage data storage solutions on physical infrastructure, focusing on security and data recovery.


• Utilize modern data processing technologies like Kafka and Spark, suited to streaming and real-time analytics.


• Document system architecture and maintain data flow diagrams for internal use.


• Develop strategies for integrating multiple data sources and APIs, ensuring smooth data flow and consistency across different platforms.


• Implement data crawling techniques to enhance the collection processes from various data sources and improve the breadth of data available for analytics.


• Conduct research to identify new data sources and evaluate their potential integration into the system to improve product offerings.

• Bachelor's or Master’s degree in Computer Science, Engineering, or a related field.


• 3+ years of experience as a Data Engineer, with a strong background in real-time data systems.


• Proficiency in Python, Java, or similar programming languages.


• Experience with real-time data processing tools (Kafka, Spark Streaming, etc.).


• Familiarity with sports data APIs and their integration into live systems.


• Strong understanding of SQL and NoSQL databases, such as PostgreSQL and Cassandra.


• Knowledge of managing physical servers and optimizing server performance.


• Proven ability to troubleshoot and optimize data pipelines and architectures for real-time operations.


• Excellent project management skills and the ability to work collaboratively in a team environment.


• Experience in data mapping and normalization across various data sources and APIs.


• Skilled in designing and implementing data crawlers to automate data collection and enhance real-time data availability.

• 
  
  • Workout for FREE at the 5-star Center.
  
  
  • Pay social insurance according to state regulations.
  
  
  • Participate in team building every year.
  
  
  • Friendly and fun working environment.
  
  

Cập nhật gần nhất lúc: 2024-08-22 07:40:02