JOB DESCRIPTION
Partner Collaboration and Template Management
● Work directly with business partners to understand their data transformation requirements
● Handle template modifications and updates efficiently to minimize disruption
● Document template specifications and maintain template change history
● Design and implement flexible NiFi templates that accommodate partner- specific data formats
● Provide technical support and guidance to partners during template implementation
● Conduct regular reviews with partners to optimize template performance
● Manage template versioning and changes based on partner requirements
Data Pipeline Development
● Create and manage NiFi templates for standardized data processing workflows
● Develop and optimize PostgreSQL- based data transformation processes
● Implement robust data synchronization solutions between AWS S3 and Google Cloud Storage
● Ensure data quality, consistency, and reliability across all systems
● Design, develop, and maintain data pipelines using Apache NiFi for data ingestion, transformation, and distribution
Template Development and Management
● Create template documentation and usage guidelines
● Manage template lifecycle and versioning
● Develop template migration strategies
● Implement template validation and testing procedures
● Design modular and reusable template components
● Monitor template performance and optimize as needed
System Optimization
● Implement error handling and recovery mechanisms
● Develop monitoring and alerting systems
● Monitor and optimize pipeline performance
● Ensure cost- effective utilization of cloud resources
Requirements:
Experience with relational SQL/ NoSQL databases
Experience with object- oriented/object function scripting languages: Python, Java, Scala, etc. is plus
Having knowledge in the fintech market, GT distribution, online payment is plus
Experience in building data models/ ETL pipeline/ data warehouse
Experience with one in big data tools: Hadoop, Spark, Kafka, etc.
Experience in Microsoft BI technologies and reporting solutions using Power BI/ PowerApps/ Microsoft Flow and other cloud- based solutions is plus
