We are seeking an experienced Data Engineer with strong SPARK, SACALA to join our data engineering team. The ideal candidate will have a solid background in data ingestion, data processing, and data management using technologies such as Spark & SCALA. Knowledge on SNOFLAKE & DBT are nice to have.

Knowledge on Apache Spark, Hive, and HDFS would be must have.. You will be responsible for migrating and testing data pipelines and ensuring efficient data storage and processing. In addition, you will work with CI/CD tools, such as Jenkins, and schedulers like CTRL-M, to automate data workflows.

Responsibilities

· Will work as an Individual contributor on Big Data platform delivering projects as mandated by business

· Perform tasks related to migrating & testing applications on Snowflake, Snowpark, Apache Spark, Hive , on new retail banking data Lakehouse.

· Good to have knowledge on DBT.

· Perform extensive testing on the developed code, debugging & fixing issues.

· Manage DevOPS as required & defined on the project

· Automate processes wherever required to improve efficiency of testing team.

· Co-ordinate with diverse geographically located teams to complete assigned activities.

· Participate in daily various agile ceremonies during the ongoing sprint and complete tasks as required.

· Conduct bug-free release validations and produce metrics, tests, and defect reports.

· Assist in developing and enforcing team guidelines.

· Being spoc for the team and provide information/status on ongoing activities on regular basis to management whenever needed.