Designation-Data Engineer

Job Description

• Develop and maintain scalable data pipelines using tools like Apache Kafka, Apache Spark, or AWS Glue to ingest, process, and transform large datasets from various sources, ensuring efficient data flow and processing.
• Design and implement data models and schemas in data warehouses (e.g., Amazon Redshift, Google BigQuery) and data lakes (e.g., AWS S3, Azure Data Lake) to support analytics and reporting needs.
• Collaborate with data scientists and analysts to understand data requirements, ensuring data availability and accessibility for analytics, machine learning, and reporting.
• Utilize ETL tools and frameworks (e.g., Apache NiFi, Talend, or custom Python scripts) to automate data extraction, transformation, and loading processes, ensuring data quality and integrity.
• Monitor and optimize data pipeline performance using tools like Apache Airflow or AWS Step Functions, implementing best practices for data processing and workflow management.
• Write, test, and maintain scripts in Python, SQL, or Bash for data processing, automation tasks, and data validation, ensuring high code quality and performance.
• Implement CI/CD practices for data engineering workflows using tools like Jenkins, GitLab CI, or Azure DevOps, automating the deployment of data pipelines and infrastructure changes.
• Collaborate with DevOps teams to integrate data solutions into existing infrastructure, leveraging Infrastructure as Code (IaC) tools like Terraform or AWS CloudFormation for provisioning and managing resources.
• Manage containerized data applications using Docker and orchestrate them with Kubernetes, ensuring scalability and reliability of data processing applications.
• Implement monitoring and logging solutions using tools like Prometheus, Grafana, or ELK Stack to track data pipeline performance, troubleshoot issues, and ensure data quality.
• Ensure compliance with data governance, security best practices, and data privacy regulations, embedding DevSecOps principles in data workflows.
• Participate in code reviews and contribute to the development of best practices for data engineering, data quality, and DevOps methodologies.
• Contribute to the documentation of data architecture, processes, and workflows for knowledge sharing, compliance, and onboarding purposes.
• Demonstrate strong communication skills to collaborate effectively with cross-functional teams, including data science, analytics, and business stakeholders.

Desired Profile

• Expert proficiency in Terraform and extensive experience across at least two major cloud platforms (AWS, Azure, GCP).
• Strong hands-on experience with Kubernetes, Helm charts, and designing/optimizing CI/CD pipelines (e.g., Jenkins, GitLab CI) is essential.
• Proficiency in Python and scripting languages (Bash/PowerShell) is also a must.
• Experience with data-related tools and technologies, including Apache Kafka, Apache Spark, or AWS Glue for data processing, as well as familiarity with data storage solutions like Amazon Redshift, Google BigQuery, or Azure Data Lake, is highly valued.
• Valued experience includes leading cloud migration initiatives, contributing to RFP/RFI processes, and mentoring teams.
• Excellent problem-solving, communication, and collaboration skills are critical.
• Experience with configuration management tools (Ansible, Puppet) and a solid understanding of DevSecOps principles is required; experience with OpenShift is a plus.

Experience

• 6 years and above

Education

• B.Tech. / BS in Computer Science

Technical Skills & Certifications

• Certifications in cloud platforms (e.g., AWS Certified Data Analytics, Google Professional Data Engineer, Azure Data Engineer Associate).
• Proficiency in data processing tools and frameworks such as Apache Kafka, Apache Spark, and AWS Glue.
• Strong experience with data storage solutions like Amazon Redshift, Google BigQuery, and Azure Data Lake.
• Expertise in ETL tools and frameworks (e.g., Apache NiFi, Talend) and scripting languages (Python, SQL, Bash).
• Familiarity with CI/CD practices for data engineering workflows using tools like Jenkins, GitLab CI, or Azure DevOps.
• Experience with Infrastructure as Code (IaC) tools like Terraform for provisioning and managing data resources.
• Knowledge of containerization technologies (Docker) and orchestration (Kubernetes) for managing data applications.
• Understanding of monitoring and logging solutions (e.g., Prometheus, Grafana, ELK Stack) to ensure data pipeline performance and quality.
• Awareness of data governance, security best practices, and data privacy regulations, embedding DevSecOps principles in data workflows.