Hadoop Lead

About this role

• Lead a mid-size team of Data/ML Engineers.
• Create and maintain optimal data pipeline architecture.
• Assemble large, complex data sets that meet functional / non-functional business requirements.
• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
• Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using ETL processes and modern cloud technologies.
• Take ownership or clarification of requirements and solutions proposition before implementation.
• Lead the building of scaled machine learning production systems by designing pipelines and engineering infrastructure.
• Facilitate the development and deployment of offline ML models into production through the use of scalable tools and services to handle machine learning training and inference processes.
• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability.
• Create data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader.
• Keep up to date with latest open-source tools for data engineering.
• Mentor junior team members in ML Production best practices.
• Design, develop, implement, and debug large and complex data platforms.
• Analyze and improve performance of existing platforms.
• Implement new technologies, policies and practices that can help increase resiliency, automation and improve platform health.
• Identifying technology gaps and help business build and deliver viable solutions.
• Drive the evolution of Data & Services products/platforms with an impact-focused on data science and engineering.
• Participate in the development of data and analytic infrastructure for product development.
• Continuously innovate and determine new approaches, tools, techniques & technologies to solve business problems and generate business insights & recommendations.
• Partner with roles across the organization including consultants, engineering, and sales to determine the highest priority problems to solve.
• Evaluate trade-offs between many possible analytics solutions to a problem, taking into account usability, technical feasibility, timelines, and differing stakeholder opinions to make a decision.
• Break large solutions into smaller, releasable milestones to collect data and feedback from product managers, clients, and other stakeholders.
• Ensure proper data governance policies are followed by implementing or validating Data Lineage, Quality checks, classification, etc.
• Work with small, cross-functional teams to define the vision, establish team culture and processes.
• Consistently focus on key drivers of organization value and prioritize operational activities accordingly.
• Maintain awareness of relevant technical and product trends through self-learning/study, training classes, and job shadowing.
  
  

• Demonstrated basic knowledge of statistical analytical techniques, coding, and data engineering.
• Experience developing and configuring dashboards is a plus.
• Demonstrated judgement when escalating issues to the project team.
• High proficiency in Python/Spark, Hadoop platforms & tools (Hive, Impala, Airflow, NiFi), SQL.
• Curiosity, creativity, and excitement for technology and innovation.
• Demonstrated quantitative and problem-solving abilities.
• Ability to multi-task and strong attention to detail.
• Motivation, flexibility, self-direction, and desire to thrive on small project teams.
• Expert proficiency in using Python/Scala, Spark(tuning jobs), SQL, Hadoop platforms to build Big Data products & platforms.
• Familiarity with RESTful APIs and micro-services architectures
• Hands-on experience with cloud computing and big data frameworks e.g. GCP, AWS, Azure, Flink, Elasticsearch, and Beam.
• Knowledge in MLOps frameworks such as TensorFlow Extended, Kubeflow, or MLFlow.
• Experience with data pipeline and workflow management tools: NIFI, Airflow.
• Comfortable in developing shell scripts for automation.
• Proficient in standard software development, such as version control, testing, and deployment.
• Experience with visualization tools like tableau, looker.
• At least 5 year leading collaborative work in complex engineering projects in an Agile setting e.g. Scrum.
• Extensive data warehousing/data lake development experience with strong data modeling and data integration experience.
• Good SQL and higher-level programming languages with solid knowledge of data mining, machine learning algorithms and tools.
• Strong hands-on experience in Analytics & Computer Science.
• Demonstrated basic knowledge of statistical analytical techniques, coding, and data engineering.
• Experience in building and deploying production-level data-driven applications and data processing workflows/pipelines and/or implementing machine learning systems at scale in Java, Scala, or Python and deliver analytics involving all phases like data ingestion, feature engineering, modeling, tuning, evaluating, monitoring, and presenting.
• Outstanding communication and organizational skills.
• Strong English written and verbal communication skills.
• At least 10 years of relevant hands-on experience as a Data Engineer in an individual contributor capacity.
• Able to lead the implementation of machine learning production systems.
• Demonstrated ability, through hands-on experience, to develop production machine learning pipelines.
• At least a bachelor?s degree in computer architecture, Computer Science, Electrical Engineering or equivalent experience. Postgraduate degree is an advantage.
  
  

• Financial Institution or a Payments experience a plus
• Experience in managing/working in Agile teams
• Experience developing and configuring dashboards