Senior Data Engineer

About the Role: Our client is looking for a Senior Data Engineer who will work closely with teams like, marketing, finance, product, operations, and help solving some of their problems from a data perspective. Day to Day Responsibilities: Build data processing pipelines to clean and transform a wide variety of data sources to enable further analyses and modeling Follow and advocate software engineering best practices to achieve predictable development cycles and high-quality output Constantly learn new platforms, technologies and tools that are best suited for the problems the team is tackling from time to time Must Have: Overall you have 12+ years of experience in software engineering. You have at least 5+ years of experience working as Data engineer dealing with many terabytes and petabytes of data, with data strong experience in data storage, data models and processing systems for structured and/or unstructured data Worked as Senior/Lead position for 2+ years. You have good experience in using cluster computing systems such as Spark, and know the programming interfaces and performance tuning of these very well You are a proficient software developer and have built and released production quality systems using languages like Java/Scala, Python, etc. You have a good practical insight into the commonly used machine learning algorithms and data mining techniques and how to use these for solving real world problems. You understand what it takes to train predictive models from messy data, and how to evaluate them with realistic metrics. Expert level data analysis and data modeling skills on large, enterprise-based initiatives Experience with relational SQL and NoSQL databases. Experience in systems like KafkaExperience with AWS cloud services: EC2, EMR, RDS, Redshift, snowflake Expert level experience in Data governance tools. Strong verbal and written communication skills with an ability to express complex business rules in a data model and to interpret the data rules from a data model. Nice to Have: Experience in data visualization using R, Shiny and JavaScript Frameworks (D3) will be a plus Knowledgeable on different database and data warehousing systems like BigQuery, Teradata Possess strong data visualization skills using programmatic tools (ggplot2) and tools like Tableau You are proficient in at least one programming language commonly used for data analysis (like R/Python)