Sr. Big Data Lead Engineer

Sr BigData Lead Engineer

Location: Milpitas, CA, USA

Type: 12+ month Contract


GSPANN Technologies is looking for motivated Sr BigData Lead Engineer to join our team. Our clients are based all across the USA and expectation is that resource will be able to commute to Milpitas, CA, USA.


Roles & Responsibilities:

  • BIG Data Platform (Google DataProc - cloud solution for hadoop platforms, OnPrem hadoop cluster, RTES Cluster, EDW, Teradata, Netezza)

  • Data Acquisition/Data Ingestion (Kafka, RDBMS sources, File transfer from Raw to Stage Data Layer)

  • Data Processing - downstream (Stage to gold Layers of datasets)

  • Advanced Analytics and reporting (Tableau, Marketing and campaign data analytics) tools including Urban Airship, Kira Advanced Analytics, Self-Serve using BIGQUERY/BIGTABLE

  • Automated Monitoring / Alerting solutions for Production Engineering for platform, Flows and end point service monitoring using tools like - NetDiagnostics/ Cavisson, Splunk Dashboards, Stackdriver, Inhouse developed tools

  • Designed and deployed Machine Learning on Azkaban Logs for alerting and predictive failures for critical flows

  • Defined Strategies for deploying Transient Clusters and defining the Preemptible/non Preemptible design policies for all the flows running in GCP production environment

  • Expertise in running offshore / onshore team structures

  • Expertise in transition of complex environments from Development Engineering to Production Engineering (deploying effective monitoring/ alerting processes and Operational processes)



    Required Skills and Experience:

  • BIGDATA Hadoop, DATAPROC, ORACLE, Azkaban, STACKDRIVER, EDW, CICD, Confluence /JIRA /Service Now.

  • Prior experience as a functional analyst with ERP system is desirable.


    Nice to have:

    Google Cloud, BIGTABLE, BIGQUERY, Machine Learning /Deep learning/Logs Analytics See less