Big Data Engineers / Leads
Big Data, Amazon Web Services, SQL, PySpark, Spark, Python, Hive, Airflow
Description
GSPANN is looking for Big Data Engineers and Leads. Our culture fosters individual initiative and excellence. Join our global workforce to unleash a pool of opportunities.
Who We Are
GSPANN has been in business for over a decade, with over 2000 employees worldwide, and servicing some of the largest retail, high technology, and manufacturing clients in North America. We provide an environment that enables career growth while still interacting with company leadership.
Visit Why GSPANN for more information.
Location: Bangalore
Role Type: Full Time
Published On: 16 February 2024
Experience: 4 - 10 Years
Description
GSPANN is looking for Big Data Engineers and Leads. Our culture fosters individual initiative and excellence. Join our global workforce to unleash a pool of opportunities.
Role and Responsibilities
- Actively participate in all phases of the software development lifecycle, including requirements gathering, functional and technical design, development, testing, roll-out, and support.
- Solve complex business problems by utilizing a disciplined development methodology.
- Produce scalable, flexible, efficient, and supportable solutions using appropriate technologies.
- Analyze source and target system data. Map the transformation that meets the requirements.
- Interact with the client and onsite coordinators during different phases of a project.
- Design and implement product features in collaboration with business and technology stakeholders.
- Anticipate, identify, and solve issues concerning data management to improve data quality.
- Clean, prepare, and optimize data at scale for ingestion and consumption.
- Support the implementation of new data management projects and re-structure the current data architecture.
- Implement automated workflows and routines using workflow scheduling tools.
- Understand and use continuous integration, test-driven development, and production deployment frameworks.
- Participate in design, code, test plans, and dataset implementation performed by other data engineers to maintain data engineering standards.
- Analyze and profile data to design scalable solutions.
- Troubleshoot straightforward data issues and perform root cause analysis to resolve product issues proactively.
- Build data lake solutions leveraging one or more of the following AWS, EMR, S3, Hive, and PySpark.
Skills and Experience
- Must have a bachelor’s degree in computer science.
- 3+ years of experience in developing data and analytic solutions.
- Expertise in relational SQL, scripting languages such as Python, and working with agile teams.
- Hands-on with source control tools such as GitHub and related development processes.
- Good understanding of workflow scheduling tools such as Airflow.
- In-depth knowledge of AWS cloud (S3, EMR, Databricks).
- Should be passionate about data solutions.
- Strong problem-solving and analytical mindset.
- Thorough with the design, development, and testing of data pipelines.
