Data Engineer

Job description

• Large multinational company
• CBD location
• Competitive salary and excellent perks

Your new role

As a Big Data Engineer you will be part of core big data technology and design team. Person would be entrusted to developed solutions/design ideas, identify design ideas to enable the software to meet the acceptance and success criteria. Work with architects/BA to build data component on the Big data environment.

RESPONSIBILITIES:
As a key member of the technical team alongside Engineers, Data Scientists and Data Users, you will be expected to define and contribute at a high-level to many aspects of their collaborative Agile development process:

• Software design, development, automated testing of new and existing components in an Agile, DevOps and dynamic environment
• Promoting development standards, code reviews, mentoring, knowledge sharing
• Product and feature design, scrum story writing
• Work on Data Engineering pipelines
• Product support & troubleshooting
• Implement the tools and processes, handling performance, scale, availability, accuracy and monitoring
• Liaison with BAs to ensure that requirements are correctly interpreted and implemented. Liaison with Testers to ensure that they understand how requirements have been implemented - so that they can be effectively tested.
• Participation in regular planning and status meetings. Input to the development process - through the involvement in Sprint reviews and retrospectives. Input into system architecture and design.
• Peer code reviews.
• 3rd line support.

CHALLENGES:

• Integrating with an established, complex project working to tight deadlines
• Responsible for implementing streaming data platform

What you'll need to succeed

MUST HAVE SKILLS:

• Experience in Big data tools: Spark, Kafka
• Must have experience implementing and designed streaming platform Big Data
• Experienced in Python is must. Python development and design
• Experience with traditional databases such as Oracle, Teradata, SQL Server
• Strong understanding of core features of Python and when to use them
• Experience supporting and working with cross-functional teams in a dynamic environment. Must be able to quickly understand technical and business requirements and can translate them into technical implementations
• Experience with Agile Development methodologies
• Experience with data ingestion and transformation
• Strong communication + someone who can articulate their solutions and drive a leadership approach across the development

Follow links to apply