Main tasks for the Data Engineer - Team Lead are outlined as follows:
• Provides strategic oversight of data engineering activities, shaping the overall platform
strategy and data architecture.
• Acts as a technical leader for the team setting best practices for other developers to
follow
• Collaborates with senior leadership and other external stakeholders by acting as the
senior technical resource for the group
• Monitor and optimize ARC platform data pipelines to ensure efficient data flow and
system performance, applying best practices in data engineering for ongoing
maintenance.
• Collaborate with stakeholders and data SMEs to identify and implement
enhancements to ARC platform capabilities, ensuring it meets evolving needs.
• Conduct advanced data profiling and quality assessments to monitor the health of the
data within the platform, using tools and techniques for ongoing quality assurance.
• Implement and manage data access policies in coordination with the Data
Governance and Engineering team, ensuring secure and compliant access to data for
new features and capabilities.
• Troubleshoot and resolve data-related issues in collaboration with operational support
teams.
• Work with source systems to identify data for transport as required.
- Profile source datasets.
- Retrieve data using different methodologies such as SQL, NoSQL, and RESTful.
- Prepare documentation containing high-level descriptions of data.
- Optimize transfer by defining and using change data capture techniques.
• Apply and/or update classifications and levels of sensitivity to all personal health
information (PHI) data elements in data sets landing on the platform, as assigned.
• Provide custom data quality rules and associated data quality scans on the data
estate.
• Facilitate access to all draft materials on request and ensure that adequate feedback
is obtained from appropriate stakeholders prior to delivery of any final documents.
Main deliverables/work products for this resource, are focused in the following areas:
• Comprehensive documentation of data pipeline optimizations, enhancements, and
operational support activities, serving as a reference for the Data Governance and
Engineering team and stakeholders.
• Operational support documentation and issue resolution reports, detailing
troubleshooting processes, solutions, and preventive measures for future incidents.
• Documentation of enhancements or updates to data profiles, extraction processes,
and transport designs to improve ARC platform capabilities.
• Data extract and transport testing/quality assurance documentation for any platform
capability changes or updates.
• Training and support materials for ARC platform users and support staff, as required.
• Proactive knowledge transfer to ensure seamless collaboration with the Data
Governance and Engineering team, operational stakeholders, and technical teams on
evolving ARC platform requirements and capabilities.
• Written status reports which outline activities completed to date. Reports to include
any outstanding issues/risks that may impact the achievement of objectives at the
current time or in the immediate future.
• Other deliverables as required.
