Data Engineer - AWS (Financial Data Reconciliation)
Posted on July 7, 2025
Job Description
- Key Responsibilities:
- 1. Data Pipeline Development:
- o Design and deploy AWS Glue ETL jobs (PySpark) to process and reconcile large-scale financial datasets from multiple sources (ERP, accounting software, databases, Excel).
- o Implement Redshift data warehousing for structured storage and efficient querying of transactional records.
- o Automate reconciliation workflows using AWS Step Functions and Lambda for validation and exception handling.
- 2. Data Reconciliation Logic:
- o Develop matching rules to validate transactional records against aggregated reports, ensuring consistency and accuracy.
- o Identify and resolve discrepancies (missing entries, mismatched values, classification errors) with detailed reporting.
- o Implement rules for cross-validating input credits against eligible transactions.
- o Implement data lineage and auditing mechanisms to track changes, pipeline failures, and reconciliation outcomes.
- o Generate Excel-based outputs (reports, exception summaries) with formatted templates for business users.
- 3. Monitoring & Alerting:
- o Configure CloudWatch alarms for job failures, data anomalies, and SLA breaches.
- o Generate and maintain audit logs for compliance.
- 4. Performance Optimization:
- o Optimize Redshift queries and PySpark jobs for high-volume data processing.
- o Use AWS Athena for ad-hoc analysis of reconciled datasets.
- 5. Collaboration & Documentation:
- o Work with finance and compliance teams to refine validation rules.
- o Document data flows, transformation logic, and reconciliation methodologies. Preferred Qualifications:
- Prior experience in financial data reconciliation (e.g., ledger matching, regulatory reporting).
- Familiarity with ERP/accounting systems (SAP, Oracle, Tally, etc.) as data sources.
- AWS Certification (Data Analytics/Big Data/Data Engineer) is a plus.
Required Skills
data engineer - aws (financial data reconciliation)