Alongside partners with organizations to drive digital transformation and build high-performing tech teams.
We are looking for a Senior Data Engineer to join a project with one of our clients operating in the financial markets industry.
You will design, build, and operate real-time data products on an enterprise data platform, taking ownership from event ingestion and stream processing to the APIs and services used by consumers. This is a hands-on role combining advanced streaming engineering, production ownership, and technical guidance.
Responsabilities:
- Design and deliver end-to-end streaming data products using Apache Flink, Apache Kafka, and event-driven architectures.
- Develop stateful stream-processing applications, including windowing, event-time handling, watermarks, checkpointing, and exactly-once processing guarantees.
- Build and maintain resilient ingestion, transformation, and delivery pipelines for real-time and batch workloads.
- Manage Kafka topics, partitioning, consumer groups, schemas, and schema evolution.
- Design and develop REST APIs, streaming interfaces, and microservices with clear contracts, versioning, authentication, and rate limiting.
- Deploy and operate services on AWS, using Infrastructure as Code with Terraform.
- Embed data quality controls, automated testing, observability, and actionable alerting throughout the delivery lifecycle.
- Build and improve CI/CD pipelines, including automated deployment, rollback, and safe release practices.
- Monitor production services against agreed SLAs, investigate incidents, and optimise latency, throughput, reliability, and cost.
- Apply security best practices across authentication, authorisation, encryption, secrets management, and least-privilege access.
- Translate business requirements into technical designs, challenge assumptions, and propose practical solutions.
- Review code and designs, mentor engineers, and contribute to engineering standards and technical documentation.
Requirements
- 7+ years of professional experience in data engineering, including at least 3 years building and operating real-time or streaming systems in production.
- Strong hands-on experience with Apache Flink in production, including DataStream APIs and Flink SQL.
- Strong Java skills, with experience in Python/PyFlink or Scala also valued.
- Solid experience with Apache Kafka, event-driven architectures, and schema management using Avro and Schema Registry.
- Proven experience building APIs and microservices using REST and OpenAPI.
- Strong AWS experience, including relevant services such as MSK, Kinesis, Managed Service for Apache Flink, Lambda, API Gateway, ECS/EKS, S3, and IAM.
- Strong SQL and data modelling skills across streaming and batch workloads.
- Experience with CI/CD tooling such as GitLab CI or GitHub Actions, alongside automated unit, integration, contract, and load testing.
- Experience with observability tools such as CloudWatch, OpenTelemetry, Prometheus, and Grafana.
- Fluency in English, both written and spoken.
Benefits
- The opportunity to build production-grade real-time data products within the financial markets industry.
- End-to-end ownership across streaming pipelines, APIs, and production operations.
- Collaboration with platform engineers, product owners, and business stakeholders.
- Scope to influence technical decisions, improve engineering practices, and mentor other engineers.
- Hybrid working model in Porto.