Distributed systems
A failed replica should not end the request.
I rebuilt a property lookup service with a cache tier, replicated storage, bounded failover and three implementations of the same contract.
My role
I extended the course exercise into a standalone repository with Python, Java and Go servers, conformance checks, failure injection and deployment tooling.
Where it stands
Working personal systems project with public code and reproducible scripts. It is not a claimed production service.
What changed.
One API contract, replayed across three implementations, with recorded behavior when a replica dies or stops answering.
Keep the contract
I implemented the dataset tier in three languages and built a conformance tool that replays keys against each. I also preserved an older protobuf contract to check compatibility in both directions.
Failover has a budget
The HTTP tier rotates between replicas, caches successful lookups and retries safe reads when a replica is unavailable or exceeds its deadline. It does not blindly replay arbitrary operations. Errors are not cached.
Make failure observable
Health checks, metrics and failure-injection scripts make it possible to see which replica answered and which failed first. The useful demonstration is a request completing through the remaining replica, followed by recovery.
The stack
- Python
- Go
- Java
- gRPC
- Docker
- Kubernetes