Challenge
The challenge
Cost-focused cloud products need streaming that stays governable — and platforms that can evolve past monolith gravity.
Phase A — Streaming / Kafka posture
As a cost-focused cloud FinOps business, they needed Kafka used in a way that was governable and cost-aware — policies and practices that prevent sprawl, waste, and operational surprise.
Phase B — Platform gravity
Over time, a monolithic Python data platform limited scale, ownership boundaries, and operability.
- Service-oriented / microservices data paths
- Analytical storage suited to high-volume metrics-style workloads (ClickHouse)
- Observability first — logs, metrics, and traces as a design requirement, not an afterthought
Approach
What BlackPotato did
Phase 1 — Kafka policies
Earlier engagement focused on durable operating guidance for a cost-sensitive FinOps product.
- Assessed Kafka usage in the context of a FinOps product
- Defined policies and guidance for sustainable Kafka operations
- Delivered a durable operating playbook, not a one-off cluster tweak
Phase 2 — Data platform redesign
Follow-on work reshaped the platform for ownership, scale, and operability.
- Redesigned away from monolithic Python toward microservices
- Centered ClickHouse for analytical and cost-relevant data access patterns
- Embedded observability-first system design so reliability and cost signals stay visible
- Aligned platform shape with FinOps realities — high cardinality metrics, continuous ingest, clear ownership
Flow
Engagement evolution
- Kafka policy & governance
- Platform redesign: monolith Python → microservices
- ClickHouse + observability-first operations
Outcomes
Outcomes
- Clearer Kafka cost and ops policies for a FinOps-native business
- Path off a Python monolith onto a microservice data architecture
- ClickHouse-centered analytics posture
- Observability treated as a first-class platform concern
