Setting up observability is a journey, not a feature toggle. Around this time, I started consolidating logs, metrics, and traces into a cohesive pipeline. The ELK stack (Elasticsearch, Logstash, Kibana) was my weapon of choice.
The setup involved:
- Filebeat agents on application nodes
- Logstash pipelines for transformation and enrichment
- Elasticsearch for storage and indexing
- Kibana dashboards for insights
Lessons Learned:
- Keep Logstash filters simple — they’re deceptively hard to debug.
- Use structured logging early. JSON logs save your sanity later.
- Index lifecycle policies help manage storage costs over time.
Observability isn't about pretty dashboards. It's about finding the "why" when something breaks — and getting there before your users do.