Use Prometheus for in-cluster monitoring!
If you need a centralized view of multiple Kubernetes clusters, checkout Thanos.
Regex
Components
Infrastructure/ Cluster components should be monitored separately from your applications. This allows you to "kill" the application Prometheus in case you have screwed up in some way (e.g., messed up application metrics causing to have a billion labeled metrics).