Deploy Prometheus Monitoring with Prometheus Operator

Deploy Prometheus Monitoring with Prometheus Operator

This guide outlines the deployment process for a custom Prometheus monitoring setup using the Prometheus Operator.

Prerequisites

Ensure your Kubernetes cluster has:

1. Prometheus Operator
2. Kubernetes Metrics Server
3. Helm

Deployment Flow Chart

Below is a visual representation of the Prometheus deployment process:

This flow chart illustrates the key steps in deploying Prometheus monitoring using the Prometheus Operator.

Deployment Steps

1. Install Prometheus Operator: First, we need to install the Prometheus Operator:
   1. Create a namespace for monitoring:

   kubectl create namespace monitoring

1. 1. Add the Prometheus Operator Helm repository:

   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
   helm repo update

1. 1. Install the Prometheus Operator:

   helm install prometheus-operator prometheus-community/kube-prometheus-stack -n monitoring

1. 1. Verify the installation:

   kubectl get pods -n monitoring

1. Configure Values: Edit `values.yaml` to set your specific configuration:

   Deployment:
     name: <project-id>
     namespace: default
   
   PersistentVolume:
     node: jiriaf2302-control-plane
     path: /var/prom
     size: 5Gi
   
   Prometheus:
     serviceaccount: prometheus-k8s
     namespace: monitoring

  Key configurations:
  * `Deployment.name`: Used for job naming, persistent volume path, and service monitoring selection
  * `Deployment.namespace`: Specifies job namespace and namespace monitoring selection
  * `PersistentVolume.*`: Configures storage for Prometheus data
  * `Prometheus.*`: Sets Prometheus server details

  Note: Only those `servicemonitors` with the namespace `default` can be monitored. To monitor additional namespaces, additional configuration is required. Refer to the Prometheus Operator documentation on customizations for details.

1. Install the Custom Prometheus Helm Chart: Run the following command, replacing `<project-id>` with your identifier:

   helm install <project-id>-prom prom/ --set Deployment.name=<project-id>

  Example:

   ID=jlab-100g-nersc-ornl
   helm install $ID-prom prom/ --set Deployment.name=$ID

1. Verify Deployment: Check that all components are running:

   kubectl get pods -n monitoring
   kubectl get pv

1. Access Grafana Dashboard:
   1. Find the Grafana service:

   kubectl get svc -n monitoring

1. 1. Set up port forwarding:

   kubectl port-forward svc/prometheus-operator-grafana -n monitoring 3000:80

1. 1. Access Grafana at `http://localhost:3000` (default credentials: admin/admin)

Components Deployed

• Prometheus Server (`prometheus.yaml`)
• Persistent Volume for data storage (`prom-pv.yaml`)
• Empty directory creation job (`prom-create_emptydir.yaml`)

Integration with Workflows

This setup is designed to monitor services and jobs created by your workflow system.

Advanced Configuration

For further customization, refer to the Helm chart templates and `values.yaml`. Ensure your cluster has the necessary permissions and resources for persistent volumes and Prometheus server operation.

Troubleshooting

If you encounter issues:

1. Check pod status: `kubectl get pods -n monitoring`
2. View pod logs: `kubectl logs <pod-name> -n monitoring`
3. Ensure persistent volume is correctly bound: `kubectl get pv`
4. Verify Prometheus configuration: `kubectl get prometheus -n monitoring -o yaml`

For more help, consult the Prometheus Operator documentation.