How to monitor your Kubernetes Kapsule cluster with Cockpit using Promtail

You can now send data plane logs from your Kapsule or Kosmos clusters to Cockpit, providing centralized, real-time access to application and system logs. Reduce complexity and manual work thanks to this integration, powered by a Promtail deployment via Easy Deploy.

This feature allows you to:

• Enhance observability: View logs from all your Kubernetes containers in one place.
• Simplify troubleshooting: Quickly drill down into specific pods or containers without needing to configure a separate logging stack.

Before you startLink to this anchor

To complete the actions presented below, you must have:

• A running Kapsule or Kosmos cluster.
• An API Key with IAM permissions to:
  • edit your cluster (KubernetesFullAccess or KubernetesSystemMastersGroupAccess)
  • write on Cockpit (ObservabilityFullAccess)
• A token with permissions to push to, and query logs from Cockpit

Architecture and limitationsLink to this anchor

Control plane vs. data planeLink to this anchor

• Control plane: Fully managed by Scaleway. Users can already monitor control plane components (e.g., kube-apiserver, CCM, CSI) via Cockpit.
• Data plane: Runs in your Scaleway Project (customer-managed instances, kubelet, containerd, customer Pods, etc.). You have full access to the data plane, including the ability to SSH into nodes.

Because the data plane is entirely under your control, logs from any components running on these nodes are considered your own data. Consequently, shipping these logs to Cockpit is billed based on data ingestion.

How it worksLink to this anchor

The system leverages Promtail (a lightweight log collector) running on your Kapsule or Kosmos cluster. Promtail forwards logs to the Loki endpoint of your Cockpit instance:

1. Promtail can collect logs from:
   • Container stdout/stderr (pods)
   • systemd journal (e.g., kubelet.service)
2. The app automatically creates a custom datasource called kubernetes-logs and a Cockpit token with push logs permission.
3. Log data is transmitted to Cockpit (Loki).
4. Cockpit stores and indexes these logs.

Step-by-step: Enabling container logs in CockpitLink to this anchor

You can use Scaleway’s Easy Deploy to add a Promtail deployment to your cluster:

1. Log in to the Scaleway console and go to your Kubernetes cluster.
2. Navigate to the Easy Deploy tab.
3. Select Promtail for Cockpit from the library.
4. Deploy the application. Promtail will install on your cluster with default settings that:
   • Collect container logs for all namespaces (by default).
   • Collect systemd journal logs (e.g., kubelet.service).
   • Forward logs securely to Cockpit.
   Note

   You can edit the default deployment configuration to filter logs by source (under config.snippets.scrapeConfigs in the YAML file). For example:

   cockpit_promtail_scrape_config_pods: "namespace1,namespace2"
   cockpit_promtail_scrape_config_journal: "kubelet.service,kube-proxy.service"
   

Example Promtail configurationLink to this anchor

Below is a simplified snippet of the configuration that Easy Deploy generates by default:

config:
  clients:
    - bearer_token: "{{{ cockpit_bearer_token }}}" # Automatically set by Easy Deploy
      url: "{{{ cockpit_loki_push_url }}}" # Automatically set by Easy Deploy
  snippets:
    scrapeConfigs: |
      {{{- cockpit_promtail_scrape_config_pods }}} # Default: log all pods
      {{{- cockpit_promtail_scrape_config_journal }}} # Default: log all system components
extraVolumeMounts:
  - mountPath: /var/log/journal
    name: journal
    readOnly: true
extraVolumes:
  - hostPath:
      path: /var/log/journal
    name: journal

Visualizing logs in CockpitLink to this anchor

Once Promtail is running:

1. Go to the Cockpit section of the Scaleway console, then click Open dashboards.
2. Log into Grafana using your Cockpit credentials.
3. In Grafana’s menu, go to dashboards and select Kubernetes Cluster Pod Log.
4. Filter by:
   • Datasource which is automatically created upon deployment, and visible in the Cockpit console
   • Cluster Name ( e.g. my-kapsule-cluster)
   • namespace, pod, or container labels to isolate specific workloads
   • Time range to limit how far back in history you want to query
5. Analyze logs in real-time or historical mode to troubleshoot issues, watch for errors, or track performance.

Usage and pricingLink to this anchor

Sending logs to Cockpit is billed based on the total volume of logs ingested. Learn more about how you are billed for using Cockpit with Scaleway data in the Cockpit FAQ.

Key points include:

• Logging rate: The more logs you produce (e.g. high-traffic workloads or verbose logging), the higher the bill.
• Filtering: Limit logs to critical namespaces or system components only.

extraLimitsConfig: |
  readline_rate_enabled: true # rate limiting
  readline_rate: 10000 # log lines / sec
  readline_burst: 10000 # cap for burst lines
  readline_rate_drop: true # drop excess lines

Security considerationsLink to this anchor

• Authentication: The Promtail client uses a Cockpit Bearer Token to authenticate. Keep this token secret; do not store it in publicly accessible repos.
• Encryption: Communication between Promtail and Cockpit (HTTPS) encrypts logs in transit.
• Access Control: Ensure only trusted team members can deploy Easy Deploy applications or modify cluster-level configurations.

TroubleshootingLink to this anchor

• No logs appearing in Cockpit:

  • Verify that the Promtail pod is running.

    kubectl get pods -n <promtail-namespace>
    

  • Inspect Promtail logs for errors.

    kubectl logs <promtail-pod-name> -n <promtail-namespace>
    

• High log ingestion cost:

  • Review your deployment configuration to filter out verbose logs or unneeded namespaces.
  • Check log ingestion rate in the dedicated dashboards for unusual spikes.

Further resourcesLink to this anchor

• Observability Cockpit Overview
• Push logs to Cockpit (How-To)
• Send logs from your Kubernetes cluster to your Cockpit (How-To)
• Send metrics from your Kubernetes cluster to your Cockpit (How-To)
• Promtail Documentation
• Scaleway Kubernetes Kapsule and Kosmos Documentation