The hubble-metrics port of the cilium DaemonSet is not in sync with the hubble-metrics service

[kforsthoevel]

/kind bug

When setting up hubble-metrics via cluster.yaml:

[...]
networking:
    cilium:
      [...]
      hubble:
        enabled: true
        metrics:
        - dns:query;ignoreAAAA
        - drop
        - tcp
        - flow
        - icmp
        - http
[...]

A hubble-metrics service is created and the cilium pods expose the hubble-metrics port. Unfortunately, the port of the service and the port of the pod are not the same. The service uses the port 9965 (which is the port cilium uses). The pod uses the port 9091 (which cilium used a long time ago, now they use port 9965). Since the service is a headless services it must use the same port as the port exposed by the pod. For more explanation see this: kubernetes/kubernetes#20488 (comment)

1. What kops version are you running? The command kops version, will display
this information.

kops version
Client version: 1.30.3 (git-v1.30.3

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

kubectl version
Client Version: v1.30.8

3. What cloud provider are you using?

AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

5. What happened after the commands executed?

6. What did you expect to happen?

7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: null
  generation: 1
  name: [REDACTED]
spec:
  api:
    loadBalancer:
      class: Network
      type: Internal
  authorization:
    rbac: {}
  certManager:
    defaultIssuer: letsencrypt-live
    enabled: true
    hostedZoneIDs:
      - [REDACTED]
    managed: true
  channel: stable
  cloudLabels:
    Environment: [REDACTED]
    Owner: [REDACTED]
  cloudProvider: aws
  clusterAutoscaler:
    balanceSimilarNodeGroups: true
    cpuRequest: 100m
    enabled: true
    memoryRequest: 800Mi
    scaleDownUtilizationThreshold: "0.8"
    skipNodesWithLocalStorage: false
  configBase: s3://[REDACTED]
  containerRuntime: containerd
  etcdClusters:
    - cpuRequest: 200m
      etcdMembers:
        - encryptedVolume: true
          instanceGroup: master-eu-west-1a
          name: a
        - encryptedVolume: true
          instanceGroup: master-eu-west-1b
          name: b
        - encryptedVolume: true
          instanceGroup: master-eu-west-1c
          name: c
      memoryRequest: 100Mi
      name: main
    - cpuRequest: 100m
      etcdMembers:
        - encryptedVolume: true
          instanceGroup: master-eu-west-1a
          name: a
        - encryptedVolume: true
          instanceGroup: master-eu-west-1b
          name: b
        - encryptedVolume: true
          instanceGroup: master-eu-west-1c
          name: c
      memoryRequest: 100Mi
      name: events
  externalPolicies:
    node:
      - arn:aws:iam::[REDACTED]
  iam:
    allowContainerRegistry: true
    legacy: false
    serviceAccountExternalPermissions:
      - aws:
          policyARNs:
            - arn:aws:iam::[REDACTED]
        name: [REDACTED]
        namespace: default
    useServiceAccountExternalPermissions: true
  kubeDNS:
    nodeLocalDNS:
      enabled: false
    provider: CoreDNS
  kubeProxy:
    metricsBindAddress: 0.0.0.0
  kubelet:
    anonymousAuth: false
    authenticationTokenWebhook: true
    authorizationMode: Webhook
    cpuCFSQuota: false
  kubernetesApiAccess:
    - [REDACTED]
  kubernetesVersion: 1.30.8
  masterPublicName: api.[REDACTED]
  networkCIDR: [REDACTED]
  networkID: [REDACTED]
  networking:
    cilium:
      hubble:
        enabled: true
        metrics:
        - dns:query;ignoreAAAA
        - drop
        - tcp
        - flow
        - icmp
        - http
  nodeTerminationHandler:
    enableSQSTerminationDraining: true
    enabled: true
    managedASGTag: kubernetes.io/cluster/[REDACTED]
  nonMasqueradeCIDR: 100.64.0.0/10
  podIdentityWebhook:
    enabled: true
  serviceAccountIssuerDiscovery:
    discoveryStore: s3://[REDACTED]
    enableAWSOIDCProvider: true
  sshAccess:
    - [REDACTED]
  subnets:
    - cidr: [REDACTED]
      egress: [REDACTED]
      id: [REDACTED]
      name: eu-west-1a
      type: Private
      zone: eu-west-1a
    - cidr: [REDACTED]
      egress: [REDACTED]
      id: [REDACTED]
      name: eu-west-1b
      type: Private
      zone: eu-west-1b
    - cidr: [REDACTED]
      egress: [REDACTED]
      id: [REDACTED]
      name: eu-west-1c
      type: Private
      zone: eu-west-1c
    - cidr: [REDACTED]
      id: [REDACTED]
      name: utility-eu-west-1a
      type: Utility
      zone: eu-west-1a
    - cidr: [REDACTED]
      id: [REDACTED]
      name: utility-eu-west-1b
      type: Utility
      zone: [REDACTED]
    - cidr: [REDACTED]
      id: [REDACTED]
      name: utility-eu-west-1c
      type: Utility
      zone: eu-west-1c
  topology:
    dns:
      type: Public
  updatePolicy: external

---
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-11-27T12:26:41Z"
  labels:
    kops.k8s.io/cluster: [REDACTED]
  name: master-eu-west-1a
spec:
  additionalSecurityGroups:
    - [REDACTED]
  cloudLabels:
    Environment: [REDACTED]
    Owner: [REDACTED]
  image: 075585003325/Flatcar-stable-4081.2.0-arm64-hvm
  machineType: m6g.2xlarge
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-eu-west-1a
    type: control-plane
  role: Master
  subnets:
    - eu-west-1a

---
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-11-27T12:26:41Z"
  labels:
    kops.k8s.io/cluster: [REDACTED]
  name: master-eu-west-1b
spec:
  additionalSecurityGroups:
    - [REDACTED]
  cloudLabels:
    Environment: [REDACTED]
    Owner: [REDACTED]
  image: 075585003325/Flatcar-stable-4081.2.0-arm64-hvm
  machineType: m6g.2xlarge
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-eu-west-1b
    type: control-plane
  role: Master
  subnets:
    - eu-west-1b

---
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-11-27T12:26:41Z"
  labels:
    kops.k8s.io/cluster: [REDACTED]
  name: master-eu-west-1c
spec:
  additionalSecurityGroups:
    - [REDACTED]
  cloudLabels:
    Environment: [REDACTED]
    Owner: [REDACTED]
  image: 075585003325/Flatcar-stable-4081.2.0-arm64-hvm
  machineType: m6g.2xlarge
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-eu-west-1c
    type: control-plane
  role: Master
  subnets:
    - eu-west-1c

---
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-11-27T12:26:41Z"
  labels:
    kops.k8s.io/cluster: [REDACTED]
  name: nodes-eu-west-1a
spec:
  additionalSecurityGroups:
    - [REDACTED]
  cloudLabels:
    Environment: [REDACTED]
    Owner: [REDACTED]
    k8s.io/cluster-autoscaler/enabled: ""
    k8s.io/cluster-autoscaler/staging: ""
  image: 075585003325/Flatcar-stable-4081.2.0-arm64-hvm
  machineType: m6g.xlarge
  maxSize: 18
  minSize: 2
  nodeLabels:
    kops.k8s.io/instancegroup: nodes-eu-west-1a
    type: node
  role: Node
  subnets:
    - eu-west-1a

---
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-11-27T12:26:42Z"
  labels:
    kops.k8s.io/cluster: [REDACTED]
  name: nodes-eu-west-1b
spec:
  additionalSecurityGroups:
    - [REDACTED]
  cloudLabels:
    Environment: [REDACTED]
    Owner: [REDACTED]
    k8s.io/cluster-autoscaler/enabled: ""
    k8s.io/cluster-autoscaler/staging: ""
  image: 075585003325/Flatcar-stable-4081.2.0-arm64-hvm
  machineType: m6g.xlarge
  maxSize: 18
  minSize: 2
  nodeLabels:
    kops.k8s.io/instancegroup: nodes-eu-west-1b
    type: node
  role: Node
  subnets:
    - eu-west-1b

---
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-11-27T12:26:42Z"
  labels:
    kops.k8s.io/cluster: [REDACTED]
  name: nodes-eu-west-1c
spec:
  additionalSecurityGroups:
    - [REDACTED]
  cloudLabels:
    Environment: [REDACTED]
    Owner: [REDACTED]
    k8s.io/cluster-autoscaler/enabled: ""
    k8s.io/cluster-autoscaler/staging: ""
  image: 075585003325/Flatcar-stable-4081.2.0-arm64-hvm
  machineType: m6g.xlarge
  maxSize: 18
  minSize: 2
  nodeLabels:
    kops.k8s.io/instancegroup: nodes-eu-west-1c
    type: node
  role: Node
  subnets:
    - eu-west-1c

8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.

n/a

9. Anything else do we need to know?

I'm happy to provide a PR to fix it. Just let me know.