CPU increase and memory leak

[tomer-epstein]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Hello Vector team
We see CPU increasement and a memory leak only in one of the agent pods.
We are using the agent as DaemonSets --> a vector pod for each node.
We are using the aggregator as StatefulSets.
The agent is sending the k8s logs to the aggregator.
Is that a bug? or did we configure it wrong?

Configuration

affinity: {}
  args:
  - -w
  - --config-dir
  - /etc/vector/
  - --log-format
  - json
  - -vv
  autoscaling:
    behavior: {}
    customMetric: {}
    enabled: false
    maxReplicas: 10
    minReplicas: 1
    targetCPUUtilizationPercentage: 80
  command: []
  commonLabels: {}
  containerPorts: []
  customConfig:
    api:
      enabled: false
    data_dir: /vector-data-dir
    sinks:
      prom_exporter:
        address: 0.0.0.0:9598
        inputs:
        - internal_metrics
        type: prometheus_exporter
      splunk_hec_dev_test:
        buffer:
          type: memory
        compression: gzip
        default_token: cf69e945-a1b2-a1b2-a1b2-be6777bea1b2
        encoding:
          codec: json
        endpoint: https://hec.a1b2.a234.log.cde.net.abc:443
        index: ugw_statistics
        inputs:
        - transform_remap_gateway_proxy_accesslog_test
        type: splunk_hec_logs
      vector_aggregator:
        address: abc-efgh-vector-aggregator.vector-aggregator.svc.cluster.local:7500
        inputs:
        - kubernetes_logs
        type: vector
    sources:
      internal_metrics:
        scrape_interval_secs: 2
        type: internal_metrics
      kubernetes_logs:
        glob_minimum_cooldown_ms: 1000
        max_read_bytes: 8192
        type: kubernetes_logs
    transforms:
      transform_remap_gateway_proxy_accesslog:
        inputs:
        - transform_route_gateway_proxy.access_log
        source: |
          ., err = parse_json(.message)
          if (err != null) {
            log("Remap gateway-proxy access_log, Unable to parse json: " + err, "error")
            abort
          }
          if (!exists(.targetLog) || .targetLog != "ABC-API-INSIGHTS-ACCESS-LOG") {
            abort
          }
          del(.targetLog)
          # verify insightsFields is not empty
          if (!exists(.insightsFields) || .insightsFields == null || .insightsFields == "") {
            log("Remap gateway-proxy access_log, insightsFields field is missing.")
            abort
          }
          result, err = merge(., .insightsFields)
          if (err != null) {
              log("Remap gateway-proxy access_log, Unable to merge insightsFields: " + err, "error")
          } else {
            . = result
          }
          del(.insightsFields)
          if (!exists(.insightsProfile) || .insightsProfile == null || .insightsProfile == "") {
            log("Remap gateway-proxy access_log, insightsProfile field is missing.")
            abort
          }
          path_parts, err = parse_regex(.path, r'(?P<pathname>[^?#]*)(?P<query>.*)')
          if (err != null) {
            log("Remap gateway-proxy access_log, Unable to parse regex: " + err, "error")
            abort
          }
          .path = path_parts.pathname
          if (.serviceTime != null) {
            .serviceTime = to_int!(.serviceTime)
          }
          if (.jwtZid != null) {
            .spcTenantId = .jwtZid
          }
          del(.jwtZid)
          if (.abcaaTenantId != null) {
            .spcTenantId = .abcaaTenantId
          }
          del(.abcaaTenantId)
          if (.jwtValStatus == null || .jwtSkipFailureReporting == true) {
            del(.jwtValStatus)
          }
          del(.jwtSkipFailureReporting)
          if (.rateLimitLimit == null) {
            del(.rateLimitLimit)
          }
          if (.rateLimitRemaining == null) {
            del(.rateLimitRemaining)
          }
          if (.rateLimitReset == null) {
            del(.rateLimitReset)
          }
          if (.rateLimitPolicy == null) {
            del(.rateLimitPolicy)
          }
        type: remap
      transform_remap_gateway_proxy_accesslog_all:
        inputs:
        - transform_remap_gateway_proxy_accesslog
        source: |
          if (.insightsProfile == "vector-e2e-test") {
            abort
          }
        type: remap
      transform_remap_gateway_proxy_accesslog_test:
        inputs:
        - transform_remap_gateway_proxy_accesslog
        source: |
          if (.insightsProfile != "vector-e2e-test") {
            abort
          }
          del(.insightsProfile)
        type: remap
      transform_route_gateway_proxy:
        inputs:
        - kubernetes_logs
        route:
          '*': .kubernetes.container_name == "gateway-proxy"
          access_log: starts_with(string!(.message), "{") && ends_with(string!(.message),
            "}")
        type: route
  dataDir: ""
  defaultVolumeMounts:
  - mountPath: /var/log/
    name: var-log
    readOnly: true
  - mountPath: /var/lib
    name: var-lib
    readOnly: true
  defaultVolumes:
  - hostPath:
      path: /var/log/
    name: var-log
  - hostPath:
      path: /var/lib/
    name: var-lib
  dnsConfig: {}
  dnsPolicy: ClusterFirst
  env: []
  envFrom: []
  existingConfigMaps: []
  extraContainers: []
  extraVolumeMounts: []
  extraVolumes:
  - name: secret-volume
    secret:
      secretName: vector-values
  fullnameOverride: ""
  global: {}
  haproxy:
    affinity: {}
    autoscaling:
      customMetric: {}
      enabled: false
      maxReplicas: 10
      minReplicas: 1
      targetCPUUtilizationPercentage: 80
    containerPorts: []
    customConfig: ""
    enabled: false
    existingConfigMap: ""
    extraContainers: []
    extraVolumeMounts: []
    extraVolumes: []
    image:
      pullPolicy: IfNotPresent
      pullSecrets: []
      repository: haproxytech/haproxy-alpine
      tag: 2.6.12
    initContainers: []
    livenessProbe:
      tcpSocket:
        port: 1024
    nodeSelector: {}
    podAnnotations: {}
    podLabels: {}
    podPriorityClassName: ""
    podSecurityContext: {}
    readinessProbe:
      tcpSocket:
        port: 1024
    replicas: 1
    resources: {}
    rollWorkload: true
    securityContext: {}
    service:
      annotations: {}
      externalTrafficPolicy: ""
      ipFamilies: []
      ipFamilyPolicy: ""
      loadBalancerIP: ""
      ports: []
      topologyKeys: []
      type: ClusterIP
    serviceAccount:
      annotations: {}
      automountToken: true
      create: true
    strategy: {}
    terminationGracePeriodSeconds: 60
    tolerations: []
  image:
    pullPolicy: IfNotPresent
    pullSecrets:
    - name: vector
    repository: build-releases-external.common.cdn.repositories.cloud.abc/timberio/vector
    tag: 0.32.1-distroless-libc
  ingress:
    annotations: {}
    className: ""
    enabled: false
    hosts: []
    tls: []
  initContainers: []
  lifecycle: {}
  livenessProbe: {}
  logLevel: info
  minReadySeconds: 0
  nameOverride: ""
  nodeSelector: {}
  persistence:
    accessModes:
    - ReadWriteOnce
    enabled: false
    existingClaim: ""
    finalizers:
    - kubernetes.io/pvc-protection
    hostPath:
      enabled: true
      path: /var/lib/vector
    selectors: {}
    size: 10Gi
  podAnnotations: {}
  podDisruptionBudget:
    enabled: false
    minAvailable: 1
  podHostNetwork: false
  podLabels:
    sidecar.istio.io/inject: "true"
    vector.dev/exclude: "true"
  podManagementPolicy: OrderedReady
  podMonitor:
    additionalLabels: {}
    enabled: false
    honorLabels: false
    honorTimestamps: true
    jobLabel: app.kubernetes.io/name
    metricRelabelings: []
    path: /metrics
    port: prom-exporter
    relabelings: []
  podPriorityClassName: ""
  podSecurityContext: {}
  psp:
    create: false
  rbac:
    create: true
  readinessProbe: {}
  replicas: 1
  resources:
    limits:
      cpu: 1000m
      memory: 2Gi
    requests:
      cpu: 10m
      memory: 128Mi
  role: Agent
  rollWorkload: false
  secrets:
    generic: {}
  securityContext: {}
  service:
    annotations: {}
    enabled: true
    externalTrafficPolicy: ""
    ipFamilies: []
    ipFamilyPolicy: ""
    loadBalancerIP: ""
    ports: []
    topologyKeys: []
    type: ClusterIP
  serviceAccount:
    annotations: {}
    automountToken: true
    create: true
  serviceHeadless:
    enabled: true
  terminationGracePeriodSeconds: 60
  tolerations:
  - effect: NoSchedule
    key: WorkGroup
    operator: Equal
    value: abproxy
  - effect: NoExecute
    key: WorkGroup
    operator: Equal
    value: abproxy
  topologySpreadConstraints: []
  ugwSecretConfigEnabled: false
  updateStrategy: {}
  workloadResourceAnnotations: {}

Version

0.32.1-distroless-libc

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

[bruceg]

This does look like a bug.

Are you able to upgrade to version 0.34.0? There were a couple of memory leaks that were handled in recent versions that may resolve your issue.

Also, do you have any metrics that could be having unbounded cardinality? This could cause memory growth through the prometheus_exporter sink, though perhaps not as rapidly as what you are seeing.

[gadisn]

@bruceg thanks.
Can you please elaborate on the unbounded cardinality metric question?
The metrics are the built-in ones from the various components (e.g. sinks) which are being used.
Is there a specific source/transform/sink you can point at which might have this problematic metric?

[bruceg]

Hmm, I keyed on the Prometheus component in the config above and misread it as a source (so used to reading sources first). I see you don't have any metric sources other than internal_metrics, which should have effectively single cardinality for its metrics, so that was a red herring. However, there were a couple of buffer-related memory leaks that were addressed between version 0.32.0 and 0.34.0. Are you able to upgrade to 0.34.0?

[gadisn]

Thanks, yes, an upgrade is certainly an option, we'll try it

[tomer-epstein]

We upgraded to version 0.34.0, but still see increment in cpu & memory.

[valerypetrov]

Hey guys, we have the same issue. By the way, I found very strange behavior on the internal_metrics sink. Please take a look:

 N=10; for ((i=1; i<=N; i++)); do curl -s  http://localhost:9598/metrics | wc -l; sleep 1; done
14329
14330
14332
14333
14333
14335
14336
14337
14337
14337

Seems the issue with vector_component_recieved_bytes_total metric:

curl -s  http://localhost:9598/metrics |  grep vector_component_received_bytes_total | wc -l
13886

The metrics are:

vector_component_received_bytes_total{component_id="tcp_input_no_tls",component_kind="source",component_name="tcp_input_no_tls",component_type="socket",host="HOST",peer_addr="IP:55080",protocol="tcp"} 310 1700521527947
vector_component_received_bytes_total{component_id="tcp_input_no_tls",component_kind="source",component_name="tcp_input_no_tls",component_type="socket",host="HOST",peer_addr="IP:55086",protocol="tcp"} 310 1700521527947

We do have an F5 loadbalancer for each vector instance. So, in these metrics, we will have only a set of F5's IPs and a bunch of ports.

[valerypetrov]

Seems like the upgrade to 0.34.1 resolved the issue with the memory leak & CPU increase.

[pront]

Seems like the upgrade to 0.34.1 resolved the issue with the memory leak & CPU increase.

Hello @tomer-epstein, can you try v0.34.1 as well?

[tomer-epstein]

Hello @pront
Upgraded to version 0.34.1 .
We see an improvement, but can still see increment in memory & cpu,

[vibhavachar]

Seeing this issue in 0.28 version, will try to upgrade. Will disabling internal metrics solves the issue?

[sharonx]

I'm using 0.34.1 and we are experiencing the same issue - 2 out of 14 pods - Daemonset, agent mode are running OOM every hour or so. The memory consumption looks steady for a while and spikes up very quickly and runs out of memory.

[tomer-epstein]

Hello,
Using version 0.34.1.
We experience out of memory for the agent pods.
We also see log messages:

• "level":"ERROR","msg":"Healthcheck failed.","error":"Queues are full"
• "level":"ERROR","msg":"Healthcheck timed out."