Score:0

Elastisearch on kubernetes - memory_lock not working

gb flag

I'm trying to deploy elasticsearch on kubernetes, with memory_lock enabled, but nothing works all the time. First, I'll paste my kubernetes config

apiVersion: apps/v1
kind: Deployment
metadata:
  name: elasticsearch
  labels:
    app: elasticsearch
spec:
  replicas: 1
  template:
    metadata:
      name: elasticsearch
      labels:
        app: elasticsearch
    spec:
      containers:
        - name: elasticsearch
          image: pawelcyrklaf/elastic:latest
          imagePullPolicy: IfNotPresent
          env:
            - name: http.host
              value: 0.0.0.0
            - name: transport.host
              value: 0.0.0.0
            - name: xpack.ml.enabled
              value: "false"
            - name: xpack.security.enabled
              value: "false"
            - name: xpack.monitoring.enabled
              value: "false"
            - name: xpack.graph.enabled
              value: "false"
            - name: xpack.watcher.enabled
              value: "false"
            - name: cluster.name
              value: elasticsearch-monitoring
            - name: bootstrap.memory_lock
              value: "true"
            - name: ES_JAVA_OPTS
              value: -Xms4G -Xmx4G
            - name: discovery.zen.ping.unicast.hosts
              value: elasticsearch
            - name: node.master
              value: "true"
            - name: node.data
              value: "false"
            - name: node.ingest
              value: "false"
            - name: discovery.zen.minimum_master_nodes
              value: "1"
          ports:
            - containerPort: 9200
              name: http
            - containerPort: 9300
              name: http2
          resources:
            requests:
              memory: 2Gi
              cpu: 200m
            limits:
              memory: 4Gi
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              add:
                - IPC_LOCK
                - SYS_RESOURCE
            privileged: true
            procMount: Default
            readOnlyRootFilesystem: false
            runAsNonRoot: false
      restartPolicy: Always
  selector:
    matchLabels:
      app: elasticsearch
---
apiVersion: v1
kind: Service
metadata:
  name: elastic-service
  labels:
    name: elastic-service
spec:
  selector:
    app: elasticsearch
  ports:
    - name: http
      port: 9200
      protocol: TCP
      targetPort: 9200
    - name: transport
      port: 9300
      protocol: TCP
      targetPort: 9300
  type: NodePort

pawelcyrklaf/elastic:latest - it's my custom image with below configuration

FROM docker.elastic.co/elasticsearch/elasticsearch:6.3.0

COPY elasticsearch.yml /usr/share/elasticsearch/config/elasticsearch.yml
COPY elasticsearch /etc/sysconfig/elasticsearch
COPY elasticsearch.service /usr/lib/systemd/system/elasticsearch.service
COPY elasticsearch.service /etc/systemd/system/elasticsearch.service.d/override.conf
RUN echo vm.max_map_count=262144 >> /etc/sysctl.conf
RUN echo fs.file-max=131072 >> /etc/sysctl.conf
COPY limits.conf /etc/security/limits.conf

EXPOSE 9200

elasticsearch.yml

bootstrap.memory_lock: true
discovery.type: single-node
xpack.security.enabled: false
http.host: 0.0.0.0
network.host: 0.0.0.0

elasticsearch

ES_JAVA_OPTS="-Xms4g -Xmx4g"  
MAX_LOCKED_MEMORY=unlimited

elasticsearch.service

[Service]
LimitMEMLOCK=unlimited

limits.conf

elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited

Of course, I built the image, started deployment. All the files that are copied when building the image are in the pod, but memory_lock doesn't work anyway. In pod logs I have enter image description here

and

[7NlQMVt] memory locking requested for elasticsearch process but memory is not locked

I tried a newer version of elastisearch but that didn't work either. The problem is that every now and then elasticsearch eats up all the RAM and I have to add more RAM, although practically nothing happens on the application that uses elasticsearch.

I've read a lot of tutorials on google how to fix it, a lot of topics on stackoverflow, but nothing works for me. I tried to run this deployment on minikube, rancher and AWS EKS, but there is the same problem everywhere and I have no idea how to fix it :(

morsik avatar
tr flag
Not sure if related, but sysctls you're trying to set will do nothing because they are inside container and they seems to be at node-level, not namespaced. See https://docs.openshift.com/container-platform/4.8/nodes/containers/nodes-containers-sysctls.html Also: why would you copy systemd unit file when no systemd is involved at all when running application inside container? Seems like no reason to do so.
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.