Skip to content

Expose limits imposed by cgroups via port library APIs #1281

Open

Description

@ashu-mehra

Runtimes like JVM rely on resource limits like memory and cpus to adjust their internal parameters which includes heap size, GC helper threads, JIT helper threads etc.
Cgroups in Linux provide a mechanism to impose limits on such resources. Therefore, when running as part of a cgroup, the runtime should adjust its internal parameters based on the resource constraints imposed by the cgroup.
Currently the information about resource constraints imposed by cgroup is not provided by the port library in any way. If a runtime wants to take into consideration the limits imposed by the cgroup, it would have to figure this out on its own.
It would be useful for the runtimes if port library can provide APIs that would enable them to query the resource constraints imposed by the cgroup.
The proposal is to add mechanism for following tasks:

  1. Add capability to read /proc/<pid>/cgroup file.
    Using this file, the process can figure out the cgroup it belongs to for a given subsystem.
  2. Expose APIs for providing memory limits and CPU limits imposed by the cgroup. For start only these two limits would be exposed. More can be added later as and when required.
  3. Now, the above mechanism would work when running on bare metal or a VM, but would break when running in a docker container. This is because in a docker container the /proc/<pid>/cgroup gives the cgroup of the process on the host, and not the one in the container. AFAIK when running in container, the cgroup would always be the root cgroup. Therefore, we also need to detect whether the process is running in a docker container or not, and use correct cgroup to get the resource limits.

As an example, if I log in to a docker container, and cat /proc/1/cgroup, I get following contents:

$ docker exec -it dfee854963d4575d263785ddee0f3c398355b354e6608c5777a3a2e40fef3612 /bin/bash
root@dfee854963d4:/# cat /proc/1/cgroup
11:pids:/docker/dfee854963d4575d263785ddee0f3c398355b354e6608c5777a3a2e40fef3612
10:perf_event:/docker/dfee854963d4575d263785ddee0f3c398355b354e6608c5777a3a2e40fef3612
9:freezer:/docker/dfee854963d4575d263785ddee0f3c398355b354e6608c5777a3a2e40fef3612
8:cpuset:/docker/dfee854963d4575d263785ddee0f3c398355b354e6608c5777a3a2e40fef3612
7:blkio:/docker/dfee854963d4575d263785ddee0f3c398355b354e6608c5777a3a2e40fef3612
6:memory:/docker/dfee854963d4575d263785ddee0f3c398355b354e6608c5777a3a2e40fef3612
5:hugetlb:/docker/dfee854963d4575d263785ddee0f3c398355b354e6608c5777a3a2e40fef3612
4:devices:/docker/dfee854963d4575d263785ddee0f3c398355b354e6608c5777a3a2e40fef3612
3:net_prio,net_cls:/docker/dfee854963d4575d263785ddee0f3c398355b354e6608c5777a3a2e40fef3612
2:cpuacct,cpu:/docker/dfee854963d4575d263785ddee0f3c398355b354e6608c5777a3a2e40fef3612
1:name=systemd:/docker/dfee854963d4575d263785ddee0f3c398355b354e6608c5777a3a2e40fef3612

which shows the process is part of /docker/dfee854963d4575d263785ddee0f3c398355b354e6608c5777a3a2e40fef3612 cgroup in memory subsystem. However, if I go to /sys/fs/cgroup/memory I don't see any such cgroup, but the same cgroup is present in the host. Probably the /sys/fs/cgroup/memory/docker/<id> cgroup on the host is mapped as the root cgroup inside the container.
In short, /proc/<pid>/cgroup cannot be used for getting the cgroup when running in a docker container.

Breaking down this work into following lists of tasks:

  • Add support for reading files for detecting resource limits for cgroup v1
  • Determine if the we are running in container or not
  • Add port library API for exposing memory limit imposed by cgroup
  • Add port library API for exposing cpu limit imposed by cgroup
  • Detect if the system is using cgroup v1 or cgroup v2
  • Add support for reading files for detecting resource limits for cgroup v2

Activity

  1. ashu-mehra commented on Jun 21, 2017

    @ashu-mehra
    ContributorAuthor
  2. DanHeidinga commented on Jun 30, 2017

    @DanHeidinga
    Contributor

    Thanks @ashu-mehra for starting on this - it's an important feature and definitely needed in OMR!

    The API for this should make it easy to make existing code cgroup-aware without having to explicitly code a cgroup vs non-cgroup path in all code requesting i.e. sys max memory.

    A very rough sketch of what I'm thinking would be:

    /* Determine if running in a cgroup namespace */
    BOOLEAN omr_sysinfo_cgroup_in_cgroup()
    
    /* Indicate to port library that limits should be taken from the cgroup, not from the machine.
     * @return previous value of this setting
     */
    BOOLEAN omr_sysinfo_cgroup_enable_cgroup_limits()
    
    /* Should the port library respect cgroups in its normal API calls */
    BOOLEAN omr_sysinfo_cgroup_limits_enabled()
    
    /* Special accessors to get the cgroup-related limit for YYYY.
     * ie: long omr_sysinfo_cgroup_get_max_memory
     */
    XXXX omr_sysinfo_cgroup_get_max_YYYYY()
    

    And existing apis would be updated to be cgroup aware:

    XXXX omr_sysinfo_get_max_memory() {
       if (omr_sysinfo_cgroup_limits_enabled()) {
          return omr_sysinfo_get_max_memory();
       } else {
          ....
          /* existing code path */
          .....
       }
    

    This makes it easy for all consuming code to be aware of cgroup limits without having to be modified. Does this seem like a reasonable (though very rough) api sketch?

  3. ashu-mehra commented on Jun 30, 2017

    @ashu-mehra
    ContributorAuthor

    @DanHeidinga Thanks for jotting down the structure. I have exactly similar usage pattern in mind and I agree existing APIs should be updated to use cgroups when available.

    Couple of points:

    1. Do we need to make omr_sysinfo_cgroup_limits_enabled() an API, or just an internal helper function?
    2. Same for omr_sysinfo_cgroup_in_cgroup(). It is checking if we are in cgroup or not, which would be needed internally by omr_sysinfo_cgroup_enable_cgroup_limits(). So would it be better to just have it as helper function?
  4. DanHeidinga commented on Jun 30, 2017

    @DanHeidinga
    Contributor

    Do we need to make omr_sysinfo_cgroup_limits_enabled() an API, or just an internal helper function?

    My gut says to make it an API so that reporting / monitoring tools such as a javacore file can report which mode the port library has been put in.

    Same for omr_sysinfo_cgroup_in_cgroup().

    For the same kinds of reasons, we want this to be query-able by the runtime using the port library.

    And to be clear, I'm not sold on the method names proposed in the rough sketch. If you think of something better, go with it - these were off the cuff examples.

  5. ashu-mehra commented on Jul 4, 2017

    @ashu-mehra
    ContributorAuthor

    As there are quite a few things to be done to fully support cgroups, I have tried to break it down into smaller items and created a task list in the first comment.
    I will try to create pull requests targeting these tasks in the list.

    If I have missed anything, feel free to add tasks in the list in the first comment.

  6. 110 remaining items

  7. added a commit that references this issue on Jul 14, 2022
  8. added a commit that references this issue on Aug 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions