Starting in December 2025, the Hubs community started seeing posts that Kubernetes versions 1.34+ were not working with Hubs Community Edition. The symptoms were as follows: visiting an instance would fail with error codes like PR_CONNECT_RESET_ERROR or PR_END_OF_FILE_ERROR, and instance owners would see things like evicted pods, pods with ContainerStatusUnknown, and messages saying the node had run out of memory when using commands like kubectl get pods -n hcce and kubectl describe pod <podname> -n hcce.
It took awhile to get anything definitive, but the community investigated the problem, verified it, and found a solution. Initially, this was thought to be to add a memory request / limit to the HAProxy container, like so:
resources:
requests:
memory: "1Mi"
limits:
memory: "1Ti"
However, while this did get the instances working again, it was found that Reticulum’s memory usage had skyrocketed (going from a couple hundred Megabytes to around 1.7 Gigabytes). Further investigation found that the actual underlying issue for everything was that the container runtime that was being used (containerd) had set their file descriptor limit to infinity, which caused both HAProxy and Reticulum to allocate far more resources than they did previously and thus, the issues observed above.
So, the final solution turned out to be to add an environment variable to Reticulum to limit the number of ports it would allocate, and to add a custom command option to HAProxy with an additional call to ulimit to limit the number of file descriptors in the HAProxy container.
Reticulum
- name: ERL_MAX_PORTS
value: "1048576"
HAProxy
command: ["sh", "-c", 'ulimit -n 1048576 && /start.sh "$@"', "--"]
Once those changes are made to hcce.yaml, all that is needed is to run npm run apply.
To make things simple, we have released a new version of Community Edition to address this and you can simply update to that: Hubs Community Edition Deployment Scripts v2.1.1 Released
For more technical background, see https://github.com/Hubs-Foundation/hubs-cloud/pull/406