At very large scale, several daemon processes hang with 100%(200%) CPU usage and clients experience timeouts
I tested GekkoFS at a relatively large scale. We had 200 nodes, 100 acted as servers while the rest 100 were clients. On each server node, we ran 4 daemons as we notice running multiple daemons on a single node can improve the performance. Each client, on the other hand, has 4 client processes running IOR benchmark. With a very high possibility, we will see some daemon processes hang with 100%(200%) CPU usage and clients experience timeouts, thus the benchmark could not run to finish. We are using UCX BTW.