gekkofs issueshttps://storage.bsc.es/gitlab/hpc/gekkofs/-/issues2023-06-29T08:34:04+02:00https://storage.bsc.es/gitlab/hpc/gekkofs/-/issues/266Leverage error injection2023-06-29T08:34:04+02:00Ramon NouLeverage error injectionThis issue wants to discuss the injection of errors for testing the replication and resilience code and complete it.
- [x] Read error injection (whole server-chunk)
- [ ] Read error injection (partial chunk, may need verification of the...This issue wants to discuss the injection of errors for testing the replication and resilience code and complete it.
- [x] Read error injection (whole server-chunk)
- [ ] Read error injection (partial chunk, may need verification of the chunk, that is expensive)
- [ ] Write error injection (whole server-chunk)
- [ ] Metadata error injection (Whole server)futurehttps://storage.bsc.es/gitlab/hpc/gekkofs/-/issues/265Docker support2023-05-10T07:56:29+02:00Ramon NouDocker supportSome folks at BSC want to execute GekkoFS server outside docker, and the clients (or docker run) inside docker.
They are asking me for advice, but I foresee a million of reason for that to not work.
I have found this: https://stackover...Some folks at BSC want to execute GekkoFS server outside docker, and the clients (or docker run) inside docker.
They are asking me for advice, but I foresee a million of reason for that to not work.
I have found this: https://stackoverflow.com/questions/51307496/ld-preload-in-docker
A quick list of things that we need/may fail inside the docker:
1. Network Docker <-> server (gkfs_hosts.txt) should be able to contact servers
2. Virtual mount point, should be plain. If we preload docker I think that the mount point would not be the same inside the image so the interception will not detect it.
3. Syscall_intercepthttps://storage.bsc.es/gitlab/hpc/gekkofs/-/issues/193Pytest check that the `ps` command exists2022-01-25T14:59:06+01:00Marc VefPytest check that the `ps` command existsGekkoFS python tests check if the daemon process is running before proceeding with the test. For this the `ps` CLI command is used. We are currently not check that the `ps` command exists on the system.
This check needs to be added to a...GekkoFS python tests check if the daemon process is running before proceeding with the test. For this the `ps` CLI command is used. We are currently not check that the `ps` command exists on the system.
This check needs to be added to avoid obscure error messages.futurehttps://storage.bsc.es/gitlab/hpc/gekkofs/-/issues/112Expand I/O tests for all possible I/O cases2020-06-10T13:34:00+02:00Marc VefExpand I/O tests for all possible I/O casesBelow a quick write up of tests we would need (from https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/47#480dff29884d162b586ebfc2577e9b3dd468f675 discussion)
---------------------------------------------------------------------...Below a quick write up of tests we would need (from https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/47#480dff29884d162b586ebfc2577e9b3dd468f675 discussion)
---------------------------------------------------------------------------
We would need tests for a single chunk and multiple chunks in terms of file size. In both cases we would need to consider the cases:
1. I/O size (i.e., transfer size per RPC) smaller than chunk size. In this case, we probably need to differentiate between very small transfer sizes (1KB or even smaller) and bigger ones (like half the chunk size)
2. I/O size (i.e., transfer size per RPC) equal the chunk size
3. I/O size (i.e., transfer size per RPC) bigger than the chunk size.
There are more permutations that we need to consider: Does one server only get one chunk, do all servers get one chunk, do all servers get multiple chunks?
For example with 512KB chunk size and 2 servers the transfer sizes per test could be:
- Does one server only get one chunk: 1KB, 512KB
- Do all servers get one chunk: 700KB, 1024KB
- Do all servers get multiple chunks: 1MB, 2MB, 64MB, 64.1MB
I tried to catch most of these in the `benchmark_check` script, but it is not complete.v1.0.0https://storage.bsc.es/gitlab/hpc/gekkofs/-/issues/111Follow-up from "Resolve "Add I/O tests""2020-06-10T12:48:30+02:00Ramon NouFollow-up from "Resolve "Add I/O tests""The following discussion from !47 should be addressed:
- [ ] @amiranda started a [discussion](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/47#note_2489): (+7 comments)
> why these values? are they dependent on the ch...The following discussion from !47 should be addressed:
- [ ] @amiranda started a [discussion](https://storage.bsc.es/gitlab/hpc/gekkofs/-/merge_requests/47#note_2489): (+7 comments)
> why these values? are they dependent on the chunk size? If so, can we find some way to propagate the configured chunk size to the tests so that they are more generic? E.g. add a property to `gkfs_daemon` or `gkfs_client`? It could be retrieved dynamically by parsing daemon log files or the harness could parse from the generated `config.hpp`.futurehttps://storage.bsc.es/gitlab/hpc/gekkofs/-/issues/95Improve tests for cp2020-03-19T19:42:12+01:00Alberto MirandaImprove tests for cpFrom @marcvef in !33:
> The current test for `cp` (`tests/shell/test_cp.py`) can be improved: I suggest to md5 checksum both files as well. Also due to how we handle chunk sizes, I suggest to test multiple file sizes. For example, mult...From @marcvef in !33:
> The current test for `cp` (`tests/shell/test_cp.py`) can be improved: I suggest to md5 checksum both files as well. Also due to how we handle chunk sizes, I suggest to test multiple file sizes. For example, multiples of the chunk size (which you currently have with 4MB), 1 file smaller than the chunk size (a few KB), and 1 file larger than the chunk size but not aligning to chunk boundaries (e.g., 2.5MB). I am just not sure if this is the right place for this test.
>
> I suggest to add a TODO so that we find out how big a write call is at most with the `cp` command. I would think 4MB is a single write call while a 100MB could cause multiple consecutive smaller writes to the file system.v1.0.0https://storage.bsc.es/gitlab/hpc/gekkofs/-/issues/93Add unit tests to testing harness2022-02-04T19:04:19+01:00Marc VefAdd unit tests to testing harnessv1.0.0Tunahan KayaTunahan Kayahttps://storage.bsc.es/gitlab/hpc/gekkofs/-/issues/83Rewrite tests using the new testing harness2020-09-16T09:22:04+02:00Alberto MirandaRewrite tests using the new testing harnessGiven the availability of the new testing harness, the few existing tests should be ported to the new system. This involves the following tasks:
- [x] rewrite directory tests
- [x] rewrite lseek tests
- [ ] rewrite I/O tests
- [ ] rewri...Given the availability of the new testing harness, the few existing tests should be ported to the new system. This involves the following tasks:
- [x] rewrite directory tests
- [x] rewrite lseek tests
- [ ] rewrite I/O tests
- [ ] rewrite path resolution tests
- [ ] rewrite truncate tests
- [ ] rewrite symlink testsv1.0.0Alberto MirandaAlberto Miranda