- Jun 19, 2023
-
-
Marc Vef authored
-
Marc Vef authored
Resolve "Support for (parallel) append operations" This MR adds (parallel) append support for write operations. There was already some append code available that was run for each `pwrite` when the file size was updated. As a result, parts of strings were serialized and deserialized within RocksDB's merge operation even if not needed. Previously, `open()` was returning `ENOTSUP` when `O_APPEND` was used. When removing this statement, append was not functional due to how size updates and the append case worked. Overall, `gkfs_pwrite()` which first updates the file size and then writes the file was quite messy with unused return values and arguments. Further, the server calculated the updated size without regard on what occurred in the KV store. Therefore, as part of this MR, the entire update size process within `pwrite()` was refactored. Parallel appends are achieved by hooking into RocksDB's `Merge Operator` which is triggered at some point (e.g., during `Get()`). Without append being used, the offset is known to the client already and therefore the file size is updated to `offset + count` set in `gkfs_pwrite()`. There is no further coordination required since overlapping offsets are the user's responsibility. The code path for non-append operations was slightly optimized but largely remains the same. Append operations are treated differently because it is not clear during a write operation where a process calling `write()` should start writing. Using the EOF information that is loaded during open may be outdated when multiple processes try to append at the same time -> causing a race condition. Since the size update on the daemon is atomic, a process (updating the size before performing a write) can be reserved a corresponding byte interval `[EOF, EOF + count]`. Now, calling `Merge()` on RocksDB does not trigger a Merge operation since multiple Merges are batched before the operation is run. For append, the Merge operation is forced by running `Get()` on RocksDB. The corresponding Merge operation then responds the starting write offset to the updating size process. Therefore, appends are more expensive than non-appends. Lastly, some missing documentation was added. As reported, this MR adds support for the DASI application, used in IO-SEA. Note: This MR does not consider failing writes which would require us to collapse a reserved interval and tie up the hole in the file. Closes #254 Closes #12 Closes #12 and #254 See merge request !164
-
Marc Vef authored
-
Marc Vef authored
-
Marc Vef authored
-
Marc Vef authored
-
Marc Vef authored
-
Marc Vef authored
-
Marc Vef authored
-
- Jun 12, 2023
-
-
Marc Vef authored
- Mar 24, 2023
-
-
Marc Vef authored
Improve cmake This MR improves the current CMake scripts in the following manner: - Dependency management is now handled more consistently: system dependencies are found using `find_package()`, whereas source-only dependencies are found using `include_from_source()`. This new function integrates a dependency provided its source code is available. If it's not, it will try to download it from its git repository if it was defined. - All external dependencies are now searched for in the top-level `CMakeLists.txt`. - More consistent use of targets (we are closer to 100% modern CMake). - Adds the `gkfs_feature_summary()` to allow printing a summary of all GekkoFS configuration options and their values. - Moves all option definitions to `CMake/gfks-options.cmake`, so that the top-level `CMakeLists.txt` is cleaner. It also defines the `gkfs_define_option()` and `gkfs_define_variable()` functions to integrate the definition of options/variables with the automated reporting provided by `gkfs_feature_summary()`. It also fixes #249. Closes #211 Closes #211 and #249 See merge request !143
-
Alberto Miranda authored
-
Alberto Miranda authored
-
Alberto Miranda authored
The function include_from_source() now checks if the source directory for a external dependency is empty. If so, it will provide instructions for the user to solve the issue (such as properly initializing submodules).
-
- Mar 17, 2023
-
-
Alberto Miranda authored
-
Alberto Miranda authored
Until #250 is fixed
-
Alberto Miranda authored
-
Alberto Miranda authored
-
Alberto Miranda authored
-
Alberto Miranda authored
-
Alberto Miranda authored
-
Alberto Miranda authored
This commit enforces replaces the use of absolute paths for several targets in favor of using target_sources(), which allows specifying source files for the same target in CMakeLists.txt files.
-
Alberto Miranda authored
-
-
-
Alberto Miranda authored
-
Alberto Miranda authored
-
Alberto Miranda authored
-
Alberto Miranda authored
-
Alberto Miranda authored
Also: - Remove FindRocksDB.cmake, FindLZ4.cmake, FindJeMalloc.cmake. Rationale: RocksDB already installs a RocksDBConfig.cmake that creates all of these targets when calling find_package(RocksDB).
-
Alberto Miranda authored
-
Alberto Miranda authored
-
Alberto Miranda authored
Remove obsolete submodules for CLI11.hpp and fmt.
-
- Mar 16, 2023
-
-
Alberto Miranda authored
Resolve "CI: Simplify coverage scripts" This MR simplifies coverage generation in the following ways: - Replaces the old `scripts/ci/coverage.sh` script with a new `scripts/dev/coverage.py` written in Python, making it more robust and simpler to modify if needed. - Adds specific CMake targets for generating coverage reports directly from CMake. - Adds specialized CMake presets to make it simpler to configure the different builds required to generate coverage information. - Replaces `gcovr` with `lcov` + `lcov_cobertura`, since `gcovr` exhibited some errors that were difficult to track down and `lcov` worked out of the box. - Adds HTML documentation about the coverage generation in `docs/sphinx/devs/coverage.md` This MR also updates the `v0.9.2` Docker images to update CMake and include coverage-related packages. The `coverage` image in particular is no longer needed and has been removed. Closes #252 #256 #257 #258 Closes #258, #257, #256, and #252 See merge request !163
-
Alberto Miranda authored
- Rename job `coverage:zerocount` to `coverage:baseline` - Move `coverage:baseline` job to `report` stage
-
Alberto Miranda authored
-