Resolve "[Alya] Create a new data distributor - Moves instantiation outside read/write"
Closes #88 (closed) Closes #125 (closed)
The distributor will read a shared file including [filename] [host] [size] [offset] and distribute the files accordingly. We assume, as it should happen in SLURM that the nodes are in alphabetical order.
Distributors are instantiated per read/write operation. They should be moved to FsData to avoid the overhead. It is also necessary to reduce the creation cost when the distributor reads the data mappings from a file or similar.
Merge request reports
Activity
changed milestone to %v1.0.0
added 1 commit
- b70793fc - Updated README.md, included log output and removed sorting nodes, as it is not needed
assigned to @rnou
added New feature label
@amiranda we can start the review process
added 53 commits
-
0a4c52ee...1c4528b4 - 51 commits from branch
master
- 29fa4652 - New Distributor setup
- 8013a633 - Merge branch '88-alya-create-a-new-data-distributor' of...
-
0a4c52ee...1c4528b4 - 51 commits from branch
added 11 commits
-
8013a633...7c034057 - 8 commits from branch
master
- 6fccbb9d - New Distributor setup
- 5eadae29 - New Distributor setup
- 79674937 - Merge branch '88-alya-create-a-new-data-distributor' of...
Toggle commit list-
8013a633...7c034057 - 8 commits from branch
added 11 commits
-
96c7af5a...d5cb12e1 - 9 commits from branch
master
- 6aec2ddf - New Distributor setup. Update README.md
- 909104c1 - Merge branch '88-alya-create-a-new-data-distributor' of...
-
96c7af5a...d5cb12e1 - 9 commits from branch
added 31 commits
-
7ac88aa6...12ac0bc9 - 29 commits from branch
master
- beecfe14 - New Distributor setup. Update README.md
- 61efe501 - Merge branch '88-alya-create-a-new-data-distributor' of...
-
7ac88aa6...12ac0bc9 - 29 commits from branch
added 90 commits
-
61efe501...a2e97663 - 88 commits from branch
master
- 7af24a5e - merge changes from master
- 59d73078 - merge branch
-
61efe501...a2e97663 - 88 commits from branch
I finished the new implementation, however performance depends from the number of entries in the file (which is imported to a map file).
A regular expression approach will be better, but then we cannot do it (easily) automatically.
I tested it with IO500, which generates a massive number of files.
added 10 commits
-
8cc3adf6...4e873c61 - 8 commits from branch
master
- 95a6804c - master rebase
- 249102d1 - merge
-
8cc3adf6...4e873c61 - 8 commits from branch
added 1 commit
- 191132bc - Implemented Guided Distributor (rebase master from GekkoFWD), Added Hostname in client logs
added 1 commit
- e31c26cd - Implemented Guided Distributor (rebase master from GekkoFWD), Added Hostname in client logs
added 1 commit
- 9479232e - Implemented Guided Distributor (rebase master from GekkoFWD), Added Hostname in client logs
added 14 commits
- 96c7af5a - New Distributor setup. Update README.md
- 6aec2ddf - New Distributor setup. Update README.md
- 909104c1 - Merge branch '88-alya-create-a-new-data-distributor' of...
- f2e061d7 - New Distributor setup. Update README.md
- 7ac88aa6 - Merge branch '88-alya-create-a-new-data-distributor' of...
- beecfe14 - New Distributor setup. Update README.md
- 61efe501 - Merge branch '88-alya-create-a-new-data-distributor' of...
- 7af24a5e - merge changes from master
- 59d73078 - merge branch
- 972e037a - endif missing
- fe83714a - restored distributor
- 8cc3adf6 - Added hostname in the TRACE_GUIDED logs
- 763892a0 - Merge remote-tracking branch 'origin/88-alya-create-a-new-data-distributor'...
- 14a24d65 - Preload Guided Distributor endif mismatch
Toggle commit listI tried to use an interval tree to store the chunks, and was too slow (1% original speed). So I decided to cut down the test for
ior
because we don't need thechunk_id
in theior_easy
(one file -> one server).The result is still a slowdown of a 50% (checking only an unordered map for a path). I will profile the code, but I suspect that the creator of the distributor is called multiple times per instance of the client.
added 1 commit
- e319a5ca - Moved distributor instantation to RPC_DATA and daemon.cpp on the server side.
added 5 commits
Toggle commit listadded 1 commit
- f308ccc3 - Moved distributor instantation to RPC_DATA and daemon.cpp on the server side....
added 9 commits
-
f308ccc3...b8385c7f - 8 commits from branch
master
- df7d54dd - Moved distributor instantation to RPC_DATA and daemon.cpp on the server side....
-
f308ccc3...b8385c7f - 8 commits from branch
added performance label
- Resolved by Ramon Nou
- Resolved by Ramon Nou
- Resolved by Ramon Nou
- Resolved by Ramon Nou
- Resolved by Ramon Nou
- Resolved by Ramon Nou
- Resolved by Ramon Nou
- Resolved by Ramon Nou
- Resolved by Ramon Nou
- Resolved by Ramon Nou
- Resolved by Ramon Nou
- Resolved by Ramon Nou