Kinesin/daggy
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Commit Graph

  • dded91220f Updating todo Ian Roddis 2021-09-23 09:32:07 -03:00
  • d19648fa84 Refactoring CMake configs, adding Catch2 benchmark support Ian Roddis 2021-09-22 16:33:50 -03:00
  • 510aa236c4 Converting underlying DAG implementation to use vectors for storage Reduces hashing required and speeds up traversals Ian Roddis 2021-09-22 14:19:20 -03:00
  • 33942e2c63 Update TODO with current goals. Ian Roddis 2021-09-22 10:43:03 -03:00
  • e7b15b3847 Implement a bunch of clang-tidy suggested changes. Remove FilesystemLogger Ian Roddis 2021-09-22 10:30:27 -03:00
  • 288ce28d29 Adding clang-format, and reformating all sourcecode Ian Roddis 2021-09-21 09:41:11 -03:00
  • 39d5ae08be Adding a No-op task executor for testing Fixing DFS implementation of DAG validation to be much faster Adding in additional tests to ensure the run order of expanded tasks is preserved Adding additional compile-time checks, resolving issues that came up as a result Ian Roddis 2021-09-20 19:05:56 -03:00
  • 2daaa83d82 Small changes for quality of life Ian Roddis 2021-09-17 16:36:59 -03:00
  • f83a17d610 Enhancing the performance of dag building by delaying validation until the end Ian Roddis 2021-09-16 17:13:40 -03:00
  • fac741742a Adding check for non-existent dagid Ian Roddis 2021-09-16 11:03:47 -03:00
  • e895103b1c Increasing max send/receive sizes for REST server to 4G Ian Roddis 2021-09-16 10:07:06 -03:00
  • 0c1a634ffa Adding in and updating previously commented-out tests Ian Roddis 2021-09-15 14:18:44 -03:00
  • d4ec744773 Fixing race condition issue with references in forking executor Ian Roddis 2021-09-15 14:12:55 -03:00
  • a6a7501d12 Changing how execution parallelism is handled, so that different executors can implement their own idea of parallelism. Ian Roddis 2021-09-15 13:05:04 -03:00
  • 4562ac755e Fixing argparse issue and populating stdin file Ian Roddis 2021-09-14 15:04:23 -03:00
  • 96c1928daf Adding fixes for slurm linking and uncommon slurm node attributes Ian Roddis 2021-09-11 00:00:21 -03:00
  • ff5a75abb0 Removing gitlab CI for now Ian Roddis 2021-09-10 13:36:14 -03:00
  • abdbf9f2bf Adding missing CMakeLists.txt Ian Roddis 2021-09-10 13:34:56 -03:00
  • d519b11247 Fixing slurm detection in cmake Ian Roddis 2021-09-10 13:33:53 -03:00
  • ea53735069 Adjusting path after argparse changed its structure out from underneath me Ian Roddis 2021-09-10 12:16:25 -03:00
  • f8cf2d48a5 Adding find_library for slurm Ian Roddis 2021-09-10 11:10:20 -03:00
  • 2f0548cb0a Disabling gitlab CI for now Ian Roddis 2021-09-10 10:55:22 -03:00
  • d731f9f5b1 Adding support for execution on slurm grids - Adding support for SlurmTaskExecutor in daggyd if DAGGY_ENABLE_SLURM is defined. - Renaming some test cases - Enabling compile-time slurm support - Adding slurm documentation Ian Roddis 2021-09-10 10:53:58 -03:00
  • d15580f47f Massive re-org to allow per-executor job specification formats and executor-specific task validation and expansion. A few different renames to try and keep things more consistent. Ian Roddis 2021-09-03 09:10:38 -03:00
  • e746f8c163 - Adding daemonization for daggyd - Changing how parameter keys are stored in the map to allow for future key handling. Ian Roddis 2021-09-01 13:32:16 -03:00
  • 4e71bf5917 Adding daggyc rough-in Ian Roddis 2021-08-31 19:37:23 -03:00
  • 77f7819584 Fixing test where return by reference would cause unexpected results when compiled with release flags Ian Roddis 2021-08-31 19:14:34 -03:00
  • 441cdb5b2e Renaming rest_server to daggyd, and daggy to libdaggy Ian Roddis 2021-08-31 17:04:03 -03:00
  • 7b07380e16 - Removing duplicate information (taskName stored in 3 places) Ian Roddis 2021-08-31 12:21:34 -03:00
  • 2c00001e0b Adding support for isGenerator tasks - Changing how DAG is represented, both in code and how DAGs are defined in JSON. - Removing std::vector<Task> representation in favour of a map that will enforce unique task names - Task names now have a name (generated), and a definedName. - Adding support to loggers to add tasks after a DAGRun has been initialized. Ian Roddis 2021-08-30 22:05:37 -03:00
  • dd6159dda8 Adding simple defaults for rest_server for testing Ian Roddis 2021-08-30 12:47:06 -03:00
  • 307e0c0559 Converting to better_enums to support gcc 8 Ian Roddis 2021-08-27 12:06:36 -03:00
  • 7fa47c1c4a More CI stupidity Ian Roddis 2021-08-25 16:16:58 -03:00
  • fcdc225e28 Adjusting test case to eventually exit with an error after a reasonable time. Ian Roddis 2021-08-25 15:20:08 -03:00
  • 212bd80df2 Squashed commit of the following: Ian Roddis 2021-08-25 13:40:05 -03:00
  • 6ed57806d0 - More fixes for CI failing tests Ian Roddis 2021-08-24 07:58:30 -03:00
  • 39fe716968 CI: Adding missing dependencies Ian Roddis 2021-08-23 21:05:27 +00:00
  • 307cff73f8 - More fixes for CI failing tests Ian Roddis 2021-08-23 17:58:43 -03:00
  • 7016cb37ba - More fixes for CI failing tests Ian Roddis 2021-08-23 17:53:47 -03:00
  • cf37f9a6fc - More fixes for CI failing tests Ian Roddis 2021-08-23 17:49:04 -03:00
  • 1bdc5dafd5 - More fixes for CI failing tests Ian Roddis 2021-08-23 17:42:43 -03:00
  • d4c9b3fcee - Adjusting tests so they run without referencing /tmp Ian Roddis 2021-08-23 17:37:33 -03:00
  • 8eb966377f CI: Adding some debugs and assuming test script picks up where build left off. Ian Roddis 2021-08-23 20:26:34 +00:00
  • 2460b24b04 CI: Adding some debugs and assuming test script picks up where build left off. Ian Roddis 2021-08-23 20:21:14 +00:00
  • 461900b18d CI: Adding some debugs and assuming test script picks up where build left off. Ian Roddis 2021-08-23 20:16:34 +00:00
  • 22accec2a9 Adding missing packages Ian Roddis 2021-08-23 20:11:06 +00:00
  • fb3c450bfc Setting image to gcc for CI Pipeline Ian Roddis 2021-08-23 20:09:09 +00:00
  • 574f6c1fb1 Update .gitlab-ci.yml file Ian Roddis 2021-08-23 20:07:00 +00:00
  • be20bb5665 Adding MIT License Ian Roddis 2021-08-23 16:51:27 -03:00
  • c12309c3f6 Adding MIT License Ian Roddis 2021-08-23 16:51:20 -03:00
  • 5b8d5fabe8 - Fixing errors in JSON Ian Roddis 2021-08-23 14:18:16 -03:00
  • 808a7f9af4 - Adding more documentation - Renaming "parameters" to "taskParameters" for clarity, and to distinguish from "executionParameters" that will be implemented down the road. Ian Roddis 2021-08-23 14:17:22 -03:00
  • 6b9baffe27 Changing Executor interface to allow for more flexible tasks down the road Ian Roddis 2021-08-23 12:03:16 -03:00
  • a83da567d0 - Updating TODOs - Adding jwt-cpp dependency (not really related to dag_execution) Ian Roddis 2021-08-21 12:18:01 -03:00
  • 2083b1c3f1 Refactoring RunState, fixing logic error in when to end errored DAG runs, added convenience functions to retry failed DAGs. Ian Roddis 2021-08-20 21:44:12 -03:00
  • dc8ea4c369 - Adding support for state to OStreamLogger to make it more useful for test cases - Making runDAG return the end DAG - Adding much more robust test for DAG execution for basic tests. Ian Roddis 2021-08-20 12:43:01 -03:00
  • 0f1f00362c - rest_server is now a complete binary, just needs more config options boilerplate. Ian Roddis 2021-08-20 11:18:26 -03:00
  • 1f2712b090 - Fixing issue with parameter expansion on DAG submission to server - Adding sections to unit_server tests - Adding cleanup Ian Roddis 2021-08-20 11:11:12 -03:00
  • 8fa9af95af Merge remote-tracking branch 'origin/rest_driver' into rest_driver Ian Roddis 2021-08-20 10:59:52 -03:00
  • 791e7a15a7 - Fixing request logic to avoid hangs when REQUIREs fail before the client has been shutdown. Ian Roddis 2021-08-19 14:46:57 -03:00
  • db47bc1593 - Adding unit tests for Server Ian Roddis 2021-08-19 14:23:40 -03:00
  • 0a2a66bc59 - Adding dagRun REST method Ian Roddis 2021-08-14 11:11:12 -03:00
  • ce2c0dd30f - Checkpointing work so I can switch computers. Ian Roddis 2021-08-14 08:36:55 -03:00
  • c1489ce967 - Checkpointing work so I can switch computers. Ian Roddis 2021-08-14 08:36:49 -03:00
  • 3fde95339e - Checkpointing work before I restart the IDE. Ian Roddis 2021-08-13 13:28:47 -03:00
  • 35857f9513 - More cmake cleanup Ian Roddis 2021-08-13 13:21:55 -03:00
  • 71344b3a2a - Sidequest: Clean up header-only library dependencies with cmake. Ian Roddis 2021-08-13 12:43:39 -03:00
  • de8025e0f9 - Fixing bug that yielded empty expanded command sets. Ian Roddis 2021-08-20 10:58:10 -03:00
  • 1668117566 - Checkpointing work on expanding commands. Ian Roddis 2021-08-20 10:39:39 -03:00
  • 6bf376984b - Fixing request logic to avoid hangs when REQUIREs fail before the client has been shutdown. Ian Roddis 2021-08-19 14:46:57 -03:00
  • 90f34c21b9 - Adding unit tests for Server Ian Roddis 2021-08-19 14:23:40 -03:00
  • 4c6bc2a540 - Adding dagRun REST method Ian Roddis 2021-08-14 11:11:12 -03:00
  • 2525731f5a - Checkpointing work so I can switch computers. Ian Roddis 2021-08-14 08:36:55 -03:00
  • a668bf81f1 - Checkpointing work so I can switch computers. Ian Roddis 2021-08-14 08:36:49 -03:00
  • eb8e530f9a - Checkpointing work before I restart the IDE. Ian Roddis 2021-08-13 13:28:47 -03:00
  • d731f8b6dc - More cmake cleanup Ian Roddis 2021-08-13 13:21:55 -03:00
  • dfb71b63b8 - Sidequest: Clean up header-only library dependencies with cmake. Ian Roddis 2021-08-13 12:43:39 -03:00
  • 9f90f54b67 - More work on DAGLoggers - Still need unit tests for the FilesystemLogger Ian Roddis 2021-08-13 10:23:55 -03:00
  • 4d519cc596 - Adding default pretty-printer for Task - Adding equivalency check for Task - Adding test to ensure serialization / deserialization is reproducible. Ian Roddis 2021-08-11 10:16:38 -03:00
  • a152588368 - Making task children unordered_set to avoid multiple edges between parent and child. Ian Roddis 2021-08-10 12:42:54 -03:00
  • 621467dd5a - Adding task serialization - Renaming DAGLogger to DAGRunLogger - Adding more functionality to FileSystemLogger Ian Roddis 2021-08-10 11:44:56 -03:00
  • 1af9ba2124 Adding more TODOs Ian Roddis 2021-08-09 16:32:14 -03:00
  • a97c3ff783 - Removing Catch2 code from codebase, will pull it via FetchContent instead. - Changing StdOutLogger to OStreamLogger, so that test cases output can be silenced. Ian Roddis 2021-08-09 15:48:53 -03:00
  • 9a3671aba4 - More refactoring - Fixing cmake source discovery with GLOB_RECURSE Ian Roddis 2021-08-09 15:24:28 -03:00
  • 30aea0818c - Running project through clang-tidy formatter. Ian Roddis 2021-08-09 15:07:16 -03:00
  • 1fcdf66829 - Adjusting file layouts to match between include and src - Updating TODO items. Ian Roddis 2021-08-09 15:05:25 -03:00
  • 28c5b3eea3 - Adding StdOutLogger and adding tests for DAG execution to verify it works. - Roughing in FileSystemLogger - Deleting Scheduler code and associated unit tests as being too complicated for maintenance. - Refactoring namespaces for loggers and executors. Ian Roddis 2021-08-09 14:59:54 -03:00
  • a8e85f8feb - Adding StdOutLogger and adding tests for DAG execution to verify it works. - Roughing in FileSystemLogger - Deleting Scheduler code and associated unit tests as being too complicated for maintenance. - Refactoring namespaces for loggers and executors. Ian Roddis 2021-08-09 14:59:23 -03:00
  • 6a2c2ddb9a Merge remote-tracking branch 'origin/daglogger' into daglogger Ian Roddis 2021-08-09 11:15:28 -03:00
  • c3b88d24ac - Rebasing on master. - Fixes so all tests pass. Ian Roddis 2021-08-09 11:15:04 -03:00
  • df63d944c1 Checkpointing work, while serialization / deserialization is figured out. Ian Roddis 2021-08-08 13:05:13 -03:00
  • 1a467428a2 - Roughing out the DAGLogger interface - Adding spec for FileSystem Logger Ian Roddis 2021-08-08 12:22:39 -03:00
  • 0516bde09f - Refactoring struct (de)serialization Ian Roddis 2021-08-09 10:33:49 -03:00
  • 7cd9fc5e6e - Refactoring Defines - Moving de/serialization routines to a separate header Ian Roddis 2021-08-08 13:13:32 -03:00
  • bea21d912f Checkpointing work, while serialization / deserialization is figured out. Ian Roddis 2021-08-08 13:05:13 -03:00
  • 5a3796a8a1 - Roughing out the DAGLogger interface - Adding spec for FileSystem Logger Ian Roddis 2021-08-08 12:22:39 -03:00
  • 1849a2fee4 - Adding TODO list to keep track of things. - Adding .idea to gitignore - Adding DAG execution to utilities, redefining the logger Ian Roddis 2021-08-05 15:24:17 -03:00
  • 745c950280 - Adding TODO list to keep track of things. - Adding .idea to gitignore Ian Roddis 2021-08-02 15:50:20 -03:00
  • afcb917cc9 Completing task building. Ian Roddis 2021-08-02 15:44:41 -03:00
  • a5b4e6ce42 Checkpointing progress on task builder. Ian Roddis 2021-08-02 11:59:43 -03:00