Changelog

Upcoming

Added

  • #231 - stateful is propagated from Node classes back to NodeSpecification s. stateful can be specified explicitly in the spec, or computed automatically by the node class. So we need to make sure to propagate that with the spec for e.g. the ZMQRunner

  • #231 - iter_epoch() and iter_ready() methods to iterate over a single epoch’s nodes, or any nodes, respectively, as they are ready. This will largely replace the previous pattern of repeatedly calling get_ready within a while scheduler.is_active() loop.

  • #231 - A special ('meta', 'previous_epoch') signal is added to TopoSorters to model statefulness and inter-epoch dependencies within the graph directly. Nodes that are stateful will depend on this signal, which is marked as done when the previous epoch completes. This ensures that nodes that must be run with ordered inputs are in all contexts, and dramatically simplifies scheduling while also making concurrent runners naturally to handle multiple epochs simultaneously with stateless tubes.

  • #231 - A Scheduler.epoch property was added to get the current epoch.

  • #231 - TopoSorter.get_state() convenience method to get a summary of a topo sorter’s state (for debugging purposes). This copies the internal sets to protect them from mutation, and so shouldn’t be used in any perf-sensitive paths.

  • #233 - Preliminary representation of assets as nodes in JS viewer

  • #234 - Inputs can be optional and have a default. These defaults have to be basic yaml types at the moment, more elaborate types are still TODO.

  • #234 - Nodes can use a tube-scoped input to control their enabledness

  • #234 - The InputCollection is now passed in the dependency-injected RunnerContext

  • #237 - Nodes now have an logger property that lazily instantiates a logger, if accessed.

Changed

  • #231 - Moved scheduling logic out of runners and into the scheduler! Runners used to have to internally keep their own counters of the current/active epochs, which was a huge mess and a bad division of labor. Now the relationship between a runner and the scheduler is much less janky.

  • #231 - The ZMQRunner was simplified and its terrible state management was streamlined. There are still some flaky test failures from threading deadlocks, but we are getting there.

  • #231 - the TubeRunner’s _get_ready method was made an iterator, yielding from the scheduler.

  • #231 - The TopoSorter no longer returns NodeSignals as “ready”. Instead it marks them as being “out” along with the node, returning only the node as ready. Previously, they could be returned with a warning in the event of a bug or an inconsistent scheduling state that marked a node as done without marking its signals as done. Signals can’t be run, it makes no sense for them to be ready. They are always out at the same time that a node is, even if they can have different ending conditions (i.e. being marked “done” by emitting an event or “expired” by emitting NoEvent)

  • #231 - The Scheduler’s node completion methods all return a list of MetaEvents rather than one or None, allowing multiple epochs to be marked as completed at once.

  • #231 - An epoch is considered completed by Scheduler.epoch_completed() only when it is fully complete, rather than when no more progress can be made, but a completion event has not been emitted. This clarifies the distinction and use between it and Scheduler.is_active(), which returns False when an epoch should be marked as completed because no more progress can be made, while epoch_completed checks if that completion event has been emitted.

  • #234 - The type of an input can be an arbitrary python type expression instead of an absolute identifier. These are still unused, but they will be used for static type checking in the future.

  • #235 - Assets (and other specifications) now include their id when being dumped to json

Fixed

  • #231 - The ZMQRunner now more correctly locks the state around its scheduler, avoiding race conditions between the command node thread and the main thread.

  • #231 - Events in the parent epoch that are from a node that is not the one that induced subepochs can mark their counterparts in subepochs as expired. This is how it was supposed to work, and how gathering works - To decrease cardinality, a node emits an event in the parent.

  • #231 - Empty sets are no longer created in the Scheduler’s _subepochs dict for every epoch, even when there are no subepochs.

  • #231 - Dynamically enabling/disabling a node with the scheduler now works in epochs that have already been scheduled, however this means that nodes should only be enabled/disabled in between running epochs, or else nodes may be re-ran in those epochs.

  • #231 - Even stateless tubes run in a “best effort” epoch order - when a bunch of epochs are queued up out of order, the scheduler ensures that ready nodes are yielded in an epoch-sorted order.

  • #231 - Fix!!!! a longstanding threadlocking problem in the ZMQRunner. Twofold: the socket iteration method never actually yielded the eventloop when it was flooeded with messages from an upstream node that is much faster than it. This meant noderunners would sometimes just never process any events because they were just receiving messages. Similarly, the command node would get flooded, and it would never actually call the coro used to quit it. So: force a context switch using sleep(0) in the socket receive iterator, and use a TaskGroup to more reliably force the command node to quit the thread, freeing the socket.

  • #233 - Correctly handle nexted prefixes in JS viewer

  • #234 - node.Tube nodes now correctly accept params in their specs and forward them and other inputs to the Tube they wrap.

  • #234 - The ZMQRunner ‘s zmq.node.NodeRunner class now correctly injects a requested RunnerContext

  • #236 - context: recursive when loading a tube specification actually recurses more than two levels

  • #236 - Validation errors for tube specifications are logged rather than suppressed so it’s possible to know what is wrong with the spec while working on it

  • 238 - No more validation errors on type annotations on node process functions from the Signal class.

Removed

  • #231 - zmq.NodeRunner ‘s await_inputs and await_node methods were removed in favor of the scheduler iterators.

Perf

  • #224 - Avoid calls to Epoch.__eq__ when unnecessary. 6-7% performance improvement on whole runner benchmark

  • #226 - Cache Node.edges. 2.5% performance improvement on whole runner benchmark.

  • #230 (@vaishnavidesai09) - Use a sets for O(1) lookups in the epoch log rather than O(n) lookups in deque. ~6% scheduler performance improvement.

  • #231 - Cached Node.signals and Node.slots as they are frequently accessed props

  • #231 - More of the TopoSorter’s methods were adapted to be batched set operations rather than iterators.

  • #231 - Epoch ‘s __eq__ method was removed to avoid using it during lookups, using hash instead and normal tuple equality. Being able to == an integer wasn’t worth the performance cost.

  • #231 - Epoch’s creation was optimized to avoid remaking EpochSegments.

  • #231 - Epoch gained __slots__

  • #232 - Use a snowflake-like identifier for event IDs rather than UUIDs, add helpers for consistently making events.

CI

  • #231 - pytest hooks were added to set the debug logging level when re-running an action in debug mode.

v1000.*

v1000.1.0 - 26-05-18

Added

  • #195, #196 - extends keyword - allow a tube to extend other tubes! reuse tubes, add on new nodes, override them, and so on. build bigger tubes out of smaller tube fragments.

  • #196 - NoEventable[] convenience generic that indicates that a return type can also be NoEvent.

  • #207, #209, #214, #222 - Big improvements to the display of tubes. A noob view cli command to show a live-updating display of a tube as it is edited on disk. Better representations of tube specs: nested tubes, correct signals and slots from inspecting nodes, better use of the ELK layout engine.

Changed

  • #211 - Allow accessing a node’s signals and slots from a classmethod, avoids needing to instantiate a node in order to inspect its edge properties

  • #212 - Make the signals and slots accessors have the same type: dicts rather than dicts and lists

  • #213 - An additional NodeInfo dictionary contains metadata about signals and slots derived from the combination of the node specification and the node class.

v1000.0.1 - 26-03-15

Fix

  • #192, #201, #202 - Support event values that can’t use the == operator by using is to check for NoEvents

v1000.0.0 - 26-03-13

First “official” beta release with all target features working :).

ok NOW the changelog officials starts since we’re now releasing versions regularly.

v0.1.*

v0.1.0 - 25-12-09

  • Start actually publishing versions.

  • Begin changelog

Recent changes

  • #54 - ZMQ Runner

  • #72 - Make NoEvent a MetaSignal enum

  • #51 - Recursive Tubes

v0.0.*

v0.0.9999999

   . @  ..8. t 8;:;8;:  :8    :8 X.   8t8S:.. 8 8    t8  .   88.8% @8;    8. S8 ..   @8tX@@8@X:  S8 
   .:..@:. t8  S :.8X;S;:8%  . 8;:;    t  8.  .:8  . ;%  %.:X88. . @@S. .t..X8.:. . %@8@X 8@; .;@8S 
 ...    88  t%  %::8:X.%. ..    t.X.   %:S%  . @@    %. .88 t8%%XXX.8%  XS X8..    ;.% tt  %  8;8@@.
...  .  .; ::.@: @::8 t.8  X    .; ;;tS8tS      @ .  8   X.t8 XXSSS%: .:S @8..  . .X:8XS%%  :@:888X.
S:         tS  X@ ::tXtt ;.:@ . .@t%t...: @    @     % . X .;t : tt%   X.88.  .   %; 8%:8  @8    . X
 8t.  .     S: .S   8%@ . %;8t    :.t ; t;.%   :     .  %t  88:  .::.   8X     . tXt;t8; :8::. . 88 
.   @   .   ..%;St   t;8   @S .   . t; 8tStS . @   @  ; X8 .tS.  S@  8;8@.  .  .8@X:8S  @@...: 88 ;8
 ...;X;    .   88@S88.t ;tX@8X%.   ;@t  t% 8.     8;:.;@%. 8%X . @ .;@8X  .  .S%t X%t :8:....88.S8tt
88::::X: .;:.@@ :@:.@;  :;8% @8@    Xt: .  ;    @  :;88 8  XS   %    8X .   .8:%@8S  8@.. :S@ X8%SSS
. @St;.t S:   ..  8SS  :  :: .% t   .8%      .  .  @X:  t  8S .:% .t8X . %. ;:8:8S :8:. t;%88Xt8t.::
S88 :%8;t8;; .      ;8X @  %;  tS..  .8.  .     @.    .   .t   ;. @8X   .S88@X8%. 8X..%ttX8S:8.X S88
@;:; ..%. .:8   .  ::8;SS;. @@  :; .  8@    . . t  .    .   ...  %;8 .   X888; ..8;t%.:.%t:8tX.@8% .
8t%. :8X 8X;:@S      .:% t@X 88%X88  . @...      .   .    ..... .X8   .   ...S.8XXt.:t :;88;:SX  t;S
.8  @:8t;S :t8;8;.  ... :@::X 8888:.  .....  .  .   .  .  ...    X  .   .  @.%%. ::88..X@.;tt .SX8%%
 ...tt%@.;8@;.S:S;...     8t8S:..X8;  .....       .     ...   .%S .    .   .8S@S @t;.tt .:. X8t%%:  
       .8@S. . .@8 8 .   . .S@8;    .       . .      . ...     .     .   .:S @.t;X.    . :.  ;t  :tt
. .:... .  :      .: : .     t%:. .    .  .     . .        . .   .  .    8t8 :88.   . .88 .:  t.. X 
%.XStX:   .  .       X.   .          .       .      . .  .     .      .:X8. ;%     .; @;;  X:.%:   .
8. .@ :@.     .    . .  .   . .  .  .   .  .   .  .     .   .    . .  .S .      . .S8. .8%t8   .@ :;
 ;S% 8..8%.%.   .    t8           .   .   .     .    .    .   .      .     . .      .88X%   .S8t@   
      ;X% X:X@t;  .      . .  . .       .   . .    .   .        .  .   .  .    . .:tS       . .  . S
           :@% tS@X..  8;          .  .          .   .   .  . .      .       .   :X: .  . .   X:8X  
 . . ..        ::8tS%     . .  .  .      . ..  .   .       .     .     . .  .  ..%Xt     .%@@;SX.   
X%%@t. .   .    .  ....       .     .  . :..  .       . . . .  .   .          .....;  .   :.   ..   
 X@X SX@;S %X.   . ......  .    .    .  ..  .   . .     ......   .   .  . .  ...  .  .St;8:   ..... 
...t@888tt 8@.@. .88:...     .    .   ... .         . .  .....     .   .   ...  . 8 t8X. .  . ..... 
.   .    ;8 . .S;  @t8   . .   .   ....      . .  .        .   . .   .    ... . XS8@;        ;X8@St:
8% S8t.         .  ;S:       .   .     .  .         .  .      .         . .    .X .     . ..t%X8S:.t
:%@8@ SS@ Xt:.%8%.  ;  . .          .   .   .  . .       .  .    . .  .      .       .     :tSt%X8S;
88@S%;;: %.  ..tX8XS8      .  . . .   .      .(kablam)     . .  .        .  .   . .  .:.  ..;%XX%;: 
    . . .   .    .8@ . .  .         .    . .    .         . .     .    .      .    . XX.       .    
 .        .   .   S%        . .  .     .      .   .  . .      . .   .     .     .  %8 ;tX88X%;..;S@8
8888@@X%%SS@XS%tt %tS  . .         .  .   .     .        .  .         .  .  .     .  :tSS:     . . .
     .8@t S8t:X8.    .  ..  .  . .      .   . .    .  .    .  .. . .    .    . .      . ;8@S;:.;SX%.
t S8t  ..  .          :..    .      . . . .      .   .  .  .:..      .    .  . . .tX888;.    .  .%8%
..;@8S@88S%: ..8  . ...  . .    .  .  .....  .     .      ...   .  .   .   ......  . .@.St%XX.@%..  
t.t8t        .%S t ... .      .       .....    . .    . ...   .      .   .  .... .      .@8S XX8 :; 
 ... .  . .   8S.t.. .8@St:  .   .  .   .   .       .  ...  .    .     .   . .     .;@@%S   .S%;SX t
%@S8@t.     .   tS.;.tS.:;. .   .     .   .   .  .     .       .   . .    .;SX .      .;;8X8tS   :S@
;S   ;S:. .8.88SS:8@;  .:@8  ..   . .       .   .  . .    .  .   .      .    ;8St. .       8X;8.8;% 
8..;X8Xt   S.:%S@ .      ; :@t.        . .    .         .   .      .  .       ..S..@St .  . .:;. 8:8
t:@ 8@S 8t. S..  :@t. ..X8X8   . .  .      .     .  .  .   .   .        . . .     8XS .:@  .      8@
X @SS;@X.S    .St8.   .8;@.;..       .  .    .    .      .    .  . . .          .   %t8..88@.       
t;8X8@.   . @X;S.. t@ S@;X8SS.;S. .      .     .     .      .          .  t.  .  %8.  .@8X@t S;;    
 ;t   . .88 ... :SS8 8 S%X X 8S    ; . .   ..   .  .   . .     .  .  .   ..%X   . 8%     ;S8.8@S.;S 
  ...:SSt 8  .%:.8X.8% tSt .t;   . 8     :;S.     .   .   . .   .      .    tS@:8t S: .%; .X: X888@S
  .t;:@@S .8@@:S @%88:.SS S S  .      . .SX. . .    .   ......   . . ;.%  :: @.8. X@%.  %88  %.S8:;S
:t ;;.. X@ .;  8 :%@:@S. :@8  8: .    ...;%      .       ..... . :   t; .  %@:; 8  .  .   %t::.: X 8
8X@@.t;: 8.X 8..:.8t8: ; @@  :;.@  ....     .  .   .  .   t.@ %  . ;. .   ..t@  %:S%        ;.%8..% 
.;:. @%@%8 8 ..S ;%8  88tX    .t.S   .  .    .    .    . .  X@:8   .8.   . .. t  %S; t. . .  t;.@ : 
% @ X@Xt 8  ;XXtS :  %::% . .   .:     .  : .X  . X .      . .S@ .  %@        ;@  %%:;X:  .t; %8tt 8
8%%@ : 8  X%t@:XS%.S.   :     . 8:. .    8.   .;  %   .  .   . .   . %8 . .  .S8X..: :.8..  X@% 8tX@
..t: 8  8: ;;SS@St;X 8 %X t.   88   8    . .  :X .t .        X:  .   .@;       @X.t  ;:..tX   @:X Xt
8: 8  8 . .X%X;:S. .% t: %@ . S@ . 8;  .%8    S. :8 ; t ; . ....  ..  S8.  .    @:SS  @X  . t  ..88.
 8 .X ;::. @t@%XX :  SS.tt   ; ;  . .   S: .     88 8 8 8  X; 8.   S8   %    .   t.88  SX ..t%:  ;S 
  SX::::8%:8t.8::t% XX8:t.  . %   S  . .8S   ;   .@ X:: S. S  .. . 8X8 . @ .   .  ;:tS. :S :;:8;   :
t8@;;.;@8 tX ;::X@ @8:.S% . 8    8%.   X8  . %8 .8S X@;t.. ..:S8.tS ; X %@S  .     8::;   ::....;@..
@XXXX.88 t;;:..@8 8X .S8    @;  .%;  ..88..  @.  .t XS;@%8...  8: ..%8S:  tt.....    . 8.  8@ . .@@%
%X8t:tX.8%..  S8 @t8%:%  .X:;   S;:   tX;.. SS  .@. @.: SX..   8X   .88  ..   ...   .;   ;  %; SS..t