anhinga_anhinga (![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png)
anhinga_anhinga) wrote2016-12-27 01:29 am
anhinga_anhinga) wrote2016-12-27 01:29 am
Dataflow matrix machines as generalized recurrent neural networks
A year ago I posted about dataflow programming and linear models of computation:
http://anhinga-anhinga.livejournal.com/82757.html
It turns out that those dataflow matrix machines are a fairly powerful generalization of recurrent neural networks.
The main feature of dataflow matrix machines (DMMs) are vector neurons. While recurrent neural networks process streams of numbers, dataflow matrix machines process streams of representations of arbitrary vectors (linear streams).
Another important feature of DMMs is that neurons of arbitrary input and output arity are allowed, and a rich set of built-in transformations of linear streams is provided.
Recurrent neural networks are Turing-complete, but they are an esoteric programming language, and not a convenient general-purpose programming platform. DMMs provide a formalism friendly to handling sparse vectors, conditionals, and more, and there are indications that DMMs will grow to become a powerful general-purpose programming platform, in addition to being a convenient machine learning platform.
In this context, it is possible to represent large classes of programs by matrices of real numbers, which allows us to modify programs in continuous fashion and to synthesize programs by synthesizing matrices of real numbers.
Further details and preprints
Self-referential mechanism: Consider a linear stream of matrices describing the connectivity pattern and weights of a DMM. Select a dedicated neuron Self emitting such a stream on its output, and use the latest value of that stream as the current network matrix (matrix describing the connectivity pattern and weights of our DMM). A typical Self neuron would work as an accumulator taking additive updates from other neurons in the network. This mechanism enables reflection facilities and powerful dynamic self-modification facilities. In particular, the networks in question have facilities for dynamic expansion.
The recent DMM-related preprints by our group:
https://arxiv.org/abs/1603.09002
https://arxiv.org/abs/1605.05296
https://arxiv.org/abs/1606.09470
https://arxiv.org/abs/1610.00831
Modern recurrent neural networks with good machine learning properties such as LSTM and Gated Recurrent Unit networks are naturally understood in the DMM framework as networks having linear and bilinear neurons in addition to neurons with more traditional sigmoid activation functions.
Our new open source effort
The new open-source implementation of core DMM primitives in Clojure:
https://github.com/jsa-aerial/DMM
This open-source implementation features a new vector space of recurrent maps (space of "mixed rank tensors"), which allows us to represent a large variety of linear streams as streams of recurrent maps. The vector space of recurrent maps also makes it possible to express variadic neurons as neurons having just one argument.
Therefore a type of neuron is simply a function transforming recurrent maps, which is a great simplification compared to the formalism presented in the preprints above. See the design notes within this open-source implementation for further details.
http://anhinga-anhinga.livejournal.com/82757.html
It turns out that those dataflow matrix machines are a fairly powerful generalization of recurrent neural networks.
The main feature of dataflow matrix machines (DMMs) are vector neurons. While recurrent neural networks process streams of numbers, dataflow matrix machines process streams of representations of arbitrary vectors (linear streams).
Another important feature of DMMs is that neurons of arbitrary input and output arity are allowed, and a rich set of built-in transformations of linear streams is provided.
Recurrent neural networks are Turing-complete, but they are an esoteric programming language, and not a convenient general-purpose programming platform. DMMs provide a formalism friendly to handling sparse vectors, conditionals, and more, and there are indications that DMMs will grow to become a powerful general-purpose programming platform, in addition to being a convenient machine learning platform.
In this context, it is possible to represent large classes of programs by matrices of real numbers, which allows us to modify programs in continuous fashion and to synthesize programs by synthesizing matrices of real numbers.
Further details and preprints
Self-referential mechanism: Consider a linear stream of matrices describing the connectivity pattern and weights of a DMM. Select a dedicated neuron Self emitting such a stream on its output, and use the latest value of that stream as the current network matrix (matrix describing the connectivity pattern and weights of our DMM). A typical Self neuron would work as an accumulator taking additive updates from other neurons in the network. This mechanism enables reflection facilities and powerful dynamic self-modification facilities. In particular, the networks in question have facilities for dynamic expansion.
The recent DMM-related preprints by our group:
https://arxiv.org/abs/1603.09002
https://arxiv.org/abs/1605.05296
https://arxiv.org/abs/1606.09470
https://arxiv.org/abs/1610.00831
Modern recurrent neural networks with good machine learning properties such as LSTM and Gated Recurrent Unit networks are naturally understood in the DMM framework as networks having linear and bilinear neurons in addition to neurons with more traditional sigmoid activation functions.
Our new open source effort
The new open-source implementation of core DMM primitives in Clojure:
https://github.com/jsa-aerial/DMM
This open-source implementation features a new vector space of recurrent maps (space of "mixed rank tensors"), which allows us to represent a large variety of linear streams as streams of recurrent maps. The vector space of recurrent maps also makes it possible to express variadic neurons as neurons having just one argument.
Therefore a type of neuron is simply a function transforming recurrent maps, which is a great simplification compared to the formalism presented in the preprints above. See the design notes within this open-source implementation for further details.
no subject
https://groups.google.com/a/tensorflow.org/forum/?utm_medium=email&utm_source=footer#!msg/discuss/LXJIWkil_mc/IEWI1398EQAJ
period. currently they can't do gradient steps on sparse weights.
I yet have to learn what people do for ML on large graphs for example. I believe there must be a workaround.
Before you pointed me to sparse tensors in TF I was thinking of making a kd-tree out of a sparse tensors and apply to the tree an RNN adopted for recurrent processing trees (instead of usual recurrent processing sequences). This way all weights are dense and, arguably, the recurrent tree RNN may mimic the plain convolution-like updates or even have some advantage over it. On the other hand, in kd-tree spatially neighboring cells (leaves) can be quite far from their nearest common ancestor. That's that tree RNN should be able to learn spatial neighboring relations for some nodes separated by quite long traverses through the tree and it makes me skeptic about this approach.
Currently I have no idea when I will be able to play with such kd-tree RNNs but if I suddenly learn something inspiring for this technique I may want to give the idea a try as seriously as I can. Basically I want to make it possible to apply DL to point clouds (which are generically sparsed samples of does not matter what embedded into does not matter which space).
no subject