Self-referential neural nets in 2018

[anhinga_anhinga]

Two series of experiments with self-referential neural nets with vector flows ("dataflow matrix machines") were done by us in 2018.

The ability of a neural net to modify itself on the fly was used to edit it interactively while it is running ("livecoding"). This also opens the way to have populations of neural nets editing each other.

Emerging "sleep-wake" behavior and other emerging bistability patterns were observed in randomly initialized neural nets (May 2019 update: a couple of video recordings of those behaviors are posted: https://youtu.be/_mZVVU8x3bs and https://youtu.be/CKVwsQEMNjY ). There is no theoretical understanding of this emerging dynamics yet.

These experiments are described in Section 1 of https://github.com/jsa-aerial/DMM/tree/master/technical-report-2018 (DMM technical report 11-2018, "Dataflow matrix machines: recent experiments and notes for next steps").

Also https://arxiv.org/abs/1610.00831 got an extra page (page 7) with two new appendices, and the main paper, https://arxiv.org/abs/1712.07447, has a new more compact abstract:

1) Dataflow matrix machines (DMMs) generalize neural nets by replacing streams of numbers with linear streams (streams supporting linear combinations), allowing arbitrary input and output arities for activation functions, countable-sized networks with finite dynamically changeable active part capable of unbounded growth, and a very expressive self-referential mechanism.

2) DMMs are suitable for general-purpose programming, while retaining the key property of recurrent neural networks: programs are expressed via matrices of real numbers, and continuous changes to those matrices produce arbitrarily small variations in the associated programs.

3) Spaces of V-values (vector-like elements based on nested maps) are particularly useful, enabling DMMs with variadic activation functions and conveniently representing conventional data structures.

We also have a better slide deck: https://github.com/jsa-aerial/DMM/blob/master/doc/DMM-IBM-Talk-Oct2018.pdf
 
Crosspost: anhinga-anhinga.livejournal.com/83697.html

Other blogs by this author:

https://dmm.dreamwidth.org/  (partial mirror: https://anhinga-travel.livejournal.com/ )
https://anhinga-drafts.livejournal.com/  (mirror: https://anhinga-drafts.dreamwidth.org/ )

Comments

[anhinga_anhinga]

> What does "paper quality" mean?

In this particular case (arxiv.org criteria), it means "suitable to be submitted for peer review in an academic journal". (It does not mean that it should successfully pass such a review, but it does mean that it can be submitted for such a review in good faith.)

> What is the reason to sacrifice readability in a draft?

None, except that there was no time budget allocated for too many iterations or too much proofreading.

Some modest feedback was collected from several readers, and some editing iterations were done, so some effort in improving readability was made, but a decision had been made not to spend excessive time on this. Basically, limiting cost of doing this particular piece of work is the reason not to invest more in presentation quality here. (We invested enough in https://arxiv.org/abs/1712.07447 ; we don't have resources to invest that much in each piece of writing we do.)

> It is easy to program variable values change.

Certainly. But in this case, it is an emerging phenomenon (no one programmed it, it just happened), and such emerging phenomena are of interest to various researchers (e.g. to people who study dynamic systems).

We hope that further research would shed some light on the nature of this particular class of emerging bistabilities here (and this was posted here, in part, because there might be people among readers of this blog, who have sufficient qualification to study this).

In any case, researchers in various related fields might find this information useful. We are not trying to sell this particular aspect too hard: if it clicks, it clicks, and we would be happy to collaborate on this, and if it does not, it's not a problem. There are people for whom emerging bistabilities (including emerging sleep-wake patterns) are of interest without too much further encouragement. That part of the post was for the readers of that kind.

[dennisgorelik]

> too much proofreading

You do not need "too much proofreading" in order to identify excessive passive voice usage.
People normally speak in active voice, unless they try to hide something.

> it is an emerging phenomenon

There are plenty of "emerging phenomenon"s in our life. For example, vibration at high speed.
Such emerging behavior does not imply intelligence.

[anhinga_anhinga]

> excessive passive voice usage

well, I am happy to hear feedback, and I heard your feedback (which I appreciate), but at the end of the day the text writers' decide which feedback to incorporate and which not to incorporate...

and in this case I made a decision not to incorporate this particular advice... although I might consider it during future writings...

> unless they try to hide something

no need to be offensive, Denis... Scientific writing is often done in passive, even though this is not recommended by style guides... I have to tell you (from a number of my observations of your various conversations at ivan-gandhi, mostly), that people tend to have difficult time interacting with you; somehow you end up getting on their nerves (more often than not)... I am not sure you want this effect...

(but really, you are discussing 4 sentences in the blog post; if the subject matter is interesting enough, read (some of) the text(s) which are referred to and I'll be grateful for further feedback... I don't think further discussing how a brief blog post is phrased is fruitful or very relevant... (of course, such reading of actual texts only makes sense, if your preliminary impression is that it might be of some interest to you))

> Such emerging behavior does not imply intelligence

We have not claimed anything about intelligence of the configurations we played with; and certainly not in connection with the emerging bistabilities.

Like I said, these kind of phenomena are of interest to some communities of people (and I belong to some of those communities to some extent), and they were reported here for the possible benefit of readers from those communities. People who don't care about these things are encouraged to ignore this part.

It is a new class of dynamical systems, it is not studied as far as I can tell, there are some nontrivial and unexpected empirical observations about them we happened to make, I am reporting those observations, and that's that.

There is no reason to read anything more into it. (There might be more, and if there is, there is a chance one of the readers would see that and tell us, but I am not aware of that "more" at the moment.)

***

Now, the other series of experiments here is a different story... I do think there will eventually be "more" in connection with it... But not yet, this just shows the road there...

[dennisgorelik]

> > unless they try to hide something
> no need to be offensive

Why do you call my diagnostic of what is happening - "offensive"?
I describe what I observe and try to be accurate at this.

> Scientific writing is often done in passive

Good scientific writing is rarely done in passive voice.
Bad scientific writings are more widespread than good scientific writings.

[dennisgorelik]

> somehow you end up getting on their nerves (more often than not)... I am not sure you want this effect...

"Getting on peoples nerves" is not my goal. But I definitely want to avoid wasting time beating around the bush.

[anhinga_anhinga]

> wasting time beating around the bush

that's what seems to be happening here, so let's "call this a thread"...