Anhinga anhinga

On May 28, 2020 OpenAI published the GPT-3 paper, "Language Models are Few-Shot Learners", arxiv.org/abs/2005.14165

This was the "AlexNet moment of the Transformer Revolution", and the qualitative jump was even more significant than the AlexNet 2012 jump.

One extremely strange and remarkable property of GPT-3 is that purely linguistic knowledge in this model is often sufficient to guess a piece of correct computer code from a natural language description of a problem (even though we don't think this model "truly understands programming").

There was already quite a boom in these novel models (invented as recently as 2017), after BERT and GPT-2, but now the field had just exploded: "efficient transformers", "vision transformers", "multimodal transformers", etc.

And tons of interesting work were done in hybrid models which combined transformers and other attention-based models with all kinds of other techniques. Hybrids of all kinds of methods with transformers and other attention-based models are probably the future. For example, the famous Alpha Fold 2 by DeepMind which "solved" protein folding in November was a hybrid model with an attention-based component at its center: deepmind.com/blog/article/alphafold-a-solution-to-a-50-year-old-grand-challenge-in-biology

All this means two things: 1) "True AI" can emerge any moment; we are seeing a flood of breakthroughs now, and one of them (or a short sequence of them) might result in a much more radical shift than anything we've seen so far. I don't know if it happens this year, but it's a possibility (from invention of convolutional neural nets in 1989 it took more than 20 years to AlexNet, from invention of Transformers in 2017 it took only 3 years to GPT-3, things can really start happening very fast).

2) If you are a practitioner in any field (especially if this field is machine learning of some kind), it makes sense to ponder hybrids between your favorite methods and "attention" (which is just a linear combination of high-dimensional vectors [sometimes with all coefficients being non-negative and summing up to 1]), hybrids between your favorite methods and matrix multiplication (which is just a way to compute a lot of linear combinations of high-dimensional vectors rapidly), hybrids between your favorite methods and Transformers (a certain way of arranging those matrix multiplications and interleaving them with modest neural connectors). This is likely to be a very fruitful thing, and this is how you can supercharge your favorite methods and produce novel results.

Cross-post: anhinga-anhinga.livejournal.com/84392.html

I hope for us all to have a creative and safe New Year.

Computer art news: I started to play with OpenGL shaders and with Shadertoy.com: dmm.dreamwidth.org/20076.html



Other news: books and stories, my own texts, open source activity and software experiments, employment change, etc: dmm.dreamwidth.org/23061.html

I generally shifted quite a bit towards dreamwidth and this blog during this year, and away from livejournal; this was not planned, but just happened "organically". Most of my activity this year was at dmm.dreamwidth.org

Crosspost: anhinga-anhinga.livejournal.com/83885.html

Continuing the last couple of posts

https://anhinga-anhinga.livejournal.com/82953.html

and( Read more... )

https://www.youtube.com/watch?v=fEWcg_A5UZc

More on linear models of computation:

( Read more... )

This is a prototype of an elderly home care robot developed by a very small group at IBM (with the benign indifference from their employer corporation that does not want to deal with robots and headaches and liabilities associated with robots). Its ability to verbally communicate with a human and to learn from a human is very impressive (basically, one can program this robot to a large extent simply by talking to it). Here is a demo video from the talk at the AGI-12 conference:

http://www.youtube.com/watch?v=M2RXDI3QYNU

The paper itself, "An Extensible Language Interface for Robot Manipulation", explaining to some extent how this works can be found here:

http://www.mindmakers.org/boards/18/topics/73

and the free online version of AGI-12 proceedings is here (scroll down to AGI-12 Contributed Paper Sessions):

http://www.mindmakers.org/projects/agiconf-2012/wiki/Schedule

"The AGI conferences are the only major conference series devoted wholly and specifically to the creation of AI systems possessing general intelligence at the human level and ultimately beyond." This is a small conference (I expect around 200 people, give or take), and this year it comes in two parts, AGI-12 and AGI-Impacts:

http://agi-conference.org/2012/schedule/

http://www.winterintelligence.org/#calendar

The videos from the AGI-11 are available here:

http://agi-conf.org/2011/

http://boingboing.net/2012/10/28/when-your-news-comes-from-nerd.html

'It takes a while watching TWC before you realize that they are such weather nerds that they sometimes tend to see things from the storm's point of view. They talk about the shape of the storm as beautiful, or "great," or "improving," and what they mean is that the storm is thriving. It's along the lines of, "This storm is looking great. Your lawn furniture? Not so much."'

(also http://boingboing.net/2012/10/28/eastern-us-hunkers-down-for.html )

A bit of information related to machine learning and philosophy talks at MIT.( Read more... )

Is there any good reason to restrict the availability of the Friends page by the most recent two weeks?

Does this restriction really help LJ to scale?

"In Charo City, in a cathedral to Christ Uploaded, which to my surprise he asked for, I married Scile according to Bremen law, in the second degree, registering as a nonconnubial love-match, and I took him to Embassytown."