2017-04-06

Learning to Learn by Gradient Descent by Gradient Descent

Authors

Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, Nando de Freitas

Abstract

A family of unimplemented computing languages is described that is intended to span differences of application area by a unified framework. This framework dictates the rules about the uses of user-coined names, and the conventions about characterizing functional relationships. Within this framework the design of a specific language splits into two independent parts. One is the choice of written appearances of programs (or more generally, their physical representation). The other is the choice of the abstract entities (such as numbers, character-strings, list of them, functional relations among them) that can be referred to in the language. The system is biased towards “expressions” rather than “statements.” It includes a nonprocedural (purely functional) subsystem that aims to expand the class of users' needs that can be met by a single print-instruction, without sacrificing the important properties that make conventional right-hand-side expressions easy to construct and understand.

Guests

Finbarr Timbers, Tylan Murphy

Discussion Notes

• At a high-level, the presented concept of learning to learn was fascinating.
• The argument that optimization of learning is only enabled with a specialization to a subclass of problems, supported by the reference to the No Free Lunch Theorems of Optimization, caused us to muse that such ability to learn to learn may be a necessary component of strong AI.
• Found it hard to understanding and interpret the results, it would have been helpful in particular to understand better before reading the paper the hand-crafted optimizers such as ADAM RMSprop, SGD, NAG, LSTM.