1st Tuesday Journal-Paper Club
After a break since march, 1TJC moves back to The Brew House.
Sam Hames has kindly volunteered to be our reader leader on an interesting and relevent topic:
Random Search for Hyper-Parameter Optimization, James Bergstra, Yoshua Bengio, Journal of Machine Learning Research 13 (2012) 281-305
http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf
About the 1st Tuesday Journal-Paper Club: the idea is to meet regularly, usually on the 1st Tuesday of the month as the name suggests (inspired by the ABC TV series "1st Tuesday Book Club"). Each month, the participants would agree on a highly cited, 'top ten' or major-prize-winning article in an SPS or ComSoc journal (but not one of our own!). We would also select a Discussion Leader. Through the month, each of the participants would read the article. At the next meeting, the Discussion Leader would lead a discussion of that article, starting with his/her own appraisal. In this way, it is hoped that we could all broaden our understanding of the field and further develop a sense of community.
1st rule of 1st Tuesday Journal-Paper Club: tell everyone about 1st Tuesday Journal-Paper Club.
Date and Time
Location
Hosts
Registration
- Date: 02 Aug 2016
- Time: 06:00 PM to 08:00 PM
- All times are (UTC+10:00) Brisbane
- Add Event to Calendar
- 601 Stanley St
- Woolloongabba
- Brisbane, Queensland
- Australia 4102
- Building: The Brew House
- Room Number: Bar area
- Contact Event Host
-
We will also be looking for suggestions for papers to discuss and volunteer reader leaders for rest of the 2016 schedule...
- Co-sponsored by Andrew Bradley
- Starts 27 June 2016 06:00 AM
- Ends 02 August 2016 12:00 PM
- All times are (UTC+10:00) Brisbane
- No Admission Charge
Agenda
Abstract
Grid search and manual search are the most widely used strategies for hyper-parameter optimiza- tion. This paper shows empirically and theoretically that randomly chosen trials are more efficient for hyper-parameter optimization than trials on a grid. Empirical evidence comes from a compar- ison with a large previous study that used grid search and manual search to configure neural net- works and deep belief networks. Compared with neural networks configured by a pure grid search, we find that random search over the same domain is able to find models that are as good or better within a small fraction of the computation time. Granting random search the same computational budget, random search finds better models by effectively searching a larger, less promising con- figuration space. Compared with deep belief networks configured by a thoughtful combination of manual search and grid search, purely random search over the same 32-dimensional configuration space found statistically equal performance on four of seven data sets, and superior performance on one of seven. A Gaussian process analysis of the function from hyper-parameters to validation set performance reveals that for most data sets only a few of the hyper-parameters really matter, but that different hyper-parameters are important on different data sets. This phenomenon makes grid search a poor choice for configuring algorithms for new data sets. Our analysis casts some light on why recent “High Throughput” methods achieve surprising success—they appear to search through a large number of hyper-parameters because most hyper-parameters do not matter much. We anticipate that growing interest in large hierarchical models will place an increasing burden on techniques for hyper-parameter optimization; this work shows that random search is a natural base- line against which to judge progress in the development of adaptive (sequential) hyper-parameter optimization algorithms.
Keywords: global optimization, model selection, neural networks, deep learning, response surface modeling