What were the main advances in machine learning/artificial intelligence in 2016?

njsuntop

　　2016may very well go down in history as the year of “the Machine Learning hype”. Everyone now seems to be doing machine learning, and if they are not, they are thinking of buying a startup to claim they do.

　　Now, to be fair, there are reasons for much of that “hype”. Can you believe that it has been only a year since Google announcedthey were open sourcing Tensor Flow? TF is already a very active project that is being used for anything ranging from drug discovery to generating music. Google has not been the only company open sourcing their ML software though, many followed lead. Microsoft open sourced CNTK, Baidu announced the>　　Besidesthe “hype” and the outpour of support from companies to machine learning open source projects, 2016 has also seen a great deal of applications of machine learning that were almost unimaginable a few months back. I was particularly impressed by the quality of Wavenet’saudio generation. Having worked on similar problems in the past I can appreciate those results. I would also highlight some of the recent results in lip reading,a great application of video recognition that is likely to be very useful (and maybe scary) in the near future. I should also mention Google’s impressive advances in machine translation. It is amazing to see how much this area has improved in a year.
　　Asa matter of fact, machine translation is not the only interesting advance we have seen in machine learning for language technologies this past year. I think it is very interesting to see some of the recent approaches to combine deep sequential networks with side-information in order to produce richer language models. In “A Neural Knowledge Language Model”, Bengio’s team combines knowledge graphs with RNNs, and in “Contextual LSTM models for Large scale NLP Tasks”,the Deepmind folks incorporate topics into the LSTM model. We have alsoseen a lot of interesting work in modeling attention and memory for language models. As an example, I would recommend “Ask Me Anything: Dynamic Memory Networks for NLP”, presented in this year’s ICML.
　　Also,I should at least mention a couple of things from NIPS 2016 in Barcelona. Unfortunately, I had to miss the conference the one time thatwas happening in my hometown. I did follow from the distance though. And from what I gathered, the two hottest topics were probably Generative Adversarial Networks (including the very popular tutorial by Ian Goodfellow) and the combination of probabilistic models with Deep Learning.
　　Letme also mention some of the advances in my main area of expertise: Recommender Systems. Of course Deep Learning has also impacted this area. While I would still not recommend DL as the default approach to recommender systems, it is interesting to see how it is already being used in practice, and in large scale, by products like Youtube.That said, there has been interesting research in the area that is not related to Deep Learning. The best paper award in this year’s ACM Recsyswent to “Local Item-Item Models For Top-N Recommendation”, an interesting extension to Sparse Linear Methods (i.e. SLIM) using an initial unsupervised clustering step. Also, “Field-aware Factorization Machines for CTR Prediction”, which describes the winning approach to the Criteo CTR Prediction Kaggle Challenge is a good reminder that Factorization Machines are still a good tool to have in your ML toolkit.

　　Icould probably go on for several other paragraphs just listing impactful advances in machine learning in the last 12 months. Note that Ihaven’t even listed any of the breakthroughs>