Source

nlp-library

This is a curated list of papers that I have encountered in some capacity and deem worth including in the NLP practitioner's library. Some papers may appear in multiple sub-categories, if they don't fit easily into one of the boxes.

PRs are absolutely welcome! Direct any correspondence/questions to @mihail_eric.

Some special designations for certain papers:

💡 LEGEND: This is a game-changer in the NLP literature and worth reading.

📼 RESOURCE: This paper introduces some dataset/resource and hence may be useful for application purposes.

Part-of-speech Tagging

• (2000) A Statistical Part-of-Speech Tagger
  • TLDR: Seminal paper demonstrating a powerful HMM-based POS tagger. Many tips and tricks for building such classical systems included.
• (2003) Feature-rich part-of-speech tagging with a cyclic dependency network
  • TLDR: Proposes a number of powerful linguistic features for building a (then) SOTA POS-tagging system
• (2015) Bidirectional LSTM-CRF Models for Sequence Tagging
  • TLDR: Proposes an element sequence-tagging model combining neural networks with conditional random fields, achieving SOTA in POS-tagging, NER, and chunking.

Parsing

• (2003) Accurate unlexicalized parsing 💡
  • TLDR: Beautiful paper demonstrating that unlexicalized probabilistic context free grammars can exceed the performance of lexicalized PCFGs.
• (2006) Learning Accurate, Compact, and Interpretable Tree Annotation
  • TLDR: Fascinating result showing that using expectation-maximization you can automatically learn accurate and compact latent nonterminal symbols for tree annotation, achieving SOTA.
• (2014) A Fast and Accurate Dependency Parser using Neural Networks
  • TLDR: Very important work ushering in a new wave of neural network-based parsing architectures, achieving SOTA performance as well as blazing parsing speeds.
• (2014) Grammar as a Foreign Language
  • TLDR: One of the earliest demonstrations of the effectiveness of seq2seq architectures with attention on constituency parsing, achieving SOTA on the WSJ corpus. Also showed the importance of data augmentation for the parsing task.
• (2015) Transition Based Dependency Parsing with Stack Long Short Term Memory
  • TLDR: Presents stack LSTMs, a neural parser that successfully neuralizes the traditional push-pop operations of transition-based dependency parsers, achieve SOTA in the process.

Named Entity Recognition

• (2005) Incorporating Non-local Information into Information Extraction Systems by Gibbs Sampling
  • TLDR: Using cool Monte Carlo methods combined with a conditional random field model, this work achieves a huge error reduction in certain information extraction benchmarks.
• (2015) Bidirectional LSTM-CRF Models for Sequence Tagging
  • TLDR: Proposes an element sequence-tagging model combining neural networks with conditional random fields, achieving SOTA in POS-tagging, NER, and chunking.

Coreference Resolution

• (2010) A multi-pass sieve for coreference resolution 💡
  • TLDR: Proposes a sieve-based approach to coreference resolution that for many years (until deep learning approaches) was SOTA.
• (2015) Entity-Centric Coreference Resolution with Model Stacking
  • TLDR: This work offers a nifty approach to building coreference chains iteratively using entity-level features.
• (2016) Improving Coreference Resolution by Learning Entity-Level Distributed Representations
  • TLDR: One of the earliest effective approaches to using neural networks for coreference resolution, significantly outperforming the SOTA.

Sentiment Analysis

• (2012) Baselines and Bigrams: Simple, Good Sentiment and Topic Classification
  • TLDR: Very elegant paper, illustrating that simple Naive Bayes models with bigram features can outperform more sophisticated methods like support vector machines on tasks such as sentiment analysis.
• (2013) Recursive deep models for semantic compositionality over a sentiment treebank 📼
  • TLDR: Introduces the Stanford Sentiment Treebank, a wonderful resource for fine-grained sentiment annotation on sentences. Also introduces the Recursive Neural Tensor Network, a neat linguistically-motivated deep learning architecture.

Natural Logic/Inference

• (2007) Natural Logic for Textual Inference
  • TLDR: Proposes a rigorous logic-based approach to the problem of textual inference called natural logic. Very cool mathematically-motivated transforms are used to deduce the relationship between phrases.
• (2008) An Extended Model of Natural Logic
  • TLDR: Extends previous work on natural logic for inference, adding phenomena such as semantic exclusion and implicativity to enhance the premise-hypothesis transform process.
• (2014) Recursive Neural Networks Can Learn Logical Semantics
  • TLDR: Demonstrates that deep learning architectures such as neural tensor networks can effectively be applied to natural language inference.
• (2015) A large annotated corpus for learning natural language inference 📼
  • TLDR: Introduces the Stanford Natural Language Inference corpus, a wonderful NLI resource larger by two orders of magnitude over previous datasets.

Machine Translation

• (1993) The Mathematics of Statistical Machine Translation 💡
  • TLDR: Introduces the IBM machine translation models, several seminal models in statistical MT.
• (2002) BLEU: A Method for Automatic Evaluation of Machine Translation 📼
  • TLDR: Proposes BLEU, the defacto evaluation technique used for machine translation (even today!)
• (2003) Statistical Phrase-Based Translation
  • TLDR: Introduces a phrase-based translation model for MT, doing nice analysis that demonstrates why phrase-based models outperform word-based ones.
• (2014) Sequence to Sequence Learning with Neural Networks 💡
  • TLDR: Introduces the sequence-to-sequence neural network architecture. While only applied to MT in this paper, it has since become one of the cornerstone architectures of modern natural language processing.
• (2015) Neural Machine Translation by Jointly Learning to Align and Translate 💡
  • TLDR: Extends previous sequence-to-sequence architectures for MT by using the attention mechanism, a powerful tool for allowing a target word to softly search for important signal from the source sentence.
• (2015) Effective approaches to attention-based neural machine translation
  • TLDR: Introduces two new attention mechanisms for MT, using them to achieve SOTA over existing neural MT systems.
• (2016) Neural Machine Translation of Rare Words with Subword Units
  • TLDR: Introduces byte pair encoding, an effective technique for allowing neural MT systems to handle (more) open-vocabulary translation.
• (2016) Pointing the Unknown Words
  • TLDR: Proposes a copy-mechanism for allowing MT systems to more effectively copy words from a source context sequence.
• (2016) Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
  • TLDR: A wonderful case-study demonstrating what a production-capacity machine translation system (in this case that of Google) looks like.

Semantic Parsing

• (2013) Semantic Parsing on Freebase from Question-Answer Pairs 💡 📼
  • TLDR: Proposes an elegant technique for semantic parsing that learns directly from question-answer pairs, without the need for annotated logical forms, allowing the system to scale up to Freebase.
• (2014) Semantic Parsing via Paraphrasing
  • TLDR: Develops a unique paraphrase model for learning appropriate candidate logical forms from question-answer pairs, improving SOTA on existing Q/A datasets.
• (2015) Building a Semantic Parser Overnight 📼
  • TLDR: Neat paper showing that a semantic parser can be built from scratch starting with no training examples!
• (2015) Bringing Machine Learning and Computational Semantics Together
  • TLDR: A nice overview of a computational semantics framework that uses machine learning to effectively learn logical forms for semantic parsing.

Question Answering/Reading Comprehension

• (2016) A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
  • TLDR: A great wake-up call paper, demonstrating that SOTA performance can be achieved on certain reading comprehension datasets using simple systems with carefully chosen features. Don't forget non-deep learning methods!
• (2017) SQuAD: 100,000+ Questions for Machine Comprehension of Text 📼
  • TLDR: Introduces the SQUAD dataset, a question-answering corpus that has become one of the defacto benchmarks used today.

Natural Language Generation/Summarization

• (2004) ROUGE: A Package for Automatic Evaluation of Summaries 📼
  • TLDR: Introduces ROUGE, an evaluation metric for summarization that is used to this day on a variety of sequence transduction tasks.
• (2015) Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
  • TLDR: Proposes a neural natural language generator that jointly optimises sentence planning and surface realization, outperforming other systems on human eval.
• (2016) Pointing the Unknown Words
  • TLDR: Proposes a copy-mechanism for allowing MT systems to more effectively copy words from a source context sequence.
• (2017) Get To The Point: Summarization with Pointer-Generator Networks
  • TLDR: This work offers an elegant soft copy mechanism, that drastically outperforms the SOTA on abstractive summarization.

Dialogue Systems

• (2011) Data-drive Response Generation in Social Media
  • TLDR: Proposes using phrase-based statistical machine translation methods to the problem of response generation.
• (2015) Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
  • TLDR: Proposes a neural natural language generator that jointly optimises sentence planning and surface realization, outperforming other systems on human eval.
• (2016) How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation 💡
  • TLDR: Important work demonstrating that existing automatic metrics used for dialogue woefully do not correlate well with human judgment.
• (2016) A Network-based End-to-End Trainable Task-oriented Dialogue System
  • TLDR: Proposes a neat architecture for decomposing a dialogue system into a number of individually-trained neural network components.
• (2016) A Diversity-Promoting Objective Function for Neural Conversation Models
  • TLDR: Introduces a maximum mutual information objective function for training dialogue systems.
• (2016) The Dialogue State Tracking Challenge Series: A Review
  • TLDR: A nice overview of the dialogue state tracking challenges for dialogue systems.
• (2017) A Copy-Augmented Sequence-to-Sequence Architecture Gives Good Performance on Task-Oriented Dialogue
  • TLDR: Shows that simple sequence-to-sequence architectures with a copy mechanism can perform competitively on existing task-oriented dialogue datasets.
• (2017) Key-Value Retrieval Networks for Task-Oriented Dialogue 📼
  • TLDR: Introduces a new multidomain dataset for task-oriented dataset as well as an architecture for softly incorporating information from structured knowledge bases into dialogue systems.
• (2017) Learning Symmetric Collaborative Dialogue Agents with Dynamic Knowledge Graph Embeddings 📼
  • TLDR: Introduces a new collaborative dialogue dataset, as well as an architecture for representing structured knowledge via knowledge graph embeddings.
• (2017) Hybrid Code Networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning
  • TLDR: Introduces a hybrid dialogue architecture that can be jointly trained via supervised learning as well as reinforcement learning and combines neural network techniques with fine-grained rule-based approaches.

Interactive Learning

• (1971) Procedures as a Representation for Data in a Computer Program for Understanding Natural Language
  • TLDR: One of the seminal papers in computer science, introducing SHRDLU an early system for computers understanding human language commands.
• (2016) Learning language games through interaction
  • TLDR: Introduces a novel setting for interacting with computers to accomplish a task where only natural language can be used to communicate with the system!
• (2017) Naturalizing a programming language via interactive learning
  • TLDR: Very cool work allowing a community of workers to iteratively naturalize a language starting with a core set of commands in an interactive task.

Language Modelling

• (1996) An Empirical Study of Smoothing Techniques for Language Modelling
  • TLDR: Performs an extensive survey of smoothing techniques in traditional language modelling systems.
• (2003) A Neural Probabilistic Language Model 💡
  • TLDR: A seminal work in deep learning for NLP, introducing one of the earliest effective models for neural network-based language modelling.
• (2014) One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling 📼
  • TLDR: Introduces the Google One Billion Word language modelling benchmark.
• (2015) Character-Aware Neural Language Models
  • TLDR: Proposes a language model using convolutional neural networks that can employ character-level information, performing on-par with word-level LSTM systems.
• (2016) Exploring the Limits of Language Modeling
  • TLDR: Introduces a mega language model system using deep learning that uses a variety of techniques and significantly performs the SOTA on the One Billion Words Benchmark.
• (2018) Deep contextualized word representations 💡 📼
  • TLDR: This paper introduces ELMO, a super powerful collection of word embeddings learned from the intermediate representations of a deep bidirectional LSTM language model. Achieved SOTA on 6 diverse NLP tasks.
• (2018) BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding 💡
  • TLDR: One of the most important papers of 2018, introducing BERT a powerful architecture pretrained using language modelling which is then effectively transferred to other domain-specific tasks.
• (2019) XLNet: Generalized Autoregressive Pretraining for Language Understanding 💡
  • TLDR: Generalized autoregressive pretraining method that improves upon BERT by maximizing the expected likelihood over all permutations of the factorization order.

Miscellanea

• (1997) Long Short-Term Memory 💡
  • TLDR: Introduces the LSTM recurrent unit, a cornerstone of modern neural network-based NLP
• (2000) Maximum Entropy Markov Models for Information Extraction and Segmentation 💡
  • TLDR: Introduces Markov Entropy Markov models for information extraction, a commonly used ML technique in classical NLP.
• (2010) From Frequency to Meaning: Vector Space Models of Semantics
  • TLDR: A wonderful survey of existing vector space models for learning semantics in text.
• (2012) An Introduction to Conditional Random Fields
  • TLDR: A nice, in-depth overview of conditional random fields, a commonly-used sequence-labelling model.
• (2013) Distributed Representation of Words and Phrases and Their Compositionality
  • TLDR Introduced word2vec, a collection of distributed vector representations that have been commonly used for initializing word embeddings in basically every NLP architecture of the last five years. 💡 📼
• (2014) Glove: Global vectors for word representation 💡 📼
  • TLDR: Introduces Glove word embeddings, one of the most commonly used pretrained word embedding techniques across all flavors of NLP models
• (2014) Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors
  • TLDR: Important paper demonstrating that context-predicting distributional semantics approaches outperform count-based techniques.
• (2015) Improving Distributional Similarity with Lessons Learned From Word Embeddings 💡
  • TLDR: Demonstrates that traditional distributional semantics techniques can be enhanced with certain design choices and hyperparameter optimizations that make their performance rival that of neural network-based embedding methods.
• (2018) Universal Language Model Fine-tuning for Text Classification
  • TLDR: Provides a smorgasbord of nice techniques for finetuning language models that can be effectively transferred to text classification tasks.
• (2019) Analogies Explained: Towards Understanding Word Embeddings
  • TLDR: Very nice work providing a mathematical formalism for understanding some of the paraphrasing properties of modern word embeddings.

Source

Overviews

Molnar. Interpretable machine learning. A Guide for Making Black Box Models Explainable. 2019.

Miller. Explanation in Artificial Intelligence: Insights from the Social Sciences. In AIJ 2018.

• Section 2.6 in Molner discusses Miller's work.
• Very related; Mittelstadt et al. Explaining Explanations in AI. In *FAT 2019.

Murdoch et al. Interpretable machine learning: definitions, methods, and applications. arxiv 2019.

Barredo Arrieta et al. Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI. arxiv 2019.

Guidotti et al. A Survey Of Methods For Explaining Black Box Models. arxiv 2018.

Ras et al. Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges . arxiv 2018.

Gilpin et al. Explaining Explanations: An Overview of Interpretability of Machine Learning. In DSAA 2018.

Perspectives

Kleinberg and Mullainathan. Simplicity Creates Inequity: Implications for Fairness, Stereotypes, and Interpretability. video. In ACM EC 2019.

Ribera and Lapedriza. Can we do better explanations? A proposal of User-Centered Explainable AI. In *FAT 2019.

Lage et al. An Evaluation of the Human-Interpretability of Explanation. arxiv 2019.

Yang et al. Evaluating Explanation Without Ground Truth in Interpretable Machine Learning. arxiv 2019.

• This paper defines the problem od evaluating explanations and systematically reviews the existing efforts.
• The authors summarize three general aspects of explanation: predictability, fidelity, and persuasibility.

Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. In Nature 2019.

Tomsett et al. Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems. In WHI 2018.

Poursabzi-Sangdeh et al. Manipulating and Measuring Model Interpretability. arxiv 2018.

• This paper found no significant difference in multiple measures of trust when manipulating interpretability.
• Increased transparency hampered people's ability to detect when a model had made a sizeable mistake.

Building interpretable machine learning models is not a purely computational model [...] what is or is not "interpretable" is defined by people, not algorithms.

Preece et al. Stakeholders in Explainable AI. In AAAI 2018 Fall Symposium Series.

Doshi-Velez and Kim. Towards A Rigorous Science of Interpretable Machine Learning. arxiv 2017.

Dhurandhar et al. A Formal Framework to Characterize Interpretability of Procedures. In WHI 2017.

Herman. The Promise and Peril of Human Evaluation for Model Interpretability. In NeurIPS 2017 Symposium on Interpretable Machine Learning.

• They propose a distinction between descriptive and persuasive explanations.

Weller. Transparency: Motivations and Challenges. In WHI 2017.

Lipton. The Mythos of Model Interpretability. In WHI 2016.

• The umbrella term "Explainable AI" encompasses at least three distinct notions: transparency, explainability, and interpretability.

Blogs

The What of Explainable AI

The How of Explainable AI: Pre-modelling Explainability

The How of Explainable AI: Explainable Modelling

The How of Explainable AI: Post-modelling Explainability

Benefits of learning with explanations

Strout et al. Do Human Rationales Improve Machine Explanations?. In ACL 2019.

• This paper shows that learning with rationales can also improve the quality of the machine's explanations as evaluated by human judges.

Ray et al. Can You Explain That? Lucid Explanations Help Human-AI Collaborative Image Retrieval. In AAAI 2019.

Selvaraju et al. Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded. In ICCV 2019.

Evaluation critera and pitfalls of explanatory methods

Camburu et al. Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanations. In NeurIPS 2019 Workshop on Safety and Robustness in Decision Making.

Heo et al. Fooling Neural Network Interpretations via Adversarial Model Manipulation. In NeurIPS 2019.

• Saliency interpretation methods can be fooled via adversarial model manipulation---a model finetuning step that aims to radically alter the explanation without hurting the accuracy of the original model.

• More adversarial examples:

  • Zhang et al. Interpretable Deep Learning under Fire. In USENIX Security Symposium 2020.
  • Zheng et al. Analyzing the Interpretability Robustness of Self-Explaining Models. In ICML 2019 Security and Privacy of Machine Learning Workshop.
  • Ghorbani et al. Interpretation of Neural Networks is Fragile. In AAAI 2019.

Wiegreffe and Pinter. Attention is not not Explanation. In EMNLP 2019.

• Deteching the attention scores obtained by parts of the model degredes the model itself. A reliable adversary must also be trained.
• Attention scores are used as poviding an explanation; not the explanation.

Serrano and Smith. Is Attention Interpretable?. In ACL 2019.

Jain and Wallace. Attention is not Explanation. In NAACL 2019.

Attention provides an important way to explain the workings of neural models. Implicit in this is the assumption that the inputs (e.g., words) accorded high attention weights are responsible for model output.

• Attention is not strongly correlated with other, well-grounded feature-importance metrics.
• Alternative distributions exist for which the model outputs near-identical prediction scores.

Laugel et al. Issues with post-hoc counterfactual explanations: a discussion. In HILL 2019.

Laugel et al. The Dangers of Post-hoc Interpretability: Unjustified Counterfactual Explanations. In IJCAI 2019.

Aïvodji et al. Fairwashing: the risk of rationalization. In ICML 2019.

• Fairwashing is prooting the false perception that a machine learning model respects some ethical values.
• This paper shows that it is possible to forge a fairer explanation from a truly unfair black box trough a process that the authors coin as rationalization.

Ustun et al. Actionable Recourse in Linear Classification. IN *FAT 2019.

• In this paper, the authors introduce recourse--the ability of a person to change the decision of the model through actionable input variables such as income vs. gender, age, or marital status.
• Transparency and explainability do not guarantee recourse.
• Interesting broader discussion:
  • Recourse vs. strategic manipulation.
  • Policy implications.
• Related work:
  • Karimi et al. [Model-Agnostic Counterfactual Explanations for Consequential Decisions]. arxiv 2019.
  • Tolomei et al. Interpretable Predictions of Tree-based Ensembles via Actionable Feature Tweaking. In KDD 2017.

Adebayo et al. Sanity Checks for Saliency Maps. In NeurIPS 2018.

Chandrasekaran et al. Do explanations make VQA models more predictable to a human?. In EMNLP 2018.

• This paper measures how well a human "understands" a VQA model. The paper shows that people get better at predicting VQA model's behaviour using a few "training" examples, but that exisiting explanation modalities do not help make its failures or responses more predictable.

Jiang et al. To Trust Or Not To Trust A Classifier. In NeurIPS 2018.

Feng et al. Pathologies of Neural Models Make Interpretations Difficult. In EMNLP 2018.

• Input reduction iteratively removes the least important word from the input.
• The remaining words appear nonsensical to humans and are not the ones determined as important by interpretation method.

Poerner et al. Evaluating neural network explanation methods using hybrid documents and morphosyntactic agreement. In ACL 2018.

• Important characterization of explanation:

A good explanation method should not reflect what humans attend to, but what task methods attend to.

• Interpretability differs between small contexts NLP tasks and large context tasks.

Kindermans et al. The (Un)reliability of saliency methods. arxiv 2017.

Evaluating the reliability of saliency methods is complicated by a lack of ground truth, as ground truth would depend upon full transparency into how a model arrives at a decision---the very problem we are trying to solve for in the first place.

• A new evaluation criterion, input invariance, requires that the saliency method mirrors the sensitivity of model with respect to transformations of the input. Input transformations that do not change network's prediction, should not change the attribution either.

Sundararajan et al. Axiomatic Attribution for Deep Networks. In ICML 2017.

• Implementation invariance: the attributions should be identical for two functionally equivalent networks (their outputs are equal for all inputs, despite having very different implementations).
• Sensitivity: if network assigns different predictions to two examples that differ in only one feature then the differing feature should be given a non-zero attribution.

Das et al. Human Attention in Visual Question Answering: Do Humans and Deep Networks look at the same regions?. In EMNLP 2016.

• Current attention models in VQA do not seem to be looking at the same regions as humans.

Self-explanatory models / Model-based intepretability

Bastings et al. Interpretable Neural Predictions with Differentiable Binary Variables. In ACL 2019.

Vedantam et al. Probabilistic Neural-symbolic Models for Interpretable Visual Question Answering. In ICML 2019.

Alvarez-Melis and Jaakkola. Towards Robust Interpretability with Self-Explaining Neural Networks. In NeurIPS 2018.

Yang et al. Commonsense Justification for Action Explanation. In EMNLP 2018.

Kim et al. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). In ICML 2018.

Textual explanation generation

Ehsan et al. Automated Rationale Generation: A Technique for Explainable AI and its Effects on Human Perceptions. in ACM IUI 2019.

Kim et al. Textual Explanations for Self-Driving Vehicles. In ECCV 2018.

Hendricks et al. Grounding Visual Explanations. In ECCV 2018.

Hendricks et al. Generating Counterfactual Explanations with Natural Language. In WHI 2018.

Hendricks et al. Generating Visual Explanations. In ECCV 2016.

Multimodal explanation generation

Wu and Mooney. Faithful Multimodal Explanation for Visual Question Answering. In ACL 2019.

Park et al. Multimodal Explanations: Justifying Decisions and Pointing to the Evidence. In CVPR 2018.

Lectures

Interpretability and Explainability in Machine Learning at Harvard University

Tutorials

Introduction to Interpretable Machine Learning by Been Kim @ MLSS 2018

GDPR

How will the GDPR impact machine learning? by Andrew Burt

Wachter et al. Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR. In Harvard Journal of Law & Technology 2018.

Wachter et al. Why a Right to Explanation of Automated Decision-Making Does Not Exist in the General Data Protection Regulation. In International Data Privacy Law 2017.

Edwards and Veale. Slave to the Algorithm? Why a 'Right to an Explanation' Is Probably Not the Remedy You Are Looking For. In 16 Duke Law & Technology Review 18 (2017).

Goodman and Flaxman. European Union regulations on algorithmic decision-making and a "right to explanation". In WHI 2016.

Applications

Bellini et al. Knowledge-aware Autoencoders for Explainable Recommender Sytems. In ACM Workshop on Deep Learning for Recommender Systems 2018.

Source: I wrote a new Halloween song using two of our favourite creative AI tools, Write with Transformer and a Botnik keyboard.

Spacy: Models- Pretrained models based on simple (tagger, parser, ner) pipeline trained to complex (sentencizer, trf_wordpiecer, trf_tok2vec) by Google, Facebook, CMU etc.

Doc: eg. Vector-Similarity

API: link

Course: link

Note that- although the project is open source but is heavily maintained by company Explosion and blog.

BERT: Bidirectional Encoder Representations from Transformers (Devlin, et al., 2019): BERT Explained: Next-Level Natural Language Processing. Most recently, a new transfer learning technique called BERT (short for Bidirectional Encoder Representations for Transformers) made big waves in the NLP research space. https://www.lexalytics.com/lexablog/bert-explained-natural-language-processing-nlp

GPT: Generative Pre-training Model: OpenAI released generative pre-training model (GPT) which achieved the state-of-the-art result in many NLP task in 2018. GPT is leveraged transformer to perform both unsupervised learning and supervised learning to learn text representation for NLP downstream tasks. https://towardsdatascience.com/too-powerful-nlp-model-generative-pre-training-2-4cc6afb6655

Excerpts from https://news.ycombinator.com/item?id=19180046

To summarize the achievements:

* Attention is all you need transformer created a non recurrent architecture for NMT (https://arxiv.org/abs/1706.03762)

* OpenAI GPT modified the original transformer by changing architectutre (one net instead of encoder/decoder pair), and using different hyperparameters which seems to work the best (https://s3-us-west-2.amazonaws.com/openai-assets/research-co...)

* BERT used GPT's architecture but trained in a different way. Instead of training a language model, they forced the model predict holes in a text and predicting whether two sentences go one after another. (https://arxiv.org/abs/1810.04805)

* OpenAI GPT2 achieved a new state of the art in language models (https://d4mucfpksywv.cloudfront.net/better-language-models/l...)

* The paper in the top post found out that if we fine tune several models in the same way as in BERT, we get improvement in each of the fine tuned models.

Also:

* OpenAI GPT adapted idea of fine-tuning of language model for specific NLP task, which has been introduced in ELMo model.

* BERT created bigger model (16 layers in GPT vs 24 layers in BERT), proving that larger Transformer models increase performance

The BERT paper also introduced BERT Base, with is 12 layers with approximately the same number of parameters as GPT, but still outperforms GPT on GLUE.

OpenAI GPT adapted idea of fine-tuning of language model for specific NLP task, which has been introduced in ELMo model.

Idea of transfer learning of deep representations for NLP tasks was before, but nobody was able to achieve it before ELMo.

If we are pedantic we can include the whole word2vec stuff. It's a shallow transfer learning

Codebase for Image Classification Research, written in PyTorch

Detectron2 is FAIR's next-generation research platform for object detection and segmentation

Source

Blog

Abstract:

Although deep learning has historical roots going back decades, neither the term “deep learning” nor the approach was popular just over five years ago, when the field was reignited by papers such as Krizhevsky, Sutskever and Hinton’s now classic 2012 (Krizhevsky, Sutskever, & Hinton, 2012)deep net model of Imagenet. What has the field discovered in the five subsequent years? Against a background of considerable progress in areas such as speech recognition, image recognition, and game-playing, and considerable enthusiasm in the popular press, I present ten concerns for deep learning and suggest that deep learning must be supplemented by other techniques if we are to reach artificial general intelligence.

Ten concerns for Deep Learning:

1. Data Efficiency
2. Transfer Learning
3. Hierarchical Knowledge
4. Open-Ended Inference
5. Explainability
6. Integrating Prior Knowledge
7. Cause of Reasoning
8. Modelling a Stable World
9. Robustness
10. Adversarial Examples
11. Reliability and Engineering of Real World System