Context Prediction Using Neural Networks
DOI:
https://doi.org/10.61841/gskqsp10Keywords:
Prediction, Ne.ural NetworksAbstract
— Word-prediction can be thought of as predicting a word, given its context. It is similar to predictive text used by multiple devices. The concept of word-embeddings is ideal for this type of a problem. Embeddings are an alternate form of word representation, such that it preserves the syntactic and semantic information. It extends the functionality to extract or study the relationships between various words, given in a corpus. The objective can be viewed as, (i) generation of quality embeddings,(ii) using neural networks to realise a model, capable of predicting words accurately, given the context as input. The generation of embeddings are a tedious task and requires a good amount of computation. To use these in an environment that is supported by machine learning algorithms requires transformation of these embeddings and corpus into vector space, i.e, a numeric form of representation. Word prediction can be of many types, depending on the model. For example, if the objective is to predict the (n+1)th word, given the previous n words; The automaton can consider only the previous n or (n,n-1) or (n,n-1,n-2…) words for predicting the next word, the earlier two are referred as bigram and trigram model respectively.
Downloads
References
1. Y. Bengio, R. Ducharme, P. Vincent. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137-1155, 2003.
2. Y. Bengio, Y. LeCun. Scaling learning algorithms towards AI. In: Large-Scale Kernel Machines, MIT Press, 2007.
3. Sebastian Sudholt, Gernot A. Fink “Evaluating Word String Embeddings and Loss Functions for NN- Based Word Spotting”. IAPR International Conference on Document Analysis and Recognition (ICDAR) 2017.
4. Wen-tau Yih, Geoffrey Zweig “Linguistic Regularities in Continuous Space Word Representations”.
North American Chapter of the Association for Computational Linguistics(NAACL) 2014.
5. Zhila, W.T. Yih, C. Meek, G. Zweig, T. Mikolov. Combining Heterogeneous Models for Measuring Relational Similarity. NAACL HLT 2013.
6. Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean” Efficient Estimation of Word Representations in Vector Space”. The International Conference on Learning Representations (ICLR) 2013.
7. D.A. Jurgens, S.M. Mohammad, P.D. Turney, K.J. Holyoak. Semeval-2012 task 2: Measuring degrees of relational similarity. In: Proceedings of the 6th International Workshop on Semantic Evaluation (SemEval 2012), 2012.
8. A.L. Maas, R.E. Daly, P.T. Pham, D. Huang, A.Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of ACL, 2011.
9. T. Mikolov, A. Deoras, D. Povey, L. Burget, J. Cernock ˇ y. Strategies for Training Large Scale ´ Neural Network Language Models, In: Proc. Automatic Speech Recognition and Understanding, 2011.
10. T. Mikolov, W.T. Yih, G. Zweig. Linguistic Regularities in Continuous Space Word Representations.
NAACL HLT 2013. H. Schwenk. Continuous space language models. Computer Speech and Language, vol. 21, 2007.
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
You are free to:
- Share — copy and redistribute the material in any medium or format for any purpose, even commercially.
- Adapt — remix, transform, and build upon the material for any purpose, even commercially.
- The licensor cannot revoke these freedoms as long as you follow the license terms.
Under the following terms:
- Attribution — You must give appropriate credit , provide a link to the license, and indicate if changes were made . You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
- No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.
Notices:
You do not have to comply with the license for elements of the material in the public domain or where your use is permitted by an applicable exception or limitation .
No warranties are given. The license may not give you all of the permissions necessary for your intended use. For example, other rights such as publicity, privacy, or moral rights may limit how you use the material.
