How to cite this WIREs title:
WIREs Data Mining Knowl Discov

Impact Factor: 4.476

A review of keyphrase extraction

Advanced Review

Eirini Papagiannopoulou, Grigorios Tsoumakas

Published Online: Sep 18 2019

DOI: 10.1002/widm.1339

Full article on Wiley Online Library: HTML PDF

Can't access this content? Tell your librarian.

Abstract Keyphrase extraction is a textual information processing task concerned with the automatic extraction of representative and characteristic phrases from a document that express all the key aspects of its content. Keyphrases constitute a succinct conceptual summary of a document, which is very useful in digital information management systems for semantic indexing, faceted search, document clustering and classification. This article introduces keyphrase extraction, provides a well‐structured review of the existing work, offers interesting insights on the different evaluation approaches, highlights open issues and presents a comparative experimental study of popular unsupervised techniques on five datasets. This article is categorized under: Ensemble Methods > Web Mining Ensemble Methods > Text Mining

Presentation structure of the unsupervised keyphrase extraction methods

[ Normal View | Magnified View ]

Differences between the F₁‐scores based on the manual evaluation and the F₁‐scores based on the exact (solid line), partial (dashed line) and average (dotted line) evaluation approaches for the 50 manually evaluated documents given on the x axis

[ Normal View | Magnified View ]

Distribution of differences between the F₁‐scores based on the manual evaluation and the F₁‐scores based on the exact (Exact), partial (Partial) and average (Average) evaluation approaches for the 50 manually evaluated documents given on the x axis

[ Normal View | Magnified View ]

Box plots of the F₁@10 scores obtained via manual evaluation, exact matching, partial matching and taking the average of exact and partial matching

[ Normal View | Magnified View ]

Popularity of evaluation approaches. ACEK, MRR, MAP and B‐pref are used along with the exact match evaluation approach

[ Normal View | Magnified View ]

Presentation structure of supervised keyphrase extraction methods

[ Normal View | Magnified View ]

References

Alzaidy,, R., Caragea,, C., & Giles,, C. L. (2019). Bi‐lstm‐crf sequence labeling for keyphrase extraction from scholarly documents. Paper presented at the World Wide Web conference, WWW 2019, San Francisco, CA, USA, May 13–17, 2019, 2551–2557. https://doi.org/10.1145/3308558.3313642.
Augenstein,, I., Das,, M., Riedel,, S., Vikraman,, L., & McCallum,, A. (2017). Semeval 2017 task 10: Scienceie—Extracting keyphrases and relations from scientific publications. Paper presented at proceedings of the 11th international workshop on semantic evaluation, SemEval@ACL 2017, Vancouver, Canada, August 3–4, 2017, 546–555. https://doi.org/10.18653/v1/S17-2091.
Basaldella,, M., Antolli,, E., Serra,, G., & Tasso,, C. (2018). Bidirectional LSTM recurrent neural network for keyphrase extraction. Paper presented at proceedings of the 14th Italian research conference on digital libraries, IRCDL 2018, Udine, Italy, January 25–26, 2018, 180–187. https://doi.org/10.1007/978-3-319-73165-0_18.
Bekoulis,, G., Deleu,, J., Demeester,, T., & Develder,, C. (2018a). Adversarial training for multi‐context joint entity and relation extraction. Paper presented at proceedings of the 2018 conference on empirical methods in natural language processing, 2830–2836.
Bekoulis,, G., Deleu,, J., Demeester,, T., & Develder,, C. (2018b). Joint entity recognition and relation extraction as a multi‐head selection problem. Expert Systems with Applications, 114, 34–45.
Bengtson,, E., & Roth,, D. (2008). Understanding the value of features for coreference resolution. Paper presented at proceedings of the conference on empirical methods in natural language processing, EMNLP 2008, 25–27 October 2008, Honolulu, Hawaii, USA, a meeting of SIGDAT, a special interest group of the ACL, 294–303. http://www.aclweb.org/anthology/D08-1031.
Bennani‐Smires,, K., Musat,, C., Hossmann,, A., Baeriswyl,, M., & Jaggi,, M. (2018). Simple unsupervised keyphrase extraction using sentence embeddings. Paper presented at proceedings of the 22nd conference on computational natural language learning, Brussels, Belgium, October, 2018, 221–229. Brussels, Belgium: Association for Computational Linguistics. https://www.aclweb.org/anthology/K18-1022.
Blei,, D. M., Ng,, A. Y., & Jordan,, M. I. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022 http://www.jmlr.org/papers/v3/blei03a.html
Bojanowski,, P., Grave,, E., Joulin,, A., & Mikolov,, T. (2017). Enriching word vectors with subword information. TACL, 5, 135–146 https://transacl.org/ojs/index.php/tacl/article/view/999
Boudin,, F. (2016). Pke: An open source python‐based keyphrase extraction toolkit. Paper presented at COLING 2016, 26th international conference on computational linguistics, proceedings of the conference system demonstrations, Osaka, Japan, December 11–16, 2016, 69–73. http://aclweb.org/anthology/C/C16/C16-2015.pdf.
Boudin,, F. (2018). Unsupervised keyphrase extraction with multipartite graphs. Paper presented at proceedings of the 2018 conference of the North American chapter of the association for computational linguistics: Human language technologies, NAACL‐HLT, New Orleans, Louisiana, USA, June 1–6, 2018, Volume 2 (Short Papers), 667–672. https://aclanthology.info/papers/N18-2105/n18-2105.
Boudin,, F., Mougard,, H., & Cram,, D. (2016). How document pre‐processing affects keyphrase extraction performance. Paper presented at proceedings of the 2nd workshop on noisy user‐generated text, NUT@COLING 2016, Osaka, Japan, December 11, 2016, 121–128. https://aclanthology.info/papers/W16-3917/w16-3917.
Bougouin,, A., Boudin,, F., & Daille,, B. (2013). TopicRank: Graph‐based topic ranking for keyphrase extraction. Paper presented at proceedings of the 6th international joint conference on natural language processing, IJCNLP 2013, Nagoya, Japan, October 14–18, 2013, 543–551. http://aclweb.org/anthology/I/I13/I13-1062.pdf.
Bougouin,, A., Boudin,, F., & Daille,, B. (2016). Keyphrase annotation with graph co‐ranking. Paper presented at COLING 2016, 26th international conference on computational linguistics, proceedings of the conference: Technical papers, December 11–16, 2016, Osaka, Japan, 2945–2955. http://aclweb.org/anthology/C/C16/C16-1277.pdf.
Brin,, S., & Page,, L. (1998). The anatomy of a large‐scale hypertextual web search engine. Computer Networks, 30, 107–117. https://doi.org/10.1016/S0169-7552(98)00110-X
Buckley,, C., & Voorhees,, E. M. (2004). Retrieval evaluation with incomplete information. Paper presented at proceedings of the 27th annual international ACM SIGIR conference on research and development in information retrieval, SIGIR 2004, Sheffield, UK, July 25–29, 2004, 25–32. http://doi.acm.org/10.1145/1008992.1009000.
Campos,, R., Mangaravite,, V., Pasquali,, A., Jorge,, A. M., Nunes,, C., & Jatowt,, A. (2018a). Yake! collection‐independent automatic keyword extractor. Paper presented at advances in information retrieval—40th European conference on ir research, ECIR 2018, Grenoble, France, March 26–29, 2018, proceedings, 806–810. https://doi.org/10.1007/978-3-319-76941-7_80.
Campos,, R., Mangaravite,, V., Pasquali,, A., Jorge,, A. M., Nunes,, C., & Jatowt,, A. (2018b). A text feature based automatic keyword extraction method for single documents. Paper presented at advances in information retrieval—40th European conference on IR research, ECIR 2018, Grenoble, France, March 26–29, 2018, proceedings, 684–691. https://doi.org/10.1007/978-3-319-76941-7_63.
Caragea,, C., Bulgarov,, F. A., Godea,, A., & Gollapalli,, S. D. (2014). Citation‐enhanced keyphrase extraction from research papers: A supervised approach. Paper presented at proceedings of the 2014 conference on empirical methods in natural language processing, EMNLP 2014, Doha, Qatar, October 25–29, 2014, a meeting of SIGDAT, a special interest group of the ACL, 1435–1446. http://aclweb.org/anthology/D/D14/D14-1150.pdf.
Caruana,, R. (1993). Multitask learning: A knowledge‐based source of inductive bias. Paper presented at machine learning, proceedings of the tenth international conference, University of Massachusetts, Amherst, MA, USA, June 27–29, 1993, 41–48.
Cer,, D., Yang,, Y., Kong,, S., Hua,, N., Limtiaco,, N., John,, R. S., … Kurzweil,, R. (2018). Universal sentence encoder. CoRR, abs/1803.11175 http://arxiv.org/abs/1803.11175
Chen,, J., Zhang,, X., Wu,, Y., Yan,, Z., & Li,, Z. (2018). Keyphrase generation with correlation constraints. Paper presented at proceedings of the 2018 conference on empirical methods in natural language processing, Brussels, Belgium, October 31–November 4, 2018, 4057–4066. https://aclanthology.info/papers/D18-1439/d18-1439.
Chen,, S. F., & Goodman,, J. (1999). An empirical study of smoothing techniques for language modeling. Computer Speech %26 Language, 13, 359–393. https://doi.org/10.1006/csla.1999.0128
Chuang,, J., Manning,, C. D., & Heer,, J. (2012a). Termite: visualization techniques for assessing textual topic models. Paper presented at proceedings of the international working conference on advanced visual interfaces, AVI 2012, Capri Island, Naples, Italy, May 22–25, 2012, 74–77. http://doi.acm.org/10.1145/2254556.2254572.
Chuang,, J., Manning,, C. D., & Heer,, J. (2012b). “Without the clutter of unimportant words”: Descriptive keyphrases for text visualization. ACM Transactions on Computer‐Human Interaction, 19, 19:1–19:29.
Danesh,, S., Sumner,, T., & Martin,, J. H. (2015). Sgrank: Combining statistical and graphical methods to improve the state of the art in unsupervised keyphrase extraction. Paper presented at proceedings of the fourth joint conference on lexical and computational semantics, Denver, Colorado, USA, June 4–5, 2015, 117–126. http://aclweb.org/anthology/S/S15/S15-1013.pdf.
Deerwester,, S. C., Dumais,, S. T., Landauer,, T. K., Furnas,, G. W., & Harshman,, R. A. (1990). Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41, 391–407. https://doi.org/10.1002/(SICI)1097-4571(199009)41:6〈391::AID-ASI1〉3.0.CO;2-9
Devlin,, J., Chang,, M., Lee,, K., & Toutanova,, K. (2019). BERT: pre‐training of deep bidirectional transformers for language understanding. Paper presented at proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: Human language technologies, NAACL‐HLT 2019, Minneapolis, MN, USA, June 2–7, 2019, Volume 1 (Long and Short Papers), 4171–4186. https://aclweb.org/anthology/papers/N/N19/N19-1423/.
Dice,, L. R. (1945). Measures of the amount of ecologic association between species. Ecology, 26, 297–302. https://doi.org/10.2307/1932409
El‐Beltagy,, S. R., & Rafea,, A. A. (2009). Kp‐miner: A keyphrase extraction system for english and arabic documents. Information Systems, 34, 132–144. https://doi.org/10.1016/j.is.2008.05.002
Elkan,, C., & Noto,, K. (2008). Learning classifiers from only positive and unlabeled data. Paper presented at proceedings of the 14th ACM SIGKDD international conference on knowledge discovery and data mining, Las Vegas, Nevada, USA, August 24–27, 2008, 213–220. http://doi.acm.org/10.1145/1401890.1401920.
El‐Kishky,, A., Song,, Y., Wang,, C., Voss,, C. R., & Han,, J. (2014). Scalable topical phrase mining from text corpora. PVLDB, 8, 305–316 http://www.vldb.org/pvldb/vol8/p305-ElKishky.pdf
Ferragina,, P., & Scaiella,, U. (2010). TAGME: On‐the‐fly annotation of short text fragments (by wikipedia entities). Paper presented at proceedings of the 19th ACM conference on information and knowledge management, CIKM, October 26–30, Toronto, Ontario, Canada, 2010, 1625–1628. https://doi.org/10.1145/1871437.1871689.
Florescu,, C., & Caragea,, C. (2017a). A new scheme for scoring phrases in unsupervised keyphrase extraction. Paper presented at proceedings of the advances in information retrieval—39th European conference on IR research, ECIR 2017, Aberdeen, UK, April 8–13, 2017, 477–483. https://doi.org/10.1007/978-3-319-56608-5_37.
Florescu,, C., & Caragea,, C. (2017b). PositionRank: An unsupervised approach to keyphrase extraction from scholarly documents. Paper presented at proceedings of the 55th annual meeting of the association for computational linguistics, ACL 2017, Vancouver, Canada, July 30–August 4, 2017, Volume 1: Long Papers, 1105–1115. https://doi.org/10.18653/v1/P17-1102.
Gollapalli,, S. D., & Caragea,, C. (2014). Extracting keyphrases from research papers using citation networks. Paper presented at proceedings of the 28th AAAI conference on artificial intelligence, Québec City, Québec, Canada, July 27–31, 2014, 1629–1635. http://www.aaai.org/ocs/index.php/AAAI/AAAI14/paper/view/8662.
Gollapalli,, S. D., Li,, X., & Yang,, P. (2017). Incorporating expert knowledge into keyphrase extraction. Paper presented at proceedings of the 31st AAAI conference on artificial intelligence, San Francisco, California, USA, February 4–9, 2017, 3180–3187. http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14628.
Gutwin,, C., Paynter,, G. W., Witten,, I. H., Nevill‐Manning,, C. G., & Frank,, E. (1999). Improving browsing in digital libraries with keyphrase indexes. Decision Support Systems, 27, 81–104. https://doi.org/10.1016/S0167-9236(99)00038-X
Hasan,, K. S., & Ng,, V. (2010). Conundrums in unsupervised keyphrase extraction: Making sense of the state‐of‐the‐art. Paper presented at proceedings of the 23rd international conference on computational linguistics, COLING 2010, Beijing, China, August 23–27, 2010, Posters Volume, 365–373. http://aclweb.org/anthology/C/C10/C10-2042.pdf.
Hasan,, K. S., & Ng,, V. (2014). Automatic keyphrase extraction: A survey of the state of the art. Paper presented at proceedings of the 52nd annual meeting of the association for computational linguistics, ACL 2014, Baltimore, MD, USA, June 22–27, 2014, Volume 1: Long Papers, 1262–1273. http://aclweb.org/anthology/P/P14/P14-1119.pdf.
Hashimoto,, K., Xiong,, C., Tsuruoka,, Y., & Socher,, R. (2017). A joint many‐task model: Growing a neural network for multiple NLP tasks. Paper presented at proceedings of the 2017 conference on empirical methods in natural language processing, EMNLP 2017, Copenhagen, Denmark, September 9–11, 2017, 1923–1933. https://aclanthology.info/papers/D17-1206/d17-1206.
Haveliwala,, T. H. (2002). Topic‐sensitive pagerank. Paper presented at proceedings of the eleventh international World Wide Web conference, WWW 2002, Honolulu, Hawaii, May 7–11, 2002, 517–526. http://doi.acm.org/10.1145/511446.511513.
Herings,, P. J.‐J., Laan,, G. V. D., & Talman,, D. (2005). Measuring the power of nodes in digraphs. Social Choice and Welfare, 24, 439–454. https://doi.org/10.1007/s00355-003-0308-9
Hulth,, A. (2003). Improved automatic keyword extraction given more linguistic knowledge. Paper presented at proceedings of the 2003 conference on empirical methods in natural language processing, EMNLP 2003, Stroudsburg, PA, USA, 2003, 216–223. Association for computational linguistics. https://doi.org/10.3115/1119355.1119383.
Hulth,, A., & Megyesi,, B. (2006). A study on automatically extracted keywords in text categorization. Paper presented at ACL 2006, 21st international conference on computational linguistics and 44th annual meeting of the association for computational linguistics, proceedings of the conference, Sydney, Australia, 17–21 July 2006. http://aclweb.org/anthology/P06-1068.
Jiang,, X., Hu,, Y., & Li,, H. (2009). A ranking approach to keyphrase extraction. Paper presented at proceedings of the 32nd annual international ACM SIGIR conference on research and development in information retrieval, SIGIR 2009, Boston, MA, USA, July 19–23, 2009, 756–757. http://doi.acm.org/10.1145/1571941.1572113.
Kim,, S. N., Baldwin,, T., & Kan,, M. (2010). Evaluating n‐gram based evaluation metrics for automatic keyphrase extraction. Paper presented at proceedings of the 23rd international conference on computational linguistics, COLING 2010, Beijing, China, August 23–27, 2010, 572–580. http://aclweb.org/anthology/C10-1065.
Kim,, S. N., & Kan,, M. (2009). Re‐examining automatic keyphrase extraction approaches in scientific articles. Paper presented at proceedings of the workshop on multiword expressions: identification, interpretation, disambiguation and applications, MWE@IJCNLP 2009, Singapore, August 6, 2009, 9–16. https://aclanthology.info/papers/W09-2902/w09-2902.
Kim,, S. N., Medelyan,, O., Kan,, M., & Baldwin,, T. (2010). Semeval‐2010 task 5: Automatic keyphrase extraction from scientific articles. Paper presented at proceedings of the 5th international workshop on semantic evaluation, SemEval@ACL 2010, Uppsala, Sweden, July 15–16, 2010, 21–26. http://aclweb.org/anthology/S/S10/S10-1004.pdf.
Kim,, S. N., Medelyan,, O., Kan,, M., & Baldwin,, T. (2013). Automatic keyphrase extraction from scientific articles. Language Resources and Evaluation, 47, 723–742. https://doi.org/10.1007/s10579-012-9210-3
Kleinberg,, J. M. (1999). Authoritative sources in a hyperlinked environment. Journal of the ACM, 46, 604–632.
Krapivin,, M., Autayeu,, A., & Marchese,, M. (2008). Large dataset for keyphrases extraction. Technical Report DISI‐09‐055. Trento, Italy.
Langville,, A. N., & Meyer,, C. D. (2003). Survey: Deeper inside pagerank. Internet Mathematics, 1, 335–380. https://doi.org/10.1080/15427951.2004.10129091
Lau,, J. H., & Baldwin,, T. (2016). An empirical evaluation of doc2vec with practical insights into document embedding generation. Paper presented at proceedings of the 1st workshop on representation learning for NLP, Rep4NLP@ACL 2016, Berlin, Germany, August 11, 2016, 78–86. https://doi.org/10.18653/v1/W16-1609.
Liu,, L., & Özsu,, M. T. (Eds.). (2009). Encyclopedia of database systems. New York, NY: Springer. https://doi.org/10.1007/978-0-387-39940-9
Liu,, Z., Huang,, W., Zheng,, Y., & Sun,, M. (2010). Automatic keyphrase extraction via topic decomposition. Paper presented at proceedings of the 2010 conference on empirical methods in natural language processing, EMNLP 2010, MIT Stata Center, Massachusetts, USA, October 9–11, 2010, A meeting of SIGDAT, a special interest group of the ACL, 366–376. http://www.aclweb.org/anthology/D10-1036.
Liu,, Z., Li,, P., Zheng,, Y., & Sun,, M. (2009). Clustering to find exemplar terms for keyphrase extraction. Paper presented at proceedings of the 2009 conference on empirical methods in natural language processing, EMNLP 2009, Singapore, August 6–7, 2009, A meeting of SIGDAT, a special interest group of the ACL, 257–266. http://www.aclweb.org/anthology/D09-1027.
Mahata,, D., Kuriakose,, J., Shah,, R. R., & Zimmermann,, R. (2018). Key2vec: Automatic ranked keyphrase extraction from scientific articles using phrase embeddings. Paper presented at proceedings of the 2018 conference of the North American chapter of the association for computational linguistics: Human language technologies, NAACL‐HLT, New Orleans, Louisiana, USA, June 1–6, 2018, Volume 2 (Short Papers), 634–639. https://aclanthology.info/papers/N18-2100/n18-2100.
Marujo,, L., Gershman,, A., Carbonell,, J. G., Frederking,, R. E., & Neto,, J. P. (2012). Supervised topical key phrase extraction of news stories using crowdsourcing, light filtering and coreference normalization. Paper presented at proceedings of the eighth international conference on language resources and evaluation, LREC 2012, Istanbul, Turkey, May 23–25, 2012, 399–403. http://www.lrec-conf.org/proceedings/lrec2012/summaries/672.html.
Marujo,, L., Viveiros,, M., & Neto,, J. P. (2011). Keyphrase cloud generation of broadcast news. Paper presented at proceedings of the INTERSPEECH 2011, 12th annual conference of the international speech communication association, Florence, Italy, August 27–31, 2011, 2393–2396. http://www.isca-speech.org/archive/interspeech_2011/i11_2393.html.
McIlraith,, S. A., & Weinberger,, K. Q. (2018). Learning feature representations for keyphrase extraction. Paper presented at proceedings of the thirty‐second AAAI conference on artificial intelligence, New Orleans, Louisiana, USA, February 2–7, 2018. https://www.aaai.org/ocs/index.php/AAAI/AAAI18/schedConf/presentations.
Medelyan,, O., Frank,, E., & Witten,, I. H. (2009). Human‐competitive tagging using automatic keyphrase extraction. Paper presented at proceedings of the 2009 conference on empirical methods in natural language processing, EMNLP 2009, Singapore, August 6–7, 2009, A meeting of SIGDAT, a special interest group of the ACL, 1318–1327. http://www.aclweb.org/anthology/D09-1137.
Meng,, R., Zhao,, S., Han,, S., He,, D., Brusilovsky,, P., & Chi,, Y. (2017). Deep keyphrase generation. Paper presented at proceedings of the 55th annual meeting of the association for computational linguistics, ACL 2017, Vancouver, Canada, July 30–August 4, 2017, Volume 1: Long Papers, 582–592. https://doi.org/10.18653/v1/P17-1054.
Merity,, S., Keskar,, N. S., & Socher,, R. (2018). An analysis of neural language modeling at multiple scales. CoRR, abs/1803.08240 http://arxiv.org/abs/1803.08240
Mihalcea,, R., & Tarau,, P. (2004). TextRank: Bringing order into text. Paper presented at proceedings of the 2004 conference on empirical methods in natural language processing, EMNLP 2004, Barcelona, Spain, July 25–26, 2004, A meeting of SIGDAT, a special interest group of the ACL, held in conjunction with ACL 2004, 404–411. http://www.aclweb.org/anthology/W04-3252.
Mikolov,, T., Chen,, K., Corrado,, G., & Dean,, J. (2013). Efficient estimation of word representations in vector space. In proceedings of the 2013 international conference on learning representations, ICLR 2013, workshop track, Scottsdale, Arizona, USA, May 2–4, 2013. http://arxiv.org/abs/1301.3781.
Miwa,, M., & Bansal,, M. (2016). End‐to‐end relation extraction using lstms on sequences and tree structures. Paper presented at proceedings of the 54th annual meeting of the association for computational linguistics, ACL 2016, August 7–12, 2016, Berlin, Germany, Volume 1: Long Papers. http://aclweb.org/anthology/P/P16/P16-1105.pdf.
Nguyen,, T. D., & Kan,, M. (2007). Keyphrase extraction in scientific publications. Paper presented at proceedings of the Asian digital libraries. Looking back 10 years and forging new frontiers, 10th international conference on Asian digital libraries, ICADL 2007, Hanoi, Vietnam, December 10–13, 2007, 317–326. https://doi.org/10.1007/978-3-540-77094-7_41.
Nguyen,, T. D., & Luong,, M. (2010). WINGNUS: Keyphrase extraction utilizing document logical structure. Paper presented at proceedings of the 5th international workshop on semantic evaluation, SemEval@ACL 2010, Uppsala, Sweden, July 15–16, 2010, 166–169. http://aclweb.org/anthology/S/S10/S10-1035.pdf.
Pagliardini,, M., Gupta,, P., & Jaggi,, M. (2018). Unsupervised learning of sentence embeddings using compositional n‐gram features. Paper presented at proceedings of the 2018 conference of the North American chapter of the association for computational linguistics: human language technologies, NAACL‐HLT 2018, New Orleans, Louisiana, USA, June 1–6, 2018, Volume 1 (Long Papers), 528–540. https://aclanthology.info/papers/N18-1049/n18-1049.
Papagiannopoulou,, E., & Tsoumakas,, G. (2018). Local word vectors guiding keyphrase extraction. Information Processing %26 Management, 54, 888–902. https://doi.org/10.1016/j.ipm.2018.06.004
Pennington,, J., Socher,, R., & Manning,, C. D. (2014). Glove: Global vectors for word representation. Paper presented at proceedings of the 2014 conference on empirical methods in natural language processing, EMNLP 2014, Doha, Qatar, October 25–29, 2014, a meeting of SIGDAT, a special interest group of the ACL, 1532–1543. http://aclweb.org/anthology/D/D14/D14-1162.pdf.
Rose,, S., Engel,, D., Cramer,, N., & Cowley,, W. (2010). Automatic keyword extraction from individual documents. In Text mining: Applications and theory (pp. 1–20). John Wiley and Sons. https://doi.org/10.1002/9780470689646.ch1
Rousseau,, F., & Vazirgiannis,, M. (2015). Main core retention on graph‐of‐words for single‐document keyword extraction. Paper presented at proceedings of the advances in information retrieval—37th European conference on IR research, ECIR 2015, Vienna, Austria, March 29–April 2, 2015, 382–393. https://doi.org/10.1007/978-3-319-16354-3_42.
Rush,, A. M., Chopra,, S., & Weston,, J. (2015). A neural attention model for abstractive sentence summarization. Paper presented at proceedings of the 2015 conference on empirical methods in natural language processing, EMNLP 2015, Lisbon, Portugal, September 17–21, 2015, 379–389. http://aclweb.org/anthology/D/D15/D15-1044.pdf.
Shi,, T., Jiao,, S., Hou,, J., & Li,, M. (2008). Improving keyphrase extraction using wikipedia semantics. Paper presented at proceedings of second international symposium on intelligent information technology application, 2008. IITA`08, vol. 2, 42–46. IEEE.
Shi,, W., Zheng,, W., Yu,, J. X., Cheng,, H., & Zou,, L. (2017). Keyphrase extraction using knowledge graphs. Data Science and Engineering, 2, 275–288. https://doi.org/10.1007/s41019-017-0055-z
Søgaard,, A., & Bingel,, J. (2017). Identifying beneficial task relations for multi‐task learning in deep neural networks. Paper presented at proceedings of the 15th conference of the European chapter of the association for computational linguistics, EACL 2017, Valencia, Spain, April 3–7, 2017, Volume 2: Short Papers, 164–169. https://aclanthology.info/papers/E17-2026/e17-2026.
Song,, M., Song,, I., Allen,, R. B., & Obradovic,, Z. (2006). Keyphrase extraction‐based query expansion in digital libraries. Paper presented at ACM/IEEE joint conference on digital libraries, JCDL 2006, Chapel Hill, NC, USA, June 11–15, 2006, proceedings, 202–209. https://doi.org/10.1145/1141753.1141800.
Sterckx,, L., Caragea,, C., Demeester,, T., & Develder,, C. (2015). When topic models disagree: Keyphrase extraction with multiple topic models. Paper presented at proceedings of the 24th international conference on World Wide Web companion, WWW 2015, Florence, Italy, May 18–22, 2015—Companion Volume, 123–124. http://doi.acm.org/10.1145/2740908.2742731.
Sterckx,, L., Caragea,, C., Demeester,, T., & Develder,, C. (2016). Supervised keyphrase extraction as positive unlabeled learning. Paper presented at proceedings of the 2016 conference on empirical methods in natural language processing, EMNLP 2016, Austin, Texas, USA, November 1–4, 2016, 1924–1929. http://aclweb.org/anthology/D/D16/D16-1198.pdf.
Sterckx,, L., Caragea,, C., Demeester,, T., & Develder,, C. (2018). Creation and evaluation of large keyphrase extraction collections with multiple opinions. Language Resources and Evaluation, 52, 503–532. https://doi.org/10.1007/s10579-017-9395-6
Sterckx,, L., Demeester,, T., Deleu,, J., & Develder,, C. (2015). Topical word importance for fast keyphrase extraction. Paper presented at proceedings of the 24th international conference on World Wide Web companion, WWW 2015, Florence, Italy, May 18–22, 2015—Companion Volume, 121–122. http://doi.acm.org/10.1145/2740908.2742730.
Stubbs,, M. (2003). Two quantitative methods of studying phraseology in english. International Journal of Corpus Linguistics, 7, 215–244.
Teneva,, N., & Cheng,, W. (2017). Salience rank: Efficient keyphrase extraction with topic modeling. Paper presented at proceedings of the 55th annual meeting of the association for computational linguistics, ACL 2017, Vancouver, Canada, July 30–August 4, Volume 2: Short Papers, 530–535. https://doi.org/10.18653/v1/P17-2084.
Tomokiyo,, T., & Hurst,, M. (2003). A language model approach to keyphrase extraction. Paper presented at proceedings of the 2003 ACL workshop on multiword expressions: Analysis, acquisition and treatment, MWE 2003, Sapporo, Japan, 2003, Volume—18, 33–40. https://doi.org/10.3115/1119282.1119287.
Tu,, Z., Lu,, Z., Liu,, Y., Liu,, X., & Li,, H. (2016). Modeling coverage for neural machine translation. Paper presented at proceedings of the 54th annual meeting of the association for computational linguistics, ACL 2016, August 7–12, 2016, Berlin, Germany, Volume 1: Long Papers. http://aclweb.org/anthology/P/P16/P16-1008.pdf.
Turney,, P. D. (2001). Mining the web for synonyms: PMI‐IR versus LSA on TOEFL. Paper presented at proceedings of the 12th european conference on machine learning: ECML 2001, Freiburg, Germany, September 5–7, 2001, 491–502. https://doi.org/10.1007/3-540-44795-4_42.
Voorhees,, E. M. (1999). The TREC‐8 question answering track report. Paper presented at proceedings of the eighth text retrieval conference, TREC 1999, Gaithersburg, Maryland, USA, November 17–19, 1999. http://trec.nist.gov/pubs/trec8/papers/qa_report.pdf.
Wan,, X., & Xiao,, J. (2008). Single document keyphrase extraction using neighborhood knowledge. Paper presented at proceedings of the 23rd AAAI conference on artificial intelligence, AAAI 2008, Chicago, Illinois, USA, July 13–17, 2008, 855–860. http://www.aaai.org/Library/AAAI/2008/aaai08-136.php.
Wang,, L., & Li,, S. (2017). Pku_icl at semeval‐2017 task 10: Keyphrase extraction with model ensemble and external knowledge. Paper presented at proceedings of the 11th international workshop on semantic evaluation, SemEval@ACL 2017, Vancouver, Canada, August 3–4, 2017, 934–937. https://doi.org/10.18653/v1/S17-2161.
Wang,, R., Liu,, W., & McDonald,, C. (2014). Corpus‐independent generic keyphrase extraction using word embedding vectors. Paper presented at software engineering research conference, 39.
Wang,, R., Liu,, W., & McDonald,, C. (2015). Using word embeddings to enhance keyword identification for scientific publications. Paper presented at proceedings of the databases theory and applications—26th Australasian database conference, ADC 2015, Melbourne, VIC, Australia, June 4–7, 2015, 257–268. https://doi.org/10.1007/978-3-319-19548-3_21.
Wang,, Y., Liu,, Q., Qin,, C., Xu,, T., Wang,, Y., Chen,, E., & Xiong,, H. (2018). Exploiting topic‐based adversarial neural network for cross‐domain keyphrase extraction. Paper presented at proceedings of the IEEE international conference on data mining, ICDM 2018, Singapore, November 17–20, 2018, 597–606. https://doi.org/10.1109/ICDM.2018.00075.
Witten,, I. H., Paynter,, G. W., Frank,, E., Gutwin,, C., & Nevill‐Manning,, C. G. (1999). KEA: Practical automatic keyphrase extraction. Paper presented at proceedings of the 4th ACM conference on digital libraries, Berkeley, CA, USA, August 11–14, 1999, 254–255. http://doi.acm.org/10.1145/313238.313437.
Won,, M., Martins,, B., & Raimundo,, F. (2019). Automatic extraction of relevant keyphrases for the study of issue competition. Paper presented at proceedings of the 20th international conference on computational linguistics and intelligent text processing, Berkeley, La Rochelle, France, April 7‐13, 2019.
Yang,, M., Liang,, Y., Zhao,, W., Xu,, W., Zhu,, J., & Qu,, Q. (2018). Task‐oriented keyphrase extraction from social media. Multimedia Tools and Applications, 77, 3171–3187. https://doi.org/10.1007/s11042-017-5041-y
Yang,, Z., Dai,, Z., Yang,, Y., Carbonell,, J. G., Salakhutdinov,, R., & Le,, Q. V. (2019). Xlnet: Generalized autoregressive pretraining for language understanding. CoRR, abs/1906.08237 http://arxiv.org/abs/1906.08237
Ye,, H., & Wang,, L. (2018). Semi‐supervised learning for neural keyphrase generation. Paper presented at proceedings of the 2018 conference on empirical methods in natural language processing, Brussels, Belgium, October 31–November 4, 2018, 4142–4153. https://aclanthology.info/papers/D18-1447/d18-1447.
Yu,, Y., & Ng,, V. (2018). Wikirank: Improving keyphrase extraction based on background knowledge. Paper presented at proceedings of the 11th edition of the language resources and evaluation conference, LREC 2018, 7–12 May 2018, Miyazaki (Japan), 3723–3727. http://www.lrec-conf.org/proceedings/lrec2018/pdf/871.pdf.
Zesch,, T., & Gurevych,, I. (2009). Approximate matching for evaluating keyphrase extraction. Paper presented at proceedings of the recent advances in natural language processing, RANLP, Borovets, Bulgaria, 14–16 September, 2009, 484–489. http://aclweb.org/anthology/R/R09/R09-1086.pdf.
Zhang,, Q., Wang,, Y., Gong,, Y., & Huang,, X. (2016). Keyphrase extraction using deep recurrent neural networks on twitter. Paper presented at proceedings of the 2016 conference on empirical methods in natural language processing, EMNLP 2016, Austin, Texas, USA, November 1–4, 2016, 836–845. http://aclweb.org/anthology/D/D16/D16-1080.pdf.
Zhang,, Y., Chang,, Y., Liu,, X., Gollapalli,, S. D., Li,, X., & Xiao,, C. (2017). MIKE: Keyphrase extraction by integrating multidimensional information. Paper presented at proceedings of the 2017 acm on conference on information and knowledge management, CIKM 2017, Singapore, November 06–10, 2017, 1349–1358. http://doi.acm.org/10.1145/3132847.3132956.
Zhang,, Y., Zincir‐Heywood,, A. N., & Milios,, E. E. (2004). World wide web site summarization. Web Intelligence and Agent Systems, 2, 39–53 http://content.iospress.com/articles/web-intelligence-and-agent-systems-an-international-journal/wia00026

Further Reading

Agarwal,, A., & Lavie,, A. (2008). Meteor, M‐BLEU and M‐TER: Evaluation metrics for high‐correlation with human rankings of machine translation output. Paper presented at proceedings of the third workshop on statistical machine translation, WMT@ACL 2008, Columbus, OH, June 19, 2008, 115–118. https://aclanthology.info/papers/W08-0312/w08-0312.
Figueroa,, G., Chen,, P., & Chen,, Y. (2018). Rankup: Enhancing graph‐based keyphrase extraction methods with error‐feedback propagation. Computer Speech %26 Language, 47, 112–131. https://doi.org/10.1016/j.csl.2017.07.004
Lin,, C., & Hovy,, E. H. (2003). Automatic evaluation of summaries using n‐gram co‐occurrence statistics. Paper presented at proceedings of the human language technology conference of the North American chapter of the association for computational linguistics, HLT‐NAACL 2003, Edmonton, Canada, May 27–June 1, 2003. http://aclweb.org/anthology/N/N03/N03-1020.pdf.
Moldovan,, D. I., Harabagiu,, S. M., Pasca,, M., Mihalcea,, R., Girju,, R., Goodrum,, R., & Rus,, V. (2000). The structure and performance of an open‐domain question answering system. Paper presented at 38th annual meeting of the association for computational linguistics, Hong Kong, China, October 1–8, 2000. http://www.aclweb.org/anthology/P00-1071.
Papineni,, K., Roukos,, S., Ward,, T., & Zhu,, W. (2002). Bleu: A method for automatic evaluation of machine translation. Paper presented at proceedings of the 40th annual meeting of the association for computational linguistics, Philadelphia, PA, USA, July 6–12, 2002, 311–318. http://www.aclweb.org/anthology/P02-1040.pdf.
Przybocki,, M. A., & Martin,, A. F. (1999). The 1999 NIST speaker recognition evaluation, using summed two‐channel telephone data for speaker detection and speaker tracking. Paper presented at proceedings of the sixth European conference on speech communication and technology, EUROSPEECH 1999, Budapest, Hungary, September 5–9, 1999. http://www.isca-speech.org/archive/eurospeech_1999/e99_2215.html.
Witten,, I. H. (2003). Browsing around a digital library. Paper presented at proceedings of the 14th annual ACM‐SIAM symposium on discrete algorithms, Baltimore, Maryland, USA, January 12–14, 2003, 99–99. http://dl.acm.org/citation.cfm?id=644108.644125.

Access to this WIREs title is by subscription only.

Recommend to Your
Librarian Now!

The latest WIREs articles in your inbox

Sign Up for Article Alerts