{"id":6501,"date":"2019-10-07T12:46:59","date_gmt":"2019-10-07T12:46:59","guid":{"rendered":"https:\/\/gtechbooster.com\/?p=6501"},"modified":"2023-04-01T01:36:48","modified_gmt":"2023-04-01T01:36:48","slug":"facebook-open-sources-roberta-nlp-model","status":"publish","type":"post","link":"https:\/\/gtechbooster.com\/facebook-open-sources-roberta-nlp-model\/","title":{"rendered":"Facebook Open Sources RoBERTA NLP Model"},"content":{"rendered":"\n

Facebook has made a new natural language processing model called \nRoBERTA available as open source. The model is an optimized version of \nGoogle’s BERT model.<\/p>\n\n\n\n

\"\"<\/a><\/div><\/div>

The Facebook researchers describe their model as a robustly optimized method for pretraining natural language processing (NLP) systems that improves on Bidirectional Encoder Representations from Transformers, or BERT, the self-supervised method released by Google in 2018.<\/p>\n\n\n\n

BERT has become know for the impressive results the technique has \nachieved on a range of NLP tasks while relying on un-annotated text \ndrawn from the web. Most similar NLP systems are based on text that has \nbeen labeled specifically for a given task.<\/p>\n\n\n\n

Facebook’s new optimized method, RoBERTa, produces state-of-the-art \nresults on the widely used NLP benchmark, General Language Understanding\n Evaluation (GLUE).<\/p>\n\n\n\n

RoBERTa has been implemented in PyTorch, and the team modified key \nhyperparameters in BERT, including removing BERT\u2019s next-sentence \npretraining objective. RoBERTa was also trained with much larger \nmini-batches and learning rates. The developers say this allows RoBERTa \nto improve on the masked language modeling objective compared with BERT \nand leads to better downstream task performance.<\/p>\n\n\n\n

After implementing these design changes, the Facebook model showed \nnotably better performance on the MNLI, QNLI, RTE, STS-B, and RACE tasks\n and a sizable performance improvement on the GLUE benchmark. With a \nscore of 88.5, RoBERTa reached the top position on the GLUE leaderboard,\n matching the performance of the previous leader, XLNet-Large. The team \nsays these results highlight the importance of previously unexplored \ndesign choices in BERT training and help disentangle the relative \ncontributions of data size, training time, and pretraining objectives.<\/p>\n\n\n\n

There’s a full description of RoBERTA and the research carried out in a paper published on arXiv.<\/p>\n\n\n\n

More Information<\/h2>\n\n\n\n