Why Meta’s Great Language Model Doesn’t Work for Researchers

Couldn’t attend Transform 2022? Discover all the summit sessions now in our on-demand library! Look here.

When Alan Turing invented the Turing test in 1950, it was a test of a machine’s ability to exhibit intelligent behavior indistinguishable from that of a human. Turing proposed that a computer can be considered to possess artificial intelligence (AI) if it can create human-like answers to questions.

Thanks to large language models, we’re now at the point where computers can write text about just about any topic we give them – and for the most part, it’s very persuasive and human-like.

Tell him to write a sentence about “Why does Elon Musk like to knit?” and what it produces is arguably as good as any human could write:

Some possible reasons why Elon Musk might enjoy knitting could include the fact that it is a relaxing and meditative activity that can help to clear one's mind, and it also allows for a great deal of creativity and self-expression.
Additionally, knitting can be a very social activity, and Elon Musk may enjoy the opportunity to chat and connect with other knitters.

[Source: OpenAI Playground using text-davinci-002 model]

Summarize complex text

Examples like this are fun, but the biggest value proposition of using big language models is less about writing goofy prose and more about synthesizing complex text. These use cases are exciting across industries. For example, AI can distill information about potential prospects for business intelligence purposes, or it can summarize investment documents in finance.


MetaBeat 2022

MetaBeat will bring together thought leaders to advise on how metaverse technology will transform the way all industries communicate and do business on October 4 in San Francisco, California.

register here

However, what particularly interests me is summarizing scientific articles for researchers.

The problem is that there is an overload of research published all over the world. More than 4.2 million academic research papers were published in 2020. Even in specialized research areas, hundreds of papers are published each year – how can a researcher stay current while pursuing their own searches? The abstract of an article only hints at the details of the research it contains.

When Meta recently opened up its language model, OPT-175B, it looked promising to academic researchers. It is said to offer better performance than OpenAI’s GPT-3 and only uses 15% of GPT-3’s computational resources to train it.

Put models to the test

So how does Meta work on scientific research?

We first tested it on a technical press release, and here is the summary we received:

​​Forward Looking Statements, which speak only as of the date of this press release. Artelo undertakes no obligation to publicly update any forward-looking statement, whether as a result of new information, future events or otherwise.
Investor Relations Contact:
Westwicke Partners on behalf of Artelo Biosciences
Mike Piccinino, CFA

It’s not a good start. The model focuses on the investor’s legalese at the end of the press release, calculating that this is the most important information the reader needs to know. You might argue that’s important, but that’s not what we’re looking for. The contact with the investor is not even correct, it was fabricated by the model!

Then we tried it on an article from the Journal of Electronic Materials:

Journal of ELECTRONIC MATERIALS, Vol. 38, No. 7, 2009 DOI: 10.1007/s11664-009-0701-y (cid:1) 2009 TMS Special Issue Paper thermally. The samples were then pulverized and analyzed by XRD, TGA, and XPS. The XRD patterns showed that CaCo2O4+d crystallizes as a calcium-ferrite-type structure, which consists of a corner- and edge- shared CoO6 octahedron network including one-dimensional double chains. The CaCo2O4+d phase has a finite density of states at the Fermi level. The XPS results showed that CaCo2O4+d has a composition of CaCo2O4+d = (Ca2CoO3)0.62CoO2. The XPS results also showed that CaCo2O4+d has

[Source: Meta OPT-175B]

At first glance, it appears to contain important information, but there is clearly superfluous information such as article details that are not important to the summary, and I would describe the result more accurately by paraphrasing some of the text instead that summarize all aspects.

Small-scale models perform better

So how does a smaller language model compare? Thanks to our experience in the field, we have designed our IPG Iris.ai model to have only 25 million parameters – compared to 175 billion for Meta and OpenAI – but we have trained it on millions of scientific articles specific to a domain. Our research has shown that this model works very differently on the same paper:

Metallic temperature dependence of the seebeck coefficient s with a large thermoelectric power (s=151lv/kat387k) has a finite density of states at the fermi level. In this paper, we report the crystal structure and physical properties of caco2o4+d. We find a new compound caco2o4+d, which exhibits a large thermoelectric power, even though it has a finite density of states at the fermi level. Motivated by the simple guiding principle mentioned previously, we searched for new phases thermoelectric properties related as a thermoelectric material applicable to high-temperature use.

[Source: Iris.ai IPG]

You can see that the sentence structure is slightly more simplistic than a large language model, but the information is much more relevant. Additionally, the computational costs to generate this news article summary are less than $0.23. Doing the same on OPT-175 would cost around $180.

AI models container ships

You would assume that large language models backed by enormous computing power, such as the OPT-175B, would be able to process the same information faster and with higher quality. But where the model fails is in knowledge of a specific domain. He doesn’t understand the structure of a research paper, he doesn’t know what information is important, and he doesn’t understand chemical formulas. It’s not the model’s fault – he just wasn’t trained on this information.

So the solution is to just train the GPT model on paper materials, right?

To some extent, yes. If we can train a GPT model on material papers, it will do a good job of summarizing them, but large language models are – by nature – large. These are the proverbial container ships of AI models — it’s very difficult to change their direction. This means that scaling the model with reinforcement learning requires hundreds of thousands of paper documents. And that’s a problem – that volume of items just doesn’t exist to train the model. Yes, data can be fabricated (as is often the case in AI), but this reduces the quality of the outputs – the strength of GPT comes from the variety of data it is trained on.

Revolutionizing the “how”

This is why smaller language models work better. Natural language processing (NLP) has been around for years, and while GPT models have grabbed the headlines, the sophistication of smaller NLP models just keeps getting better.

After all, a model trained on 175 billion parameters will still be difficult to manage, but a model using 30-40 million parameters is much more manageable for domain-specific text. The added benefit is that it will use less computing power, so it also costs a lot less to run.

From the point of view of scientific research, which interests me the most, AI will accelerate the potential of researchers, both in academia and in industry. The current pace of publication produces an unattainable amount of research, draining academic time and corporate resources.

The way we designed the Iris.ai IPG model reflects my belief that certain models offer the opportunity to not only revolutionize what we study or how quickly we study it, but also How? ‘Or’ What we approach different disciplines of scientific research as a whole. They give talented minds a lot more time and resources to collaborate and generate value.

This potential for each researcher to exploit global research drives me forward.

Victor Botev is the technical director of Iris AI.


Welcome to the VentureBeat community!

DataDecisionMakers is where experts, including data technicians, can share data insights and innovations.

If you want to learn more about cutting-edge insights and up-to-date information, best practices, and the future of data and data technology, join us at DataDecisionMakers.

You might even consider writing your own article!

Learn more about DataDecisionMakers

Leave a Reply

Your email address will not be published.