New research center | Hochreiter, the pioneer of artificial intelligence: Investors from “Saudi Arabia or China” came knocking on the door

Joseph “Sepp” Hochreiter is a man in demand worldwide. In the early 1990s, the Bavarian native began researching how “dynamic neural networks” worked, writing a thesis on the topic and developing the idea of ​​long-term neural memory for the first time. Later, it gained great importance as “long-term memory (LSTM)”. Used by Google for speech recognition on smartphones or Google Translate. Apple used LSTM early on with Siri, and Amazon with Alexa.

When OpenAI unleashed its chatbot ChatGPT on humanity at the end of 2022 and suddenly made AI socially acceptable, Hochreiter, who is currently head of the Machine Learning Institute at Linz University, seemed upset in the conversation. “Basically, the technology behind ChatGPT is trivial,” he told the Klein Zeitung newspaper about a year ago. Hochreiter even recreated and improved the first OpenAI models himself. However, at some point, the company's computing power, fueled by Microsoft's billions, increased immeasurably. Sepp Hochreiter in February 2023: “Now we no longer stand a chance. We simply don't have that computational and data power. OpenAI has greatly expanded the scope of this.”

With memory against transformers

Today, the researcher feels hopeful again. This is again related to its “long and short-term memory” – and to shortcomings in the technology used by OpenAI, Meta, Alphabet & Co. But first things first.

Let's start with the technology that popular language learning models currently rely on – whether it's from OpenAI, Meta, Google, or Mistral. If new words are generated, the number of computation steps required in so-called “transformer models” increases quadratically.

“So, if a user enters 1,000 words into ChatGPT, the model needs 1 million steps for the next word,” he wrote. German Handelsblatt With reference to the Institute for Industrial Manufacturing and Plant Operations at the University of Stuttgart. Each query requires a new calculation. This leads to a significant increase in energy demand. The International Energy Agency calculated that it takes 2.9 watt-hours of electricity to answer the ChatGPT question. Ten times more than what Google search uses.

“Same performance with fewer computers”

This is where Sepp Hochreiter comes in. He works on technology that makes AI models “faster and more efficient.” In the best-case scenario, which must be proven first, Hochreiter's model is also able to “better recognize complex connections.” Technologically, the models being worked on in Linz are based on the same LSTM model that Hochreiter made famous. Naturally, the researcher has continued to develop it further in recent years. “The core technology is LSTM, but a lot of new findings and techniques are coming together,” Hochreiter says in an interview.

With xLSTM, calculations will only increase linearly with text length. Hochreiter: “We offer the same service, but you pay less because you need fewer computers.” The idea of ​​interacting with storage is central to this.

Stephane Perrier enters

Particularly interesting: in order to develop the models, a new startup was founded, which is a combination of science and business. Sepp Hochreiter now owns 26% of NXAI, based at a tobacco factory in Linz, together with Pierer Digital Holding and Netural.

“I'm sitting on something cool here in Linz, but I don't have the money to do it,” Sepp Hochreiter said recently. He was then in touch with several potential investors. “From Germany, China or Saudi Arabia,” Hochreiter says. However, many were very interested in the result and wanted to claim a product that would soon expire. It's different with current donors, says Hochreiter, who sees a “Silicon Valley” attitude. First of all, investment will be made in “technology research and development”.

Will €300 million actually flow into the startup, as was rumored? Hochreiter puts it in perspective. These are the costs that arise when you “build a really big model”. At NXAI, investments are now made “peu à peu”, i.e. step by step. However, Pierer Digital Holding is talking about a “significant amount” in the millions.