After a year of rapid and furious development in generative AI, the industry is at a crossroads. This technology is poised to deliver unprecedented productivity gains, but may be hampered by technology limitations and usage guardrails.
These are the words of Stanford University's Human-Centered AI Institute (HAI), which just released the 2024 edition of its annual AI Index. Interestingly, the study authors observed that while AI continues to become mainstream, the wave of investment and the need for AI development skills is fading.
Certain realities seem to prevail in the business world. Although AI is a powerful tool, it is far from being able to seamlessly take over most tasks.
“AI can beat humans at some tasks, but not at all,” the report's authors said. “AI outperformed humans on several benchmarks, including image classification, visual reasoning, and English comprehension. However, it outperformed humans on more complex tasks, such as competitive level math, visual common sense reasoning, and planning. I’m falling behind.”
There is no doubt that AI has become smarter and more powerful over the past 12 months. At the same time, the cost of building and maintaining large-scale language models (LLMs) has increased astronomically. Additionally, the industry still lacks standards for responsible AI best practices.
According to the report, the number of new large-scale language models being released worldwide in 2023 has doubled from the previous year. “Two-thirds were open source, but the best-performing models were from industry players using closed systems.
Gemini Ultra became the first LLM to reach human-level performance on the Massive Multitask Language Understanding (MMLU) benchmark. And benchmark performance improved by 15 percentage points over last year.
The HAI authors also observed that LLM prices have also increased significantly. “For example, OpenAI's GPT-4 used an estimated $78 million worth of computing for training, while Google's Gemini Ultra costs $191 million for computing,” they estimate.
At the same time, investment in generative AI has skyrocketed over the past 12 months. “Funding for generative AI has jumped 8x since 2022, reaching $25.2 billion. Major companies in the generative AI space, including OpenAI, Anthropic, Hugging Face, and Inflection, have raised large funding rounds. I reported it.”
The report also suggests that those working to design, build, and implement AI systems need to be more open about their methods. “AI developers score low on transparency,” the co-authors suggest. “This is especially true when it comes to disclosing training data and methodologies.” This lack of openness is hampering efforts to better understand the robustness and safety of AI systems. ”
Responsible AI is still an open and incomplete endeavor. “There is a significant lack of robust and standardized assessments of LLM responsibilities,” the HAI authors report. “There is a significant lack of standardization in responsible AI reporting. Major developers such as OpenAI, Google, and Anthropic primarily test their models against various responsible AI benchmarks. This practice It complicates efforts to systematically compare the risks and limitations of top AI models.”
It's no wonder, then, that the number of AI regulations has exploded in the United States. In 2023, there will be 25 AI-related regulations, up from 1 in 2016. Over the past year, the total number of AI-related regulations has increased by 56%. Regulations come from the U.S. Department of Transportation, Department of Energy, and Occupational Safety and Health Administration.
Another issue that has surfaced over the past 12 months is intellectual property and copyright infringement, as generative AI synthesizes existing information from many sources. “Several researchers have shown that the output produced by popular LLMs can include copyrighted material such as excerpts from the New York Times and scenes from movies. ” points out the HAI researchers. “Whether such output constitutes copyright infringement is becoming a central legal question.”