Cerebras Systems, an American artificial intelligence (AI) startup, claims to have created a single processor the size of an iPad that can accomplish the computational equivalent of the human brain or the equivalent of 100 trillion synapses.
Cerebras Wafer Scale Engine-2 (WSE-2) is also claimed to be the largest chip ever produced by the business.
The processor contains 2.6 trillion transistors
Cerebras’ WSE-2 is a single processor built on TSMC’s seven-nanometer process technology, with 2.6 trillion transistors crammed into 850,000 AI-optimized cores deployed on a massive 46,225 square millimeter surface area.
For calculations, the device includes 40GB of onboard SRAM memory with a memory bandwidth of 20 petabytes/second that interfaces with 2.4 petabytes of off-chip memory.
AI models with over 120 trillion parameters can be processed by the computer
This absolute unit of a processor is located within the Cerebras CS-2 custom computer. It can easily take up a full server rack.
The processor can handle complicated AI models with over 120 trillion parameters in a matter of seconds.
Parameters are machine learning algorithm blocks that encapsulate what the machine has learned from the data used to train it.
Andrew Feldman, CEO of Cerebras, explains the relevance of the WSE-2.
Cerebras CEO and founder Andrew Feldman told VentureBeat that “larger networks, like as (Microsoft’s) GPT-3, have already transformed the natural language processing (NLP) landscape, making possible what was previously unimaginable.”
“The industry is moving past… trillion-parameter models, and we are extending that boundary by two orders of magnitude, enabling brain-scale neural networks with 120 trillion parameters,” he added.
A single silicon ingot is used to create this massive CPU
At semiconductor foundries like TSMC, hundreds of traditional desktop computing processors are carved from a 12-inch diameter ingot of silicon. The new WSE-2 processor is a single chip produced from a single ingot.
This is more efficient than distributing a computational effort among multiple smaller, traditional workstation processors.
Cerebras’ latest software-side technological improvements aid matters.
WSE-2 benefits from Cerebras’ linked technology
Weight Streaming, a software execution architecture, allows researchers to stream AI model parameters to the chip from its attached off-die memory without worrying about latency or memory bandwidth. This enables researchers to use anywhere from one to 192 CS-2 machines at the same time.
This is linked to Cerebras’ MemoryX technology, which helps the 2.4 petabytes of off-chip storage efficiently transmit data to the processor.
However, each Cerebras CS-2 cannot be readily upscaled
While researchers may run numerous CS-2 computers concurrently, upscaling the capabilities of an existing CS-2 computer is very hard, says Tom’s Hardware.
This is due to the massive processor requiring 15kW of power from a bespoke power source. It also necessitates custom cooling techniques, making it difficult to fit another wafer-sized device into a single computer.
Cerebras’ WSE-2 can compute as much as a human brain
Cerebras was created in 2016 and currently employs over 350 employees.
In January, Google trained a 1.6 trillion parameter model in comparison to the Cerebras chip. Feldman’s Cerebras system is 1,000 times more powerful than Google’s system and can process 1,000 times more data.
While the WSE-2 can handle 100 trillion synapses, Intel crammed 768 Loihi “brain chips” to rival a mole rat’s brain in 2020.