Archive for the 'AI' Category
-
January 28, 2025
Marvell leaders share how data centers are transforming to accommodate AI
By Kirt Zimmer, Head of Social Media Marketing, Marvell
What do you get when you combine some of the world’s leading technology analysts with incredibly smart subject matter experts? Answer: the SixFive Media video podcast. It’s must-view content for anyone interested in understanding exactly how AI technologies are evolving.
At Marvell’s recent Investor Analysts Day, company leaders were happy to chat with Patrick Moorhead, CEO and Chief Analyst at Moor Insights & Strategy, and Daniel Newman, CEO and Chief Analyst at The Futurum Group. The resulting conversations (captured on video) were enlightening:
How Custom HBM is Shaping AI Chip Technology
Fresh off Marvell’s announcement of a partnership with SK Hynix, Micron Technology and Samsung Semiconductor, Patrick and Daniel dove into the details with leaders from those organizations. The partnership centers around custom high bandwidth memory (HBM), which fits inside AI accelerators to store data close to the processors.
Custom designs alleviate the physical and thermal constraints traditionally faced by chip designers by dramatically reducing the size and power consumption of the interface and HBM base die. Marvell estimates that up to 25% of the real estate inside the chip package can be recovered via customization.
Will Chu, SVP and GM of Custom Compute and Storage at Marvell, says the company estimates that the total addressable market (TAM) for data centers in 3-4 years is $75B. Last year it was $21B. Out of that, Marvell estimates that $40-43B is for custom accelerators.
Attached to that is custom HBM, which alleviates bottlenecks for AI workloads. In Dong Kim, VP of Product Planning at Samsung Semiconductor said, “Custom HBM will be the majority portion of the market towards the 2027-28 timeframe.” As Patrick Moorhead said, “The rate of change is phenomenal.”
-
January 21, 2025
AI at Scale: A Special Report
By Kirt Zimmer, Head of Social Media Marketing, Marvell
Marvell’s business is accelerated infrastructure for the AI era, which is a fast-evolving space that can occasionally confuse even the most earnest student. To help you keep up, we’ve partnered with VentureBeat to explore a multitude of content about that subject:
- Build or buy? Scaling your enterprise GenAI pipeline in 2025
Enterprise leaders are debating whether to buy AI tools, build their own, or some combination of the two. Companies like Wayfair and Expedia offer valuable insights for organizations looking to scale LLMs effectively. - Purpose-built AI hardware: Smart strategies for scaling infrastructure
Custom AI hardware is the unsung hero of scalable AI infrastructure, helping to tackle a range of issues including performance, cost, and security. For enterprises looking to transition in this rapidly evolving landscape, there’s some great advice here. - AI factories are factories: Overcoming industrial challenges to commoditize AI
Sixty years ago, Alabama was home to a 1.6GW coal fired power plant with the world's tallest chimney. That same site today houses a Google data center. The operations are obviously very different, but some of the infrastructure challenges are somewhat familiar. Read what AI 'factory' really means. - 4 bold AI predictions for 2025
We’ve seen plenty of “predictions for the coming year” pieces in other publications that are honestly pure fluff, but this ain’t that. If your brain is activated by inference costs, reasoning models, transformer alternatives and LLM scaling laws, you’ll appreciate that even annual predictions can be smart and thought-provoking.
- Build or buy? Scaling your enterprise GenAI pipeline in 2025
-
January 15, 2025
Next Up for Custom AI Accelerators: Co-Packaged Optics
By Michael Kanellos, Head of Influencer Relations, Marvell
Computer architects have touted the performance and efficiency gains that can be achieved by replacing copper interconnects with optical technology in servers and processors for decades1.
With AI, it’s finally happening.
Marvell earlier this month announced that it will integrate co-packaged optics (CPO) technology into custom AI accelerators to improve the bandwidth, performance and efficiency of the chips powering AI training clusters and inference servers and opening the door to higher-performing scale-up servers.
The foundation of the offering is the Marvell 6.4Tbps 3D SiPho Engine announced in December 2023 and first demonstrated at OFC in March 2024. The 3D SiPho Engine effectively combines hundreds of components—drivers, transimpedance amplifiers, modulators, etc.—into a chiplet that itself becomes part of the XPU.
With CPO, XPUs will connect directly into an optical scale-up network, transmitting data further, faster, and with less energy per bit. LightCounting estimates that shipments of CPO-enabled ports in servers and other equipment will rise from a nominal number of shipments per year today to over 18 million by 20292.
Additionally, the bandwidth provided by CPO lets system architects think big. Instead of populating data centers with conventional servers containing four or eight XPUs, clouds can shift to systems sporting hundreds or even thousands of CPO-enhanced XPUs spread over multiple racks based around novel architectures—innovative meshes, torus networks—that can slash cost, latency and power. If supercomputers became clusters of standard servers in the 2000s, AI is shifting the pendulum back and turning servers into supercomputers again.
“It enables a huge diversity of parallelism schemes that were not possible with a smaller scale-up network domain,” wrote Dylan Patel of SemiAnalysis in a December article.
-
December 19, 2024
Custom, Copper and Cross-Country Connectivity: Eight Big Trends for Marvell in 2024
By Michael Kanellos, Head of Influencer Relations, Marvell
What happened in semis and accelerated infrastructure in 2024? Here is the recap:
1. Custom Controls the Future
Until relatively recently, computing performance was achieved by increasing transistor density à la Moore’s Law. In the future, it will be achieved through innovative design, and many of those innovative design ideas will come to market first—and mostly— through custom processors tailored to use cases, software environments and performance goals thanks to a convergence of unusual and unstoppable forces1 that quietly began years ago.
- At Marvell AI Day, we announced that custom processors would represent 25% of the AI accelerator market by 2028 and exceed $42 billion. We also announced we had secured four designs with three of the major hyperscalers.
- In July, we unveiled StructeraTM A and Structera X,2 CXL controllers for near memory compute and increasing memory capacity, respectively. Merchant and custom versions of the chips were released at the same time, a reverse of the traditional pattern where custom chips might follow merchant ones by a lengthy lag.
- At OCP, Meta showed off FBNIC, a custom network interface developed in collaboration with Marvell to reduce the time needed to resolve network issues.3 Make that four out of four hyperscalers. See the image below.
FB NIC on display at OFC- In December, we announced a five-year collaboration that includes supplying custom AI products and other devices for AWS.4 The collaboration also encompasses performing semiconductor design tasks in the cloud, which promises to reduce the time and cost of creating new chips.
- Finally, we ended the year with an alliance for custom high bandwidth memory with Micron, Samsung Electronics, and SK hynix.5 Custom HBM can reduce chip real estate by up to 25% and increase memory capacity by up to 33%.6 Custom HBM runs against industry orthodoxy,7 writes analyst Joseph Byrne, but it could pay dividends.
-
December 11, 2024
O-Band Optics: A New Market for Optimizing the Cloud
By Michael Kanellos, Head of Influencer Relations, Marvell
Data infrastructure needs more: more capacity, speed, efficiency, bandwidth and, ultimately, more data centers. The number of data centers owned by the top four cloud operators has grown by 73% since 20201, while total worldwide data center capacity is expected to double to 79 megawatts (MW) in the near future2.
Aquila, the industry’s first O-band coherent DSP, marks a new chapter in optical technology. O-band optics lower the power consumption and complexity of optical modules for links ranging from two to 20 kilometers. O-band modules are longer in reach than PAM4-based optical modules used inside data centers and shorter than C-band and L-band coherent modules. They provide users with an optimized solution for the growing number of data center campuses emerging to manage the expected AI data traffic.
Take a deep dive into our O-band technology with Xi Wang’s blog, O-Band Coherent, An Idea Whose Time is (Nearly) Here, originally published in March, below:
O-Band Coherent: An Idea Whose Time Is (Nearly) Here
By Xi Wang, Vice President of Product Marketing of Optical Connectivity, MarvellOver the last 20 years, data rates for optical technology have climbed 1000x while power per bit has declined by 100x, a stunning trajectory that in many ways paved the way for the cloud, mobile Internet and streaming media.
AI represents the next inflection point in bandwidth demand. Servers powered by AI accelerators and GPUs have far greater bandwidth needs than typical cloud servers: seven high-end GPUs alone can max out a switch that ordinarily can handle 500 cloud two-processor servers. Just as important, demand for AI services, and higher-value AI services such as medical imaging or predictive maintenance, will further drive the need for more bandwidth. The AI market alone is expected to reach $407 billion by 2027.
