Constellation Network and Common Crawl Foundation are revolutionizing web data accessibility and AI development through blockchain technology
The Common Crawl Foundation, a nonprofit organization founded in 2007 dedicated to providing the public with a copy of the Internet, and Constellation Network, a Web3 blockchain ecosystem that excels in providing solutions to the U.S. Department of Defense today announced a strategic partnership aimed at democratizing and improving the accessibility and utility of web crawl data on blockchain technology for artificial intelligence (AI) and data applications.
This collaboration will explore potential ways to improve large language models used by AI. The starting point is Common Crawl’s massive dataset, used by 80% of major language models, having crawled over 250 billion web pages to date (19 billion in 2024 alone) and consisting of an archive of almost nine petabytes of archived crawl data. Leveraging Constellation’s Hypergraph decentralized network, which ensures immutability, provenance and auditability of data, the partnership is designed to provide shared solutions for responsible and transparent AI.
With AI set to be a $3 trillion industry by 2030, there is a growing need for secure solutions for sharing datasets used for training large language models, improved storage of queried and cleaned data, opportunities to monetize data and improved transparency regarding the source of the data. With Constellation’s unique approach to providing tools to converge existing infrastructure with distributed and decentralized networks, and Common Crawl’s history of data and data usage growth, this partnership is designed to further democratize data.
“This partnership represents a significant step forward in ensuring the trusted distribution of Common Crawl,” said Rich Skrenta, Executive Director of the Common Crawl Foundation. “By combining our comprehensive web archive with Constellation’s proven blockchain technology, researchers can and developers around the world trust what they get from Common Crawl and have a model for authenticating large open datasets, such as those used for AI training.”
Ben Jorgensen, Managing Director of Constellation Network, explains: “The partnership between Constellation Network and Common Crawl underscores the widespread adoption of web3 solutions outside the echo chambers of the crypto economy. This alignment continues Constellation’s mission to leverage our Zero Trust network as a public good for a data-driven future.” Jorgensen continues: “Our goal is to attract new developers by building capabilities such as integrating immutability into show digital workflows and thus further differentiate us from previous generations of blockchain technology.”
The two organizations will implement this initiative gradually, starting with a customizable subnet called a metagraph that incorporates a subset of Common Crawl’s data. This subnet is currently operational on their testnet and will soon be integrated into Hypergraph, Constellation’s public network. More details about the live metagraph will be announced in the coming weeks, as well as information about how organizations and developers can participate.
Further information can be found at:
Information about the Common Crawl Foundation
The Common Crawl Foundation is a 501(c)(3) nonprofit organization dedicated to providing the public with a free copy of the Internet. Their web archive consists of petabytes of data collected through years of web crawling and serves as an important resource for researchers, companies, and developers worldwide.
About Constellation Network
Constellation Network is a Web3 blockchain ecosystem that builds bridges between crypto economies and traditional businesses. Its flagship Hypergraph offers a solution for fast, scalable and fee-free transactions. Constellation’s network is validated by the US Department of Defense, which has been a customer since 2019.
Note: This press release contains forward-looking statements. Actual results may differ materially from those projected.
Logo – https://mma.prnewswire.com/media/2537537/ConstellationxCommonCrawl_Logo.jpg
View original content:https://www.prnewswire.com/news-releases/common-crawl-foundation-und-constellation-network-kundigen-partnerschaft-an-um-eine-brucke-zwischen-blockchain-und-ki-zu-schlagen-302285946.html