AI Lab Scales Storage for Datasets Built Using Crowdsourced Bandwidth

How it works

With three million users, Grass effectively has a network of 3 million nodes worldwide with a network bandwidth of around 100 Gbits per second and growing. These nodes receive web requests from Grass’ validators and scrape data from websites by passing these requests through as normal traffic. The scraped data is then sent back to Grass’ servers where it is cleaned and reformatted before being stored on Backblaze. This ethically sourced, massive dataset is then curated by Grass and sold to enterprises for AI training and other data-driven applications.

‍

The details 

When the math doesn’t math

The founders of Wynd Labs envisioned a future where AI is fueled by ethically sourced, high-quality, unbiased data. The path to that future looked like Grass, a global, distributed, opt-in network that allows users to contribute their unused bandwidth and compute resources to scrape the internet. Grass initially stored the datasets it gathered with a traditional cloud provider but hit a blocker when approached by large enterprise prospects interested in the product: the cloud provider’s cost structure would effectively kill the deals, especially when clients demanded faster data retrieval at scale.

‍

The storage provider fell short on three counts:

• High costs: The baseline cost of storage was too high for their needs, especially with the massive data sets they were handling.
• Data egress fees: Their previous provider charged high fees for data egress, which was a significant expense due to the volume of data being transferred.
• Data deletion fees: The cloud provider’s minimum storage duration policy added costs for deleting data, which became a challenge when frequent data cleanup and reformatting were necessary.

Finding a better calculator

Grass explored multiple storage providers, comparing large, general-purpose cloud providers to more storage-specialized options. Backblaze won out with its affordability, usability, and positive reviews on platforms like Reddit—Wynd Labs CTO, Chris Nguyen, particularly trusted the reviews from engineers about the functionality of the product.

‍

Nguyen then kicked off a proof of concept by starting with a pay-as-you-go plan to test upload speeds and integration via the Backblaze S3 compatible API, which he found to be straightforward. Convinced by the results, he engaged with Backblaze for a larger commitment and migrated data while continuing to run the previous provider parallel to mitigate downtime.

High speed and low cost balance the equation

Backblaze's scalable object storage solution has been instrumental in supporting the Grass network’s rapid growth and ambitious data collection goals. Specifically, Backblaze delivers:

• Fast upload speeds: Backblaze’s fast upload speeds enable Grass to efficiently transfer and store their massive datasets, minimizing delays and ensuring data is readily available for processing, analysis, and deliverability.
• Cost-effectiveness and scalability: Grass can store massive datasets without incurring excessive delete fees or egress costs, allowing the team to focus on innovation and expansion.
• High-speed data retrieval: Backblaze’s fast retrieval supports Grass’s vision for real-time data access, crucial for building unbiased AI models and competing with industry giants.

By eliminating the high costs and limitations faced with their previous provider, Backblaze has freed up resources Grass can reinvest into further developing its infrastructure and technology. The partnership empowers Grass to pursue its goal of not only competing with industry giants but ultimately reshaping the AI landscape by making ethically sourced, real-time data accessible to all.