Skip to content

EthStorage

What is EthStorage?

EthStorage is a decentralized storage solution that uses a key-value storage paradigm. Unlike the market-leading decentralized storage projects Arweave and Filecoin, which use a static file storage paradigm, EthStorage employs key-value storage. Thanks to this paradigm, EthStorage supports CRUD operations — creating new storage data, updating storage data, reading storage data, and deleting storage data. While this is easily achievable in centralized storage, EthStorage is currently the only decentralized storage solution that supports full CRUD operations. The reason is that update operations conflict with blockchain's immutability characteristic. To enable data updates, EthStorage implemented an ingenious storage solution: a KV storage paradigm combined with a custom storage proof algorithm (EthStorage's storage proof algorithm is a combination of PoW and Proof-of-Random-Access).

EthStorage is fully EVM-compatible. EVM-compatible EthStorage provides smart contracts with near-perfect interoperability, which is unachievable with other decentralized storage solutions.

EthStorage is Ethereum's storage L2. EthStorage actually adopts an architecture similar to L2. A storage contract is deployed on Ethereum as the entry point for EthStorage data operations. At the same time, proofs of off-chain storage data from data nodes must also be verified through this contract. Current Op Rollup or ZK Rollup scaling solutions focus on scaling Ethereum's computational capacity (executing new state trees off-chain), while EthStorage Rollup's scaling direction is scaling Ethereum's data storage capacity.

EthStorage's client is a superset of the Ethereum client Geth, which means that when running an EthStorage node, it can still normally participate in any Ethereum process. When starting an EthStorage Node, you are actually running a combination of Geth and Data Provider. The internally running Geth ensures the node can normally participate in the Ethereum network. For example, a node can simultaneously be an Ethereum validator node and an EthStorage data node. Each EthStorage Node's Data Provider module initiates connection requests with other EthStorage Node Data Providers. Once they connect to each other, they effectively form a decentralized storage network.

How Does EthStorage Store and Read Data?

First, EthStorage deploys a smart contract on the Ethereum mainnet to support CRUD operations.

Specific use cases for each contract method:

  • put: This is a write data method. Calling this method stores the data you provide in the corresponding data shard, and you can read it through the corresponding key in the read method
  • get: This is a query method. Calling this method retrieves the data corresponding to the key you want to query
  • remove: This is a delete data method. Calling this method deletes the data corresponding to the key
  • verify: This is a verification method. It checks whether the data hash of the data stored in the shard matches the data hash in the contract. If they match, it proves the off-chain stored data is correct

Data Storage Process:

Writing data to EthStorage is done by calling the put(bytes32 key, bytes memory data) method in the contract.

We deploy a smart contract on Ethereum. When a user fills in any key and data in the put(bytes32 key, bytes memory data) method and initiates an Ethereum transaction calling this method, upon successful transaction execution, the contract records the key of this write operation, along with the corresponding data hash (dataHash) and a kvIdx assigned by EthStorage. When the EthStorage client detects a put transaction, it stores the data in the corresponding position in the data shard based on the assigned kvIdx.

During this process, the data that users need to store is stored in the EthStorage network, which is an off-chain network of Ethereum. The only data stored in the on-chain smart contract are three values: key, datahash, and kvIdx.

Suppose a user submits a storage transaction (calling put(key,data)) and it is included in the current proposed block. After this transaction in the block enters the virtual machine (EVM) and executes successfully, the Ethereum world state records the key, the data to be stored, and the data hash (dataHash), while also assigning a kvIdx to this key.

When the entire block has been executed successfully, if a validator is simultaneously running both a validator node and a data node, the validator node updates its world state following the normal process (which includes updating the latest stored key, data, dataHash in the contract account), and then the data node stores the raw data from memory to the corresponding position in the data shard according to the assigned kvIdx.

Data Update Process:

When a user needs to update data stored on EthStorage, they still initiate an Ethereum transaction calling the contract to execute the put method:

  • Method Call

put( key , new_data )

key: The key corresponding to the user's previously stored data

new_data: The new data the user wants to store

  • Execution Process

The update data execution process is essentially similar to the storage data process, specifically:

  1. The user initiates a transaction calling the contract's put method: submit Tx{ put( key , new_data )};

  2. This transaction updates the hash recorded in the contract for the corresponding key: dataHash = hash(new_data);

  3. The EthStorage client's ShardManager detects a new put transaction and retrieves the data to be updated (new_data) from the put method;

  4. The ShardManager masks the data to be updated and replaces the previously stored data: store( kvIdx , mask(new_data))

Data Verification:

The core principle is that EthStorage records the hash of the data corresponding to each key in the contract. During each Proof-of-Random-Access, the data uploaded by the miner is hashed and verified against the data hash stored in the contract. Only when the contract-stored data hash matches the uploaded data hash will the random access proof generate a valid proof.

Data Reading:

You can make a local call to the get(key) method provided in the EthStorage contract to find the corresponding kvIdx through the key, then send a read request to a data node based on the kvIdx. When the data node determines that the data corresponding to the kvIdx is in its stored data shard, it replies to the user with the requested data. If the miner does not store the corresponding data, it will forward the data read request on behalf of the user.

Data Deletion:

Assume the following scenario:

KeyA has a kvIdx of 1, and the user wants to remove the data corresponding to KeyA. The current maximum kvIdx corresponds to the key KeyB.

  • Method Call: remove(key)

key: The index key of the data you want to delete

  • Deletion Steps:

  • Initiate a transaction calling the remove method: submit Tx{remove(KeyA)};

  • The ShardManager reads out the data corresponding to KeyB and copies it to KeyA's data storage location: copy(chunkIdx(KeyA), readMasked(keyB));
  • The ShardManager deletes KeyB's data (chunk99): delete(chunk99);
  • The contract changes KeyB's corresponding kvIdx from 99 to 1: updateKvIdx(KeyB, 1);
  • The contract changes lastKVIdx from 99 to 98: lastKvIdx = lastKvIdx - 1;

Future Applications of EthStorage

On-Chain NFT

Currently, most NFTs store their metadata on Arweave and IPFS, only submitting the returned content hash on-chain. This is because storing NFT metadata on-chain is extremely expensive, as the data would occupy significant block space. Therefore, most NFTs previously kept their metadata off-chain.

However, with EthStorage's solution, users can store NFT data directly on-chain, custody the data through smart contracts, and render it on the frontend via the Web3 Access Protocol. By comparison, OpenSea — the most widely used NFT marketplace — does not directly fetch metadata for its frontend display either. Instead, it stores the corresponding NFT metadata on OpenSea's centralized servers for a better access experience. With EthStorage's solution, users can not only store NFT data directly on-chain but also fetch on-chain NFT metadata directly from the frontend, greatly improving the user experience.

Beyond on-chain NFTs, the programmability of smart contracts enables dynamic NFTs, 3D NFTs, and NFT composability. Programmability means unlimited possibilities. For on-chain games, the composability and programmability of on-chain NFTs can spark game creativity and generate more gameplay mechanics.

Personal Website

Personal website data is uploaded from local directories through smart contracts. Through the Web3 Access Protocol, local directories are mapped to a file system hosted on smart contracts. When users want to access resources at relative or absolute paths, the data content is accessed through calldata.

DeWeb

We know that Ethereum is a decentralized network that has given birth to many decentralized DApps. However, these DApps are not fully decentralized — many application frontends are still hosted on centralized cloud services. Incidents like Uniswap's frontend downtime, removal of trading pairs, and Tornado Cash's frontend being shut down due to regulatory action over alleged money laundering all occurred because their frontends were hosted on centralized servers, unable to effectively resist censorship. With EthStorage's solution, web files and data are hosted in smart contracts, jointly operated and maintained by a decentralized network, greatly enhancing censorship resistance. Through the programmability of smart contracts to implement DeWeb, many interesting applications can be built, such as De-GitHub, De-Blog, and frontends for various DApps.

Modifiable Mirror

We know that Mirror uses Arweave for data storage, and Arweave is a permanent storage protocol. Articles created by users through Mirror are ultimately stored on Arweave. Due to its permanent storage and immutability characteristics, articles published through Mirror cannot be modified in place. To make changes, the modified article must be stored on Arweave through a new transaction, resulting in redundant data storage on the network and double storage costs. By choosing EthStorage to build a Mirror-like application, users can directly update stored data through an Update operation without overwriting previously stored data.

Excerpted from:

https://literate-wolfsbane-bf0.notion.site/EthStorage-Ethereum-L2-003f3828c1c34341ac48fe494a30d71c