Scale of data aggregation

The main challenge with a product like Unified is the scale of data aggregation that occurs with multiple concurrent users.

Essentially, the Unified dashboard product automates the collection of data from multiple sources at one time and then displays it to the user.

This is similar to "scraping data" from multiple sources through the use of application programming interfaces (APIs).

Price/volume comparison sites, blockchain explorers and centralized exchanges tend to enforce rate limitations to prevent the APIs from being called frequently.

Consider what will happen when:

  • Multiple users send out a query, and

  • Each user has multiple addresses, which reside in multiple blockchain protocols, and

  • Each user's address may be participating in multiple projects, and

  • Each user may be using multiple centralized exchanges, and

  • All those queries are sent out simultaneously.

To solve for this, Unified employs several approaches.

  • The first solution is that Unified focuses on the major blockchain protocols, although the practical reality is that Unified will eventually cover all blockchain protocols.

  • The second solution is that Unified focuses on the major projects (as represented by TVL and activity). This is a more practical, restrictive condition as there are many projects that eventually fall by the wayside, and it is not practical for Unified to automatically query the explorers in relation to such projects.

  • The third solution is that Unified uses 1,000s of accounts to separate out API calls by users so as to reduce rate limitation by price/volume comparison sites, blockchain explorers and centralized exchanges.

  • The fourth solution is that Unified develops specific partnerships with price/volume comparison sites, blockchain explorers and centralized exchanges so that addresses from which API calls originate are whitelisted for minimal rate limitations.

Last updated