austintgriffith/indexing

Onchain Data & Indexing

What You Probably Got Wrong

You try to query historical state via RPC calls. You can't cheaply read past state. eth_call reads current state. Reading state at a historical block requires an archive node (expensive, slow). For historical data, you need an indexer.

You loop through blocks looking for events. Scanning millions of blocks with eth_getLogs is O(n) — it will timeout, get rate-limited, or cost a fortune in RPC credits. Use an indexer that has already processed every block.

You store query results onchain. Leaderboards, activity feeds, analytics — these belong offchain. Compute offchain, index events offchain. If you need an onchain commitment, store a hash.

You don't know about The Graph. The Graph turns your contract's events into a queryable GraphQL API. It's how every serious dApp reads historical data. Etherscan uses indexers. Uniswap uses indexers. So should you.

You treat events as optional. Events are THE primary way to read historical onchain activity. If your contract doesn't emit events, nobody can build a frontend, dashboard, or analytics on top of it. Design contracts event-first.

---

Events Are Your API

Solidity events are cheap to emit (~375 gas base + 375 per indexed topic + 8 gas per byte of data) and free to read offchain. They're stored in transaction receipts, not in contract storage, so they don't cost storage gas.

Design Contracts Event-First

Every state change should emit an event. This isn't just good practice — it's how your frontend, indexer, and block explorer know what happened.

// ✅ Good — every action emits a queryable event
contract Marketplace {
    event Listed(
        uint256 indexed listingId,
        address indexed seller,
        address indexed tokenContract,
        uint256 tokenId,
        uint256 price
    );
    event Sold(uint256 indexed listingId, address indexed buyer, uint256 price);
    event Cancelled(uint256 indexed listingId);

    function list(address token, uint256 tokenId, uint256 price) external {
        uint256 id = nextListingId++;
        listings[id] = Listing(msg.sender, token, tokenId, price, true);
        emit Listed(id, msg.sender, token, tokenId, price);
    }

    function buy(uint256 listingId) external payable {
        // ... transfer logic ...
        emit Sold(listingId, msg.sender, msg.value);
    }
}

Index the fields you'll filter by. You get 3 indexed topics per event. Use them for addresses and IDs that you'll query — seller, buyer, tokenContract, listingId. Don't index large values or values you won't filter on.

Reading Events Directly (Small Scale)

For recent events or low-volume contracts, you can read events directly via RPC:

import { createPublicClient, http, parseAbiItem } from 'viem';

const client = createPublicClient({
  chain: mainnet,
  transport: http(),
});

// Get recent events (last 1000 blocks)
const logs = await client.getLogs({
  address: '0xYourContract',
  event: parseAbiItem('event Sold(uint256 indexed listingId, address indexed buyer, uint256 price)'),
  fromBlock: currentBlock - 1000n,
  toBlock: 'latest',
});

This works for: Last few thousand blocks, low-volume contracts, real-time monitoring. This breaks for: Historical queries, high-volume contracts, anything scanning more than ~10K blocks.

---

The Graph (Subgraphs)

The Graph is a decentralized indexing protocol. You define how to process events, deploy a subgraph, and get a GraphQL API that serves historical data instantly.

When to Use The Graph

• Any dApp that needs historical data (activity feeds, transaction history)
• Leaderboards, rankings, analytics dashboards
• NFT collection browsers (who owns what, transfer history)
• DeFi dashboards (position history, PnL tracking)
• Any query that would require scanning more than ~10K blocks

How It Works

1. Define a schema — what entities you want to query
2. Write mappings — TypeScript handlers that process events into entities
3. Deploy — subgraph indexes all historical events and stays synced

Example: NFT Collection Subgraph

schema.graphql:

type Token @entity {
  id: ID!
  tokenId: BigInt!
  owner: Bytes!
  mintedAt: BigInt!
  transfers: [Transfer!]! @derivedFrom(field: "token")
}

type Transfer @entity {
  id: ID!
  token: Token!
  from: Bytes!
  to: Bytes!
  timestamp: BigInt!
  blockNumber: BigInt!
}

mapping.ts:

import { Transfer as TransferEvent } from './generated/MyNFT/MyNFT';
import { Token, Transfer } from './generated/schema';

export function handleTransfer(event: TransferEvent): void {
  let tokenId = event.params.tokenId.toString();

  // Create or update token entity
  let token = Token.load(tokenId);
  if (token == null) {
    token = new Token(tokenId);
    token.tokenId = event.params.tokenId;
    token.mintedAt = event.block.timestamp;
  }
  token.owner = event.params.to;
  token.save();

  // Create transfer record
  let transfer = new Transfer(
    event.transaction.hash.toHex() + '-' + event.logIndex.toString()
  );
  transfer.token = tokenId;
  transfer.from = event.params.from;
  transfer.to = event.params.to;
  transfer.timestamp = event.block.timestamp;
  transfer.blockNumber = event.block.number;
  transfer.save();
}

Query the subgraph:

{
  tokens(where: { owner: "0xAlice..." }, first: 100) {
    tokenId
    mintedAt
    transfers(orderBy: timestamp, orderDirection: desc, first: 5) {
      from
      to
      timestamp
    }
  }
}

Deploying a Subgraph

# Install
npm install -g @graphprotocol/graph-cli

# Initialize from contract ABI
graph init --studio my-subgraph

# Generate types from schema
graph codegen

# Build
graph build

# Deploy to Subgraph Studio
graph deploy --studio my-subgraph

Subgraph Studio (studio.thegraph.com) — development and testing environment. Free during development. Publish to the decentralized network for production.

---

Alternative Indexing Solutions

| Solution | Best for | Tradeoffs | |----------|----------|-----------| | The Graph | Production dApp backends, decentralized | GraphQL API, requires subgraph development | | Dune Analytics | Dashboards, analytics, ad-hoc queries | SQL interface, great visualization, not for app backends | | Alchemy/QuickNode APIs | Quick token/NFT queries | getTokenBalances, getNFTs, getAssetTransfers — fast but centralized | | Etherscan/Blockscout APIs | Simple event log queries | Rate-limited, not for high-volume | | Ponder | TypeScript-first indexing | Local-first, simpler than The Graph for single-app use | | Direct RPC | Real-time current state only | Only for current state reads, not historical |

Dune Analytics

Write SQL queries over decoded onchain data. Best for analytics and dashboards, not for app backends.

-- Top 10 buyers on your marketplace (last 30 days)
SELECT
    buyer,
    COUNT(*) as purchases,
    SUM(price / 1e18) as total_eth_spent
FROM mycontract_ethereum.Marketplace_evt_Sold
WHERE evt_block_time > NOW() - INTERVAL '30' DAY
GROUP BY buyer
ORDER BY total_eth_spent DESC
LIMIT 10

Enhanced Provider APIs

For common queries, provider APIs are faster than building a subgraph:

// Alchemy: get all tokens held by an address
const balances = await alchemy.core.getTokenBalances(address);

// Alchemy: get all NFTs owned by an address
const nfts = await alchemy.nft.getNftsForOwner(address);

// Alchemy: get transfer history
const transfers = await alchemy.core.getAssetTransfers({
  fromAddress: address,
  category: ['erc20', 'erc721'],
});

---

Reading Current State (Not Historical)

For current balances, allowances, and contract state, direct RPC reads are fine. No indexer needed.

Single Reads

import { createPublicClient, http } from 'viem';

const client = createPublicClient({ chain: mainnet, transport: http() });

// Read current balance
const balance = await client.readContract({
  address: tokenAddress,
  abi: erc20Abi,
  functionName: 'balanceOf',
  args: [userAddress],
});

Batch Reads with Multicall

For multiple reads in one RPC call, use Multicall3 (deployed at the same address on every chain):

// Multicall3: 0xcA11bde05977b3631167028862bE2a173976CA11
// Same address on Ethereum, Arbitrum, Optimism, Base, Polygon, and 50+ chains

const results = await client.multicall({
  contracts: [
    { address: tokenA, abi: erc20Abi, functionName: 'balanceOf', args: [user] },
    { address: tokenB, abi: erc20Abi, functionName: 'balanceOf', args: [user] },
    { address: tokenC, abi: erc20Abi, functionName: 'balanceOf', args: [user] },
    { address: vault, abi: vaultAbi, functionName: 'totalAssets' },
  ],
});
// One RPC call instead of four

Real-Time Updates

For live updates, subscribe to new events via WebSocket:

import { createPublicClient, webSocket } from 'viem';

const client = createPublicClient({
  chain: mainnet,
  transport: webSocket('wss://eth-mainnet.g.alchemy.com/v2/YOUR_KEY'),
});

// Watch for new sales in real-time
const unwatch = client.watchContractEvent({
  address: marketplaceAddress,
  abi: marketplaceAbi,
  eventName: 'Sold',
  onLogs: (logs) => {
    for (const log of logs) {
      console.log(`Sale: listing ${log.args.listingId} for ${log.args.price}`);
    }
  },
});

---

Common Patterns

| What you need | How to get it | |---------------|---------------| | Activity feed for a dApp | Emit events → index with The Graph → query via GraphQL | | Token balances for a user | Alchemy getTokenBalances or Multicall | | NFT collection browser | The Graph subgraph or Alchemy getNftsForContract | | Price history | Dune Analytics or DEX subgraphs | | Real-time new events | WebSocket subscription via viem | | Historical transaction list | The Graph or Alchemy getAssetTransfers | | Dashboard / analytics | Dune Analytics (SQL + charts) | | Protocol TVL tracking | DeFiLlama API or custom subgraph |