agiprolabs/shredstream

ShredStream — Pre-Execution Solana Data

ShredStream gives you transaction data before the validator executes the block — typically 100-500ms earlier than standard Yellowstone gRPC. You see transaction intent, not confirmed results.

This is the fastest path to Solana data for time-critical trading strategies.

How It Works

Solana validators produce blocks by serializing transactions into shreds (~1,228 bytes each, sized for UDP MTU). Shreds propagate through Turbine (Solana's fanout protocol, 2-3 hops). ShredStream bypasses Turbine by receiving shreds directly from leader validators via Jito's Block Engine.

Leader Validator
     │
     ├── Turbine (standard, 2-3 hops, 200-500ms)
     │       └── Your RPC Node → Yellowstone gRPC (post-execution)
     │
     └── Jito Block Engine (direct)
              └── ShredStream Proxy (your server)
                   ├── UDP shreds → Your RPC/Validator (faster block building)
                   └── gRPC entries → Your Trading Bot (decoded transactions)

What You Get vs. What You Don't

| Available (Pre-Execution) | NOT Available (Needs Execution) | |---------------------------|--------------------------------| | Transaction signatures | Success/failure status | | Account keys (pubkeys) | Balance changes (pre/post) | | Instructions (program, accounts, data) | Log messages | | Address lookup table references | Inner instructions (CPI) | | Slot number | Token balance changes | | | Compute units consumed |

Key tradeoff: Speed for completeness. You see what's about to happen but can't confirm it actually succeeded. Some transactions you see will ultimately fail.

Three Ways to Get Pre-Execution Data

| Provider | Product | Latency | Access | Cost | |----------|---------|---------|--------|------| | Jito | ShredStream Proxy | ~10-50ms from leader | Apply + auth keypair | Free (beta) | | Shyft | RabbitStream | ~15-100ms faster than gRPC | Shyft gRPC plan | From $199/mo | | Triton | Deshred (SubscribeDeshred) | ~6.3ms p50 from shred | Triton customer | ~$2,900+/mo |

See references/providers_compared.md for detailed comparison.

Option 1: Jito ShredStream Proxy

The most direct approach — run Jito's open-source proxy on your own server.

Get Access

1. Generate a Solana keypair: solana-keygen new -o shred_auth.json
2. Apply at Jito's form with your public key
3. Wait for approval (your keypair gets whitelisted)
4. No staking requirement, free during beta

Run the Proxy

# Clone and build
git clone https://github.com/jito-labs/shredstream-proxy.git --recurse-submodules
cd shredstream-proxy

# Run with gRPC enabled (key flag: --grpc-service-port)
RUST_LOG=info cargo run --release --bin jito-shredstream-proxy -- shredstream \
    --block-engine-url https://mainnet.block-engine.jito.wtf \
    --auth-keypair /path/to/shred_auth.json \
    --desired-regions ny,amsterdam \
    --dest-ip-ports 127.0.0.1:8001 \
    --grpc-service-port 7777

Docker (host networking required for UDP):

docker run -d --name shredstream-proxy --rm \
  --network host \
  -e RUST_LOG=info \
  -e BLOCK_ENGINE_URL=https://mainnet.block-engine.jito.wtf \
  -e AUTH_KEYPAIR=/app/shred_auth.json \
  -e DESIRED_REGIONS=ny,amsterdam \
  -e DEST_IP_PORTS=127.0.0.1:8001 \
  -e GRPC_SERVICE_PORT=7777 \
  -v /path/to/shred_auth.json:/app/shred_auth.json \
  jitolabs/jito-shredstream-proxy shredstream

Configuration

| Parameter | Description | Example | |-----------|-------------|---------| | BLOCK_ENGINE_URL | Jito block engine endpoint | https://mainnet.block-engine.jito.wtf | | AUTH_KEYPAIR | Path to whitelisted Solana keypair | shred_auth.json | | DESIRED_REGIONS | Max 2, comma-separated | ny,amsterdam | | DEST_IP_PORTS | Where to forward raw shreds (UDP) | 127.0.0.1:8001 | | GRPC_SERVICE_PORT | Enable gRPC entry streaming | 7777 | | SRC_BIND_PORT | Incoming shred UDP port | 20000 |

Available regions: amsterdam, dublin, frankfurt, london, ny, salt-lake-city, singapore, tokyo

Verify It's Working

# Check shreds are arriving via UDP
sudo tcpdump 'udp and dst port 20000'
# Should see many ~1200-byte packets continuously

Consume via gRPC

use jito_protos::shredstream::{
    shredstream_proxy_client::ShredstreamProxyClient,
    SubscribeEntriesRequest,
};

let mut client = ShredstreamProxyClient::connect("http://127.0.0.1:7777").await?;
let mut stream = client
    .subscribe_entries(SubscribeEntriesRequest {})
    .await?
    .into_inner();

while let Some(entry) = stream.message().await? {
    let entries: Vec<solana_entry::entry::Entry> =
        bincode::deserialize(&entry.entries)?;

    for e in &entries {
        for tx in &e.transactions {
            let sig = tx.signatures[0];
            let msg = tx.message();
            // Parse instructions, accounts, etc.
        }
    }

    println!("Slot {}: {} entries, {} transactions",
        entry.slot,
        entries.len(),
        entries.iter().map(|e| e.transactions.len()).sum::<usize>()
    );
}

Option 2: Shyft RabbitStream

Drop-in replacement for Yellowstone gRPC — same SubscribeRequest format, just a different endpoint. Easiest way to get pre-execution data without running infrastructure.

export GRPC_ENDPOINT="https://rabbitstream.ny.shyft.to"
export GRPC_TOKEN="your-shyft-x-token"

# Same code as yellowstone-grpc, just different endpoint
import grpc

endpoint = "rabbitstream.ny.shyft.to"
token = os.environ["GRPC_TOKEN"]

# ... standard Yellowstone connection code ...
# Subscribe to transactions — same filter format
request = SubscribeRequest(
    transactions={
        "pumpfun": SubscribeRequestFilterTransactions(
            account_include=["6EF8rrecthR5Dkzon8Nwu78hRvfCKubJ14M5uBEwF6P"],
            vote=False,
            failed=False,
        )
    },
    commitment=CommitmentLevel.PROCESSED,
)

Limitations: Only transaction filters work. No account, slot, or block subscriptions. The meta field is empty (no execution results).

Regional endpoints: rabbitstream.{ny,va,ams,fra}.shyft.to

Option 3: Triton Deshred

Lowest latency (~6.3ms p50) via Triton's SubscribeDeshred RPC. Same Yellowstone client, different method.

// Requires yellowstone-grpc-client with Deshred support
let (mut tx, mut stream) = client.subscribe_deshred().await?;

tx.send(SubscribeDeshredRequest {
    deshred_transactions: hashmap!{
        "pumpfun".to_string() => SubscribeRequestFilterDeshredTransactions {
            vote: Some(false),
            account_include: vec!["6EF8rrecthR5Dkzon8Nwu78hRvfCKubJ14M5uBEwF6P".into()],
            ..Default::default()
        }
    },
    ..Default::default()
}).await?;

Access: Triton customers only, paid beta, requires their custom Agave validator fork.

Parsing Pre-Execution Transactions

Without execution metadata, parsing is simpler but requires program-specific knowledge.

Identify the Program

# From a raw VersionedTransaction (post-deserialization)
account_keys = [str(k) for k in tx.message.account_keys]
for ix in tx.message.instructions:
    program_id = account_keys[ix.program_id_index]
    if program_id == "6EF8rrecthR5Dkzon8Nwu78hRvfCKubJ14M5uBEwF6P":
        # This is a PumpFun instruction
        discriminator = ix.data[:8]
        # Decode instruction data per PumpFun IDL

Instruction Discriminators

Most Solana programs use 8-byte discriminators (Anchor SHA256 hash of the instruction name). Match instruction.data[:8] against known values for each program.

# Common approach
PUMPFUN_CREATE = bytes.fromhex("181ec828051c0777")
PUMPFUN_BUY = bytes.fromhex("66063d1201daebea")
PUMPFUN_SELL = bytes.fromhex("33e685a4017f83ad")

disc = ix.data[:8]
if disc == PUMPFUN_BUY:
    # Parse buy parameters from remaining bytes
    ...

Warning: Discriminator values are program-specific and can change between program versions. Always verify against the current program IDL. See the pumpfun-mechanics skill for PumpFun-specific parsing.

Common Architecture: ShredStream + Yellowstone

Most production systems use both:

ShredStream (pre-execution)          Yellowstone gRPC (post-execution)
         │                                      │
         ▼                                      ▼
  Intent Detection                    Confirmation + Reconciliation
  "Wallet X is buying token Y"       "Buy succeeded, wallet now holds Z"
         │                                      │
         ▼                                      ▼
  Pre-compute Response                Execute / Update State
  (route, sign, prepare bundle)       (record PnL, update positions)

This gives you the speed advantage of ShredStream for signal detection while using Yellowstone for reliable state management.

Deployment Requirements

• Public IP required — NAT breaks UDP shred delivery
• Host networking — Docker bridge mode drops shred packets
• UDP port 20000 open for incoming shreds
• Co-locate near validators — Frankfurt, NY, Amsterdam, London, Tokyo recommended
• Bare metal preferred — Cloud VMs add 1-5ms jitter from shared NICs

See references/deployment.md for full infrastructure guide.

Files

References

• references/providers_compared.md — Jito ShredStream vs Shyft RabbitStream vs Triton Deshred
• references/deployment.md — Infrastructure requirements, region selection, firewall configuration
• references/proto_reference.md — ShredStream protobuf definitions and Entry parsing

Scripts

• scripts/parse_shredstream_entries.py — Decode and analyze ShredStream gRPC entries
• scripts/rabbitstream_monitor.py — Connect to Shyft RabbitStream for pre-execution transaction monitoring