mohitmujawdiya/websocket-streaming

WebSocket & Gemini Live API Patterns

Connection Management

1. Always disconnect before reconnecting. The connect() method must call disconnect() first to clean up any existing WebSocket and URLSession. Never leave orphaned connections.

2. Invalidate URLSession on disconnect. URLSession retains its delegate — calling invalidateAndCancel() is required to prevent memory leaks:

   func disconnect() {
       let (ws, session) = lock.withLock {
           let ws = webSocket
           let session = urlSession
           webSocket = nil
           urlSession = nil
           _isConnected = false
           return (ws, session)
       }
       ws?.cancel(with: .goingAway, reason: nil)
       session?.invalidateAndCancel()
   }
   

3. isConnected must be thread-safe. It's read from the audio streaming thread (high frequency) and written from the WebSocket thread. Protect with NSLock.

Audio Streaming (Hot Path)

4. sendAudioChunk is called thousands of times per second. Keep it minimal:

   • Guard on isConnected first (fast exit)
   • Base64 encode the data
   • Build the JSON message
   • Send via fire-and-forget with error logging
   • Do NOT use async/await here — the audio callback can't await

5. JSON message structure for Gemini Live audio:

   {
     "realtime_input": {
       "media_chunks": [
         { "data": "<base64>", "mime_type": "audio/pcm;rate=16000" }
       ]
     }
   }
   

Response Handling

6. Gemini Live responses use tagged text for routing. The listening prompt instructs Gemini to prefix output:

   • [QUESTION] <text> — detected interview question
   • [TRANSCRIPT] <text> — general transcript
   • Untagged text — answer tokens

7. Use String.dropFirst() for tag stripping, not replacingOccurrences. It's more efficient and semantically correct for prefix removal.

8. turnComplete signals end of response. Use this to update UI state (stop spinner, mark answer as complete).

Gemini Live Setup Message

9. The setup message must be sent before any audio. It configures:

   • Model: gemini-2.0-flash-live-001
   • Response modalities: ["TEXT"] (we don't need audio output)
   • Temperature: 0.7 (natural-sounding but not too random)
   • System instruction: the listening prompt with question detection rules

10. System prompt goes in setup, mode prompt goes in client_content. The listening prompt (always active) is set during setup. The answer mode prompt is sent as a user turn when generating answers. This means Gemini has full conversation context when answering.