OpenAI Realtime Transcription

[{"_1":2,"_3":-5,"_4":-5},"loaderData",{"_5":6,"_7":8,"_9":10},"actionData","errors","root",{"_328":329,"_330":331},"routes/_app.x.$handle.$name",{"_97":96,"_31":43,"_105":84,"_26":106,"_13":14,"_107":-5,"_11":108,"_109":110,"_111":112,"_113":114,"_115":116,"_20":-5,"_117":118,"_119":120,"_121":24,"_122":123,"_124":56,"_125":-5,"_126":-5,"_127":128},"routes/_app.x.$handle.$name.branch.$branch.version.$version.diff",{"_11":12,"_13":14,"_15":16,"_17":-5,"_18":-5},"project",{"_41":42,"_31":43,"_36":44,"_45":-5,"_46":24,"_47":48,"_49":48,"_50":-5,"_51":-5,"_18":52,"_53":54,"_55":56,"_57":56,"_58":-5,"_59":60,"_61":62,"_63":-5,"_64":-5,"_65":-5,"_66":-5,"_67":68,"_69":68,"_70":71,"_72":71,"_73":71,"_74":68,"_75":68,"_76":77,"_78":79,"_80":81,"_82":24},"version",26,"changes",[19],"aiChatInfo","userId",{"_20":21,"_22":-7,"_23":24,"_25":24,"_26":27,"_28":27},"file",{"_29":30,"_31":32,"_33":20,"_34":35,"_36":37,"_38":14,"_39":40,"_23":24},"previous","deleted",false,"deletedHere","path","docs/guides/realtime-transcription.md","latestPath","fileId","39772a88-8935-11f0-9096-0224a6c84d84","name","realtime-transcription.md","fileType","content","Realtime transcription\n======================\n\nLearn how to transcribe audio in real-time with the Realtime API.\n\nYou can use the Realtime API for transcription-only use cases, either with input from a microphone or from a file. For example, you can use it to generate subtitles or transcripts in real-time. With the transcription-only mode, the model will not generate responses.\n\nIf you want the model to produce responses, you can use the Realtime API in [speech-to-speech conversation mode](/docs/guides/realtime-conversations).\n\nRealtime transcription sessions\n-------------------------------\n\nTo use the Realtime API for transcription, you need to create a transcription session, connecting via [WebSockets](/docs/guides/realtime?use-case=transcription#connect-with-websockets) or [WebRTC](/docs/guides/realtime?use-case=transcription#connect-with-webrtc).\n\nUnlike the regular Realtime API sessions for conversations, the transcription sessions typically don't contain responses from the model.\n\nThe transcription session object is also different from regular Realtime API sessions:\n\n```json\n{\n object: \"realtime.transcription_session\",\n id: string,\n input_audio_format: string,\n input_audio_transcription: [{\n model: string,\n prompt: string,\n language: string\n }],\n turn_detection: {\n type: \"server_vad\",\n threshold: float,\n prefix_padding_ms: integer,\n silence_duration_ms: integer,\n } | null,\n input_audio_noise_reduction: {\n type: \"near_field\" | \"far_field\"\n },\n include: list[string] | null\n}\n```\n\nSome of the additional properties transcription sessions support are:\n\n* `input_audio_transcription.model`: The transcription model to use, currently `gpt-4o-transcribe`, `gpt-4o-mini-transcribe`, and `whisper-1` are supported\n* `input_audio_transcription.prompt`: The prompt to use for the transcription, to guide the model (e.g. \"Expect words related to technology\")\n* `input_audio_transcription.language`: The language to use for the transcription, ideally in ISO-639-1 format (e.g. \"en\", \"fr\"...) to improve accuracy and latency\n* `input_audio_noise_reduction`: The noise reduction configuration to use for the transcription\n* `include`: The list of properties to include in the transcription events\n\nPossible values for the input audio format are: `pcm16` (default), `g711_ulaw` and `g711_alaw`.\n\nYou can find more information about the transcription session object in the [API reference](/docs/api-reference/realtime-sessions/transcription_session_object).\n\nHandling transcriptions\n-----------------------\n\nWhen using the Realtime API for transcription, you can listen for the `conversation.item.input_audio_transcription.delta` and `conversation.item.input_audio_transcription.completed` events.\n\nFor `whisper-1` the `delta` event will contain full turn transcript, same as `completed` event. For `gpt-4o-transcribe` and `gpt-4o-mini-transcribe` the `delta` event will contain incremental transcripts as they are streamed out from the model.\n\nHere is an example transcription delta event:\n\n```json\n{\n \"event_id\": \"event_2122\",\n \"type\": \"conversation.item.input_audio_transcription.delta\",\n \"item_id\": \"item_003\",\n \"content_index\": 0,\n \"delta\": \"Hello,\"\n}\n```\n\nHere is an example transcription completion event:\n\n```json\n{\n \"event_id\": \"event_2122\",\n \"type\": \"conversation.item.input_audio_transcription.completed\",\n \"item_id\": \"item_003\",\n \"content_index\": 0,\n \"transcript\": \"Hello, how are you?\"\n}\n```\n\nNote that ordering between completion events from different speech turns is not guaranteed. You should use `item_id` to match these events to the `input_audio_buffer.committed` events and use `input_audio_buffer.committed.previous_item_id` to handle the ordering.\n\nTo send audio data to the transcription session, you can use the `input_audio_buffer.append` event.\n\nYou have 2 options:\n\n* Use a streaming microphone input\n* Stream data from a wav file\n\nVoice activity detection\n------------------------\n\nThe Realtime API supports automatic voice activity detection (VAD). Enabled by default, VAD will control when the input audio buffer is committed, therefore when transcription begins.\n\nRead more about configuring VAD in our [Voice Activity Detection](/docs/guides/realtime-vad) guide.\n\nYou can also disable VAD by setting the `turn_detection` property to `null`, and control when to commit the input audio on your end.\n\nAdditional configurations\n-------------------------\n\n### Noise reduction\n\nYou can use the `input_audio_noise_reduction` property to configure how to handle noise reduction in the audio stream.\n\nThe possible values are:\n\n* `near_field`: Use near-field noise reduction.\n* `far_field`: Use far-field noise reduction.\n* `null`: Disable noise reduction.\n\nThe default value is `near_field`, and you can disable noise reduction by setting the property to `null`.\n\n### Using logprobs\n\nYou can use the `include` property to include logprobs in the transcription events, using `item.input_audio_transcription.logprobs`.\n\nThose logprobs can be used to calculate the confidence score of the transcription.\n\n```json\n{\n \"type\": \"transcription_session.update\",\n \"input_audio_format\": \"pcm16\",\n \"input_audio_transcription\": {\n \"model\": \"gpt-4o-transcribe\",\n \"prompt\": \"\",\n \"language\": \"\"\n },\n \"turn_detection\": {\n \"type\": \"server_vad\",\n \"threshold\": 0.5,\n \"prefix_padding_ms\": 300,\n \"silence_duration_ms\": 500,\n },\n \"input_audio_noise_reduction\": {\n \"type\": \"near_field\"\n },\n \"include\": [ \n \"item.input_audio_transcription.logprobs\",\n ],\n}\n```\n\nWas this page useful?","createdAt",["D",1756952213233],"branchVersion","branchId","509ca3b6-882f-11f0-9954-0224a6c84d84","id","50991944-882f-11f0-9954-0224a6c84d84","hello-transcription",["D",1756839724543],"deletedAt","legacyEnvVars","privacy","public","httpPrivacy","description","imageUrl","0198f8d1-d70f-76ea-a55f-2b5ba506023d","pinnedFileId","01990bce-c638-714e-97a4-20de8c449fb9","pinPreview",true,"pinMigrated","screenshotVersion","likeCount",0,"orgId","0198f8d1-d70f-76ea-a55f-2ed16889c1b0","tursoName","tursoHostname","tursoMeasuredAt","pinned","userPerm","r","valPerm","blobPerm","rw","emailPerm","sqlitePerm","projectPerm","telemetryPerm","user",{"_97":96,"_104":100},"org",{"_97":96,"_98":-5,"_99":100,"_18":52,"_101":102,"_103":24},"branch",{"_13":83,"_31":84,"_41":40,"_85":42,"_86":87,"_88":89,"_90":48,"_91":92,"_93":62,"_94":43,"_95":96},"isOwned",45,"main","projectId","forkedBranchId","43b7750c-87f0-11f0-ba02-0224a6c84d84","forkedBranchVersion",15,"forkedProjectPrivacy","forkedProjectId","43b401c4-87f0-11f0-ba02-0224a6c84d84","forkedProjectOrgId","projectName","userHandle","emcho","handle","displayName","avatar","https://img.clerk.com/eyJ0eXBlIjoicHJveHkiLCJzcmMiOiJodHRwczovL2ltYWdlcy5jbGVyay5kZXYvb2F1dGhfZ2l0aHViL2ltZ18zMXpKcFhORzY4VDlCZlRBZWUybTNFMmV3bDYifQ","tier","free","canRestrictVals","profileImageUrl","branchName","","adoptOrgId",{"_41":42,"_31":43,"_36":324,"_45":-5,"_46":24,"_47":48,"_49":48,"_50":-5,"_51":-5,"_18":52,"_53":54,"_55":56,"_57":56,"_58":-5,"_59":60,"_61":62,"_63":-5,"_64":-5,"_65":-5,"_66":-5,"_67":68,"_69":68,"_70":71,"_72":71,"_73":71,"_74":68,"_75":68,"_76":325,"_78":326,"_80":327,"_82":24},"tree",[218,220,224,226,230,231,232,233,234],"files",[218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234],"counts",{"_208":60,"_209":154,"_210":154,"_211":212,"_213":214},"contents",[139,140,141,142,143,144,145,146,147],"upstreamProject",{"_41":92,"_31":137,"_50":-5,"_51":-5,"_47":48,"_76":138},"upstreamBranch",{"_41":87,"_36":129,"_130":131,"_31":84,"_85":92,"_13":132,"_86":133,"_88":134,"_135":136,"_45":-5,"_82":24,"_47":48,"_95":96,"_61":62,"_94":137},"emptyEnvVars","flags",[],"hasHttpFiles","grants","orgs","customDomains",{},["D",1756812644593],"updatedAt",["D",1757114821629],23,"be33420e-792a-11f0-bfa3-0224a6c84d84",197,"lockfileKey","1764516869134-esm.town_v_emcho_hello-mcp@23-main","hello-mcp",{"_97":96,"_104":100},{"_41":202,"_31":203,"_29":204,"_150":-5,"_39":40,"_38":205,"_152":56,"_36":206,"_13":60,"_33":189,"_34":-5,"_23":24,"_157":-5,"_26":207},{"_41":196,"_31":197,"_29":198,"_150":-5,"_39":40,"_38":199,"_152":56,"_36":200,"_13":60,"_33":189,"_34":-5,"_23":24,"_157":-5,"_26":201},{"_41":191,"_31":192,"_29":193,"_150":-5,"_39":40,"_38":154,"_152":56,"_36":194,"_13":60,"_33":189,"_34":-5,"_23":24,"_157":-5,"_26":195},{"_41":185,"_31":186,"_29":187,"_150":-5,"_39":40,"_38":154,"_152":56,"_36":188,"_13":60,"_33":189,"_34":-5,"_23":24,"_157":-5,"_26":190},{"_41":179,"_31":180,"_29":181,"_150":-5,"_39":40,"_38":154,"_152":56,"_36":182,"_13":60,"_33":20,"_34":183,"_23":24,"_157":-5,"_26":184},{"_41":172,"_31":173,"_29":174,"_150":-5,"_39":40,"_38":175,"_152":56,"_36":176,"_13":60,"_33":20,"_34":177,"_23":24,"_157":-5,"_26":178},{"_41":165,"_31":166,"_29":167,"_150":-5,"_39":40,"_38":168,"_152":56,"_36":169,"_13":154,"_33":20,"_34":170,"_23":24,"_157":-5,"_26":171},{"_41":159,"_31":160,"_29":161,"_150":-5,"_39":40,"_38":154,"_152":56,"_36":162,"_13":60,"_33":20,"_34":163,"_23":24,"_157":-5,"_26":164},{"_41":148,"_31":149,"_29":54,"_150":-5,"_39":40,"_38":151,"_152":24,"_36":153,"_13":154,"_33":155,"_34":156,"_23":24,"_157":-5,"_26":158},"1a3cd812-8830-11f0-8b63-0224a6c84d84","main.tsx","parentId",3,"latest",["D",1756840062830],1,"http","import { Hono } from \"npm:hono\";\nimport { parseVal, serveFile } from \"https://esm.town/v/std/utils/index.ts\";\nimport rtc from \"./routes/rtc.ts\";\nimport observer from \"./routes/observer.ts\";\n\nconst SOURCE_URL = parseVal(Deno.env.get(\"VALTOWN_ENTRYPOINT\"))?.links?.self?.val || \"#\";\n\nconst app = new Hono();\napp.route(\"/rtc\", rtc);\napp.route(\"/observer\", observer);\napp.get(\"/source\", (c) => c.redirect(SOURCE_URL));\n\napp.get(\"/\", () => serveFile(\"/frontend/index.html\"));\n\nexport default app.fetch;","label",[149],"50a3ea9a-882f-11f0-9954-0224a6c84d84","deno.json","01990bce-c638-714e-97a4-1fa9e6167c76",["D",1756839724543],"{\n \"$schema\": \"https://raw.githubusercontent.com/denoland/deno/348900b8b79f4a434cab4c74b3bc8d4d2fa8ee74/cli/schemas/config-file.v1.json\",\n \"lock\": false,\n \"compilerOptions\": {\n \"noImplicitAny\": false,\n \"strict\": false,\n \"types\": [\n \"https://www.val.town/types/valtown.d.ts\"\n ],\n \"lib\": [\n \"dom\",\n \"dom.iterable\",\n \"dom.asynciterable\",\n \"deno.ns\",\n \"deno.unstable\"\n ]\n },\n \"lint\": {\n \"files\": {\n \"include\": [\n \"deno:/https/esm.town/**/*\"\n ]\n },\n \"rules\": {\n \"exclude\": [\n \"no-explicit-any\"\n ]\n }\n },\n \"node_modules_dir\": false,\n \"experimental\": {\n \"unstable-node-globals\": true,\n \"unstable-temporal\": true,\n \"unstable-worker-options\": true,\n \"unstable-sloppy-imports\": true\n }\n}",[160],"1a739988-8830-11f0-a821-0224a6c84d84","README.md","01990bce-c638-714e-97a4-1a67f3821e53",10,["D",1756840062842],"# hello-transcription\n\nReal-time speech transcription using OpenAI's Realtime API - a demonstration of transcription-only mode without AI responses.\n\n## Features\n\n- **Real-time Transcription**: Live speech-to-text conversion as you speak\n- **Multiple Models**: Choose between GPT-4o Transcribe, GPT-4o Mini Transcribe, or Whisper-1\n- **Language Support**: Transcribe in multiple languages (English, Spanish, French, German, Italian, Portuguese, Chinese, Japanese, Korean)\n- **Voice Activity Detection (VAD)**: Automatic detection of speech segments\n- **Logprobs Support**: Optional confidence scores for transcriptions\n- **Split View**: See transcriptions and event logs side-by-side\n\n## How It Works\n\nThis app uses OpenAI's Realtime API in transcription-only mode:\n1. Your voice is captured via WebRTC\n2. Audio is streamed to OpenAI's transcription service\n3. Transcriptions are returned in real-time\n4. No AI responses are generated (transcription only)\n\n## Model Differences\n\n- **GPT-4o Transcribe**: Streaming transcription with incremental updates\n- **GPT-4o Mini Transcribe**: Smaller model, streaming transcription \n- **Whisper-1**: Complete transcription after each speech segment (no streaming)\n\n## Configuration Options\n\n- **Model**: Select the transcription model\n- **Language**: Choose the primary language for better accuracy\n- **VAD**: Enable/disable automatic voice activity detection\n- **Logprobs**: Include confidence scores (for advanced use)\n\n## Usage\n\n1. Click \"Start\" to begin transcription\n2. Allow microphone access when prompted\n3. Start speaking - transcriptions appear in real-time\n4. Partial transcriptions update as you speak\n5. Final transcriptions are marked in green\n6. Click \"Stop\" to end the session\n\n## API Endpoints\n\n- `GET /` - Serves the transcription interface\n- `POST /rtc` - Creates a WebRTC transcription session\n- `POST /observer/:callId` - WebSocket observer for transcription events\n\n## Environment Variables\n\nSet in your Val Town environment:\n- `OPENAI_API_KEY` - Your OpenAI API key (required)\n\n## Local Development\n\n```bash\n# Install Deno\ncurl -fsSL https://deno.land/install.sh | sh\n\n# Run locally\ndeno run --allow-all main.tsx\n```\n\n## Val Town Deployment\n\n1. Fork/remix this val on Val Town\n2. Add your `OPENAI_API_KEY` to Val Town secrets\n3. Your app will be available at `https://[your-val-name].val.run`\n\n## Technical Details\n\nThe app uses OpenAI's Realtime API in transcription mode:\n- Session type: `transcription` (not `realtime`)\n- Audio format: PCM16\n- Noise reduction: Near-field (optimized for close microphones)\n- WebRTC data channel for receiving transcription events\n\n## Event Flow\n\n1. `input_audio_buffer.committed` - Audio chunk received\n2. `conversation.item.input_audio_transcription.delta` - Partial transcription\n3. `conversation.item.input_audio_transcription.completed` - Final transcription\n\n## Credits\n\nBuilt with OpenAI's Realtime API for transcription-only use cases.",[166],"1a3463bc-8830-11f0-a44c-0224a6c84d84","CLAUDE.md","1a34648e-8830-11f0-a44c-0224a6c84d84",2,["D",1756840062786],"# Hello-Transcription - OpenAI Realtime API Transcription Demo\n\n## 🎯 Project Overview\n\nHello-Transcription demonstrates the transcription-only mode of OpenAI's Realtime API. Unlike the conversational mode, this implementation focuses purely on speech-to-text conversion without generating AI responses, making it ideal for subtitles, live captions, meeting transcriptions, and other transcription-focused use cases.\n\n**Created:** September 2, 2025 \n**Platform:** Val Town \n**API:** OpenAI Realtime API (Transcription Mode) \n**Key Feature:** Real-time streaming transcription with multiple model support\n\n## 🏗️ Technical Stack\n\n- **Runtime:** Deno (Val Town platform)\n- **Framework:** Hono (lightweight web framework)\n- **Transcription:** OpenAI Realtime API in transcription mode\n- **Connection:** WebRTC with data channel for events\n- **Frontend:** Vanilla JavaScript with split-view interface\n- **Models:** GPT-4o Transcribe, GPT-4o Mini Transcribe, Whisper-1\n\n## 📁 Project Structure\n\n```\nhello-transcription/\n├── frontend/\n│ └── index.html # Split-view transcription interface\n├── routes/\n│ ├── rtc.ts # WebRTC session setup for transcription\n│ ├── observer.ts # WebSocket observer for transcription events\n│ └── utils.ts # Transcription session configuration\n├── main.tsx # Main entry point\n├── deno.json # Deno configuration\n├── README.md # User documentation\n└── CLAUDE.md # This file - technical documentation\n```\n\n## 🔑 Core Concepts\n\n### Transcription vs Conversation Mode\n\nThe Realtime API supports two distinct modes:\n\n1. **Conversation Mode** (`type: \"realtime\"`):\n - Two-way interaction with AI\n - User speaks → AI responds\n - Used in hello-realtime and hello-mcp\n\n2. **Transcription Mode** (`type: \"transcription\"`):\n - One-way speech-to-text only\n - User speaks → Text output\n - No AI responses generated\n - Lower latency, lower cost\n - This demo uses transcription mode\n\n### Transcription Session Object\n\n```typescript\n{\n type: \"transcription\",\n input_audio_format: \"pcm16\",\n input_audio_transcription: {\n model: \"gpt-4o-transcribe\", // or \"gpt-4o-mini-transcribe\", \"whisper-1\"\n prompt: \"\", // Optional context hint\n language: \"en\" // ISO-639-1 language code\n },\n turn_detection: {\n type: \"server_vad\",\n threshold: 0.5,\n prefix_padding_ms: 300,\n silence_duration_ms: 500\n },\n input_audio_noise_reduction: {\n type: \"near_field\" // or \"far_field\", null\n },\n include: [\"item.input_audio_transcription.logprobs\"] // Optional\n}\n```\n\n## 🛠️ Key Components\n\n### 1. Utils Configuration (`/routes/utils.ts`)\n\nHandles transcription session configuration with sensible defaults:\n\n```typescript\nexport function makeTranscriptionSession(config: TranscriptionConfig = {}) {\n const {\n model = \"gpt-4o-transcribe\",\n language = \"en\",\n prompt = \"\",\n enableVAD = true,\n noiseReduction = \"near_field\",\n includeLogprobs = false\n } = config;\n \n // Build session object...\n}\n```\n\n**Key Configuration Options:**\n- **model**: Transcription model selection\n- **language**: Primary language for better accuracy\n- **prompt**: Context hints (e.g., \"Expect medical terminology\")\n- **enableVAD**: Automatic voice activity detection\n- **noiseReduction**: Audio preprocessing type\n- **includeLogprobs**: Confidence scores for words\n\n### 2. RTC Route Handler (`/routes/rtc.ts`)\n\nCreates WebRTC sessions specifically for transcription:\n\n```typescript\n// Get config from query params\nconst model = c.req.query(\"model\") || \"gpt-4o-transcribe\";\nconst language = c.req.query(\"language\") || \"en\";\nconst vad = c.req.query(\"vad\") !== \"false\";\nconst logprobs = c.req.query(\"logprobs\") === \"true\";\n\n// Create transcription session\nconst sessionConfig = makeTranscriptionSession({\n model,\n language,\n enableVAD: vad,\n includeLogprobs: logprobs\n});\n```\n\n**Important:** Uses `type: \"transcription\"` not `type: \"realtime\"`\n\n### 3. Observer WebSocket (`/routes/observer.ts`)\n\nMonitors transcription events via server-side WebSocket:\n\n```typescript\nws.onmessage = (event) => {\n const data = JSON.parse(event.data);\n \n if (data.type === \"conversation.item.input_audio_transcription.delta\") {\n // Streaming partial transcription\n console.log(`📝 Transcription delta: \"${data.delta}\"`);\n } else if (data.type === \"conversation.item.input_audio_transcription.completed\") {\n // Final transcription for segment\n console.log(`✅ Transcription completed: \"${data.transcript}\"`);\n }\n};\n```\n\n### 4. Frontend Interface (`/frontend/index.html`)\n\nSplit-view interface with real-time transcription display:\n\n#### Left Panel - Transcriptions\n- Shows live transcription stream\n- Partial transcriptions update in real-time\n- Final transcriptions marked with green border\n- Each segment timestamped\n\n#### Right Panel - Event Logs\n- Technical event stream\n- Debug information\n- Connection status\n\n#### Data Channel Handling\n```javascript\ndataChannel.onmessage = (event) => {\n const data = JSON.parse(event.data);\n \n if (data.type === \"conversation.item.input_audio_transcription.delta\") {\n // Update partial transcription\n addTranscript(data.item_id, data.delta, false);\n } else if (data.type === \"conversation.item.input_audio_transcription.completed\") {\n // Mark transcription as final\n addTranscript(data.item_id, data.transcript, true);\n }\n};\n```\n\n## 📊 Model Comparison\n\n### GPT-4o Transcribe\n- **Streaming:** Yes - incremental updates via delta events\n- **Latency:** Low\n- **Accuracy:** High\n- **Use Case:** Live subtitles, real-time captions\n\n### GPT-4o Mini Transcribe\n- **Streaming:** Yes - incremental updates\n- **Latency:** Very low\n- **Accuracy:** Good\n- **Use Case:** Quick transcriptions, lower cost\n\n### Whisper-1\n- **Streaming:** No - complete segments only\n- **Latency:** Higher (waits for complete utterance)\n- **Accuracy:** Very high\n- **Use Case:** High-accuracy transcriptions, post-processing\n\n## 🔄 Event Flow\n\n### Transcription Event Sequence\n\n1. **Audio Input**\n ```\n User speaks → Microphone → WebRTC → OpenAI\n ```\n\n2. **VAD Processing** (if enabled)\n ```\n Voice detected → Buffer audio → Silence detected → Commit buffer\n ```\n\n3. **Transcription Events**\n ```\n input_audio_buffer.committed\n ↓\n conversation.item.input_audio_transcription.delta (streaming models)\n ↓\n conversation.item.input_audio_transcription.completed\n ```\n\n### Event Types\n\n#### Delta Event (Streaming)\n```json\n{\n \"type\": \"conversation.item.input_audio_transcription.delta\",\n \"item_id\": \"item_003\",\n \"content_index\": 0,\n \"delta\": \"Hello, how\"\n}\n```\n\n#### Completed Event\n```json\n{\n \"type\": \"conversation.item.input_audio_transcription.completed\",\n \"item_id\": \"item_003\",\n \"content_index\": 0,\n \"transcript\": \"Hello, how are you today?\"\n}\n```\n\n## ⚙️ Configuration Details\n\n### Voice Activity Detection (VAD)\n\nVAD automatically detects speech segments:\n\n```typescript\nturn_detection: {\n type: \"server_vad\",\n threshold: 0.5, // Sensitivity (0-1)\n prefix_padding_ms: 300, // Audio before speech\n silence_duration_ms: 500 // Silence to end segment\n}\n```\n\n**VAD Disabled:**\n```typescript\nturn_detection: null // Manual control required\n```\n\n### Noise Reduction\n\nThree noise reduction modes:\n\n1. **near_field**: Optimized for close microphones (default)\n2. **far_field**: For distant microphones/speakers\n3. **null**: No noise reduction\n\n### Language Configuration\n\nISO-639-1 language codes improve accuracy:\n- \"en\" - English\n- \"es\" - Spanish \n- \"fr\" - French\n- \"de\" - German\n- \"zh\" - Chinese\n- \"ja\" - Japanese\n\n### Logprobs (Confidence Scores)\n\nWhen enabled, provides word-level confidence:\n\n```typescript\ninclude: [\"item.input_audio_transcription.logprobs\"]\n```\n\nReturns probability scores for each transcribed word, useful for:\n- Highlighting uncertain words\n- Quality assessment\n- Post-processing decisions\n\n## 🧪 Testing Guide\n\n### Local Development\n\n1. **Setup Environment**\n ```bash\n # Create .env file\n echo \"OPENAI_API_KEY=sk-...\" > .env\n \n # Install Deno\n curl -fsSL https://deno.land/install.sh | sh\n ```\n\n2. **Run Development Server**\n ```bash\n # With auto-reload\n deno run --watch --allow-all main.tsx\n \n # Or standard\n deno run --allow-all main.tsx\n ```\n\n3. **Test Transcription**\n - Open http://localhost:8000\n - Select model and language\n - Click \"Start\"\n - Speak clearly\n - Watch transcriptions appear\n\n### Testing Different Models\n\n1. **Test Streaming (GPT-4o)**\n - Select \"GPT-4o Transcribe\"\n - Speak continuously\n - Notice incremental updates\n\n2. **Test Non-Streaming (Whisper-1)**\n - Select \"Whisper-1\"\n - Speak, then pause\n - Notice complete segments only\n\n3. **Test VAD**\n - Enable VAD\n - Speak with pauses\n - Notice automatic segmentation\n\n4. **Test Without VAD**\n - Disable VAD\n - Requires manual commit (not implemented in this demo)\n\n## 🐛 Common Issues & Solutions\n\n### Issue: No transcriptions appearing\n**Solutions:**\n- Check microphone permissions\n- Verify OPENAI_API_KEY is set\n- Check browser console for errors\n- Ensure WebRTC connection established\n\n### Issue: Transcriptions cut off mid-sentence\n**Solutions:**\n- Adjust VAD silence_duration_ms (longer = fewer cuts)\n- Try different VAD threshold\n- Consider using different model\n\n### Issue: Poor transcription accuracy\n**Solutions:**\n- Set correct language parameter\n- Use appropriate noise reduction setting\n- Provide context via prompt parameter\n- Try higher accuracy model (Whisper-1)\n\n### Issue: High latency\n**Solutions:**\n- Use GPT-4o Mini for lower latency\n- Check network connection\n- Consider disabling logprobs\n\n## 📈 Performance Characteristics\n\n### Latency Comparison\n- **GPT-4o Mini**: ~100-200ms first token\n- **GPT-4o**: ~150-250ms first token \n- **Whisper-1**: ~500-1000ms (full segment)\n\n### Throughput\n- Handles real-time audio (16kHz PCM16)\n- Multiple concurrent sessions supported\n- No buffering required for streaming models\n\n### Cost Optimization\n- Transcription-only mode is cheaper than conversation mode\n- No AI inference costs\n- GPT-4o Mini most cost-effective\n- Whisper-1 for batch/accuracy needs\n\n## 🔒 Security Considerations\n\n### Current Implementation\n- API key stored in environment variable\n- No authentication on endpoints\n- No rate limiting\n- Single-tenant design\n\n### Production Recommendations\n1. **Authentication**: Add user authentication\n2. **Rate Limiting**: Implement per-user limits\n3. **CORS**: Configure appropriate origins\n4. **Monitoring**: Track usage and errors\n5. **Encryption**: Ensure HTTPS only\n6. **Token Security**: Use short-lived tokens\n\n## 🚀 Deployment\n\n### Val Town Deployment\n\n1. **Create/Remix Val**\n ```bash\n vt remix emcho/hello-transcription my-transcription\n ```\n\n2. **Set Environment**\n - Add `OPENAI_API_KEY` in Val Town secrets\n\n3. **Deploy**\n ```bash\n vt push\n ```\n\n4. **Access**\n - URL: `https://[your-val-name].val.run`\n\n### Environment Variables\n- `OPENAI_API_KEY` - Required for OpenAI API access\n\n## 📝 Future Enhancements\n\n### Potential Features\n\n1. **Recording & Export**\n - Save transcriptions to file\n - Export as SRT/VTT subtitles\n - Download audio recordings\n\n2. **Advanced Controls**\n - Manual VAD control\n - Custom VAD parameters UI\n - Prompt templates\n\n3. **Multi-Stream**\n - Multiple speaker support\n - Speaker diarization\n - Parallel transcriptions\n\n4. **Post-Processing**\n - Punctuation enhancement\n - Grammar correction\n - Translation\n\n5. **Visualization**\n - Audio waveform display\n - VAD activity indicator\n - Confidence heat map\n\n6. **Integration**\n - Webhook support\n - Real-time API streaming\n - Database storage\n\n## 🔗 References\n\n### Documentation\n- [OpenAI Realtime Transcription Guide](https://platform.openai.com/docs/guides/realtime-transcription)\n- [Realtime API Reference](https://platform.openai.com/docs/api-reference/realtime)\n- [Voice Activity Detection Guide](https://platform.openai.com/docs/guides/realtime-vad)\n- [Val Town Documentation](https://docs.val.town)\n\n### Related Projects\n- [hello-realtime](https://hello-realtime.val.run) - Conversation mode demo\n- [hello-mcp](https://github.com/hello-mcp) - MCP tool execution demo\n\n### Key Differences from Conversation Mode\n1. Session type: `transcription` vs `realtime`\n2. No AI responses generated\n3. Different event types\n4. Lower latency and cost\n5. Focused on speech-to-text only\n\n## 💡 Implementation Notes\n\n### Critical Discoveries\n1. **Session Type**: Must use `type: \"transcription\"` not `type: \"realtime\"`\n2. **Event Names**: Different from conversation mode events\n3. **Model Behavior**: Whisper doesn't stream, GPT-4o models do\n4. **VAD Impact**: Significantly affects transcription segmentation\n5. **Language Setting**: Dramatically improves accuracy for non-English\n\n### Best Practices\n1. Always specify language for non-English content\n2. Use near_field noise reduction for headset mics\n3. Enable VAD for natural speech segmentation\n4. Choose model based on latency vs accuracy needs\n5. Monitor item_id for proper segment ordering\n\n## 🎯 Summary\n\nHello-Transcription successfully demonstrates the transcription-only capabilities of OpenAI's Realtime API. Key achievements:\n\n1. **Pure Transcription**: No AI responses, focused solely on speech-to-text\n2. **Model Flexibility**: Support for three different transcription models\n3. **Real-time Streaming**: Live transcription updates for supported models\n4. **Configuration Options**: VAD, noise reduction, language, logprobs\n5. **Clean Interface**: Split-view design for transcriptions and logs\n\nThis implementation serves as a foundation for building transcription-focused applications like live captioning, meeting transcription, subtitle generation, and accessibility tools.",[173],"50a3e2c0-882f-11f0-9954-0224a6c84d84",".vtignore","01990bce-c638-714e-97a4-157f5fb8f430",["D",1756839724543],".git\n.vscode\n.cursorrules\n.DS_Store\nnode_modules\nvendor",[180],"50a3e18a-882f-11f0-9954-0224a6c84d84","routes","01990bce-c638-714e-97a4-1048c460355e",["D",1756839724543],"directory",[186],"50a3d50a-882f-11f0-9954-0224a6c84d84","frontend","01990bce-c637-779b-aded-96ab9fc690df",["D",1756839724543],[192],"38dfaf46-8935-11f0-9d99-0224a6c84d84","docs","38dfaffa-8935-11f0-9d99-0224a6c84d84",22,["D",1756952212884],[197],"38d12796-8935-11f0-b12f-0224a6c84d84",".claude","38d1285e-8935-11f0-b12f-0224a6c84d84",21,["D",1756952212789],[203],"pulls","branches","forks","envVars",{"_215":154,"_217":60},"connections",{"_215":60,"_216":60},"count","unfilledCount","emptyCount",{"_41":202,"_31":203,"_29":204,"_150":-5,"_39":40,"_38":205,"_152":56,"_36":321,"_13":60,"_33":189,"_34":-5,"_23":24,"_157":-5,"_236":322,"_26":323},{"_41":313,"_31":314,"_29":315,"_150":204,"_39":40,"_38":316,"_152":56,"_36":317,"_13":60,"_33":20,"_34":318,"_23":24,"_157":-5,"_236":319,"_26":320},{"_41":196,"_31":197,"_29":198,"_150":-5,"_39":40,"_38":199,"_152":56,"_36":310,"_13":60,"_33":189,"_34":-5,"_23":24,"_157":-5,"_236":311,"_26":312},{"_41":306,"_31":301,"_29":295,"_150":198,"_39":40,"_38":132,"_152":56,"_36":307,"_13":60,"_33":189,"_34":-5,"_23":24,"_157":-5,"_236":308,"_26":309},{"_41":302,"_31":32,"_29":30,"_150":295,"_39":40,"_38":14,"_152":56,"_36":303,"_13":60,"_33":20,"_34":35,"_23":24,"_157":-5,"_236":304,"_26":305},{"_41":292,"_31":293,"_29":294,"_150":295,"_39":40,"_38":296,"_152":56,"_36":297,"_13":60,"_33":20,"_34":298,"_23":24,"_157":-5,"_236":299,"_26":300},{"_41":191,"_31":192,"_29":193,"_150":-5,"_39":40,"_38":154,"_152":56,"_36":289,"_13":60,"_33":189,"_34":-5,"_23":24,"_157":-5,"_236":290,"_26":291},{"_41":280,"_31":281,"_29":282,"_150":193,"_39":40,"_38":283,"_152":24,"_36":284,"_13":285,"_33":20,"_34":286,"_23":24,"_157":-5,"_236":287,"_26":288},{"_41":185,"_31":186,"_29":187,"_150":-5,"_39":40,"_38":154,"_152":56,"_36":277,"_13":60,"_33":189,"_34":-5,"_23":24,"_157":-5,"_236":278,"_26":279},{"_41":269,"_31":270,"_29":271,"_150":187,"_39":40,"_38":272,"_152":24,"_36":273,"_13":154,"_33":257,"_34":274,"_23":24,"_157":-5,"_236":275,"_26":276},{"_41":261,"_31":262,"_29":263,"_150":187,"_39":40,"_38":264,"_152":24,"_36":265,"_13":154,"_33":257,"_34":266,"_23":24,"_157":-5,"_236":267,"_26":268},{"_41":251,"_31":252,"_29":253,"_150":187,"_39":40,"_38":254,"_152":24,"_36":255,"_13":256,"_33":257,"_34":258,"_23":24,"_157":-5,"_236":259,"_26":260},{"_41":179,"_31":180,"_29":181,"_150":-5,"_39":40,"_38":154,"_152":56,"_36":248,"_13":60,"_33":20,"_34":183,"_23":24,"_157":-5,"_236":249,"_26":250},{"_41":172,"_31":173,"_29":174,"_150":-5,"_39":40,"_38":175,"_152":56,"_36":245,"_13":60,"_33":20,"_34":177,"_23":24,"_157":-5,"_236":246,"_26":247},{"_41":165,"_31":166,"_29":167,"_150":-5,"_39":40,"_38":168,"_152":56,"_36":242,"_13":154,"_33":20,"_34":170,"_23":24,"_157":-5,"_236":243,"_26":244},{"_41":159,"_31":160,"_29":161,"_150":-5,"_39":40,"_38":154,"_152":56,"_36":239,"_13":60,"_33":20,"_34":163,"_23":24,"_157":-5,"_236":240,"_26":241},{"_41":148,"_31":149,"_29":54,"_150":-5,"_39":40,"_38":151,"_152":24,"_36":235,"_13":154,"_33":155,"_34":156,"_23":24,"_157":-5,"_236":237,"_26":238},["D",1756840062830],"children",[],[149],["D",1756839724543],[],[160],["D",1756840062842],[],[166],["D",1756840062786],[],[173],["D",1756839724543],[],[180],"c3998b74-8837-11f0-9cba-0224a6c84d84","utils.ts","01990bce-c638-714e-97a4-0caf2e9a1c1c",19,["D",1756843353453],8,"script","const OPENAI_API_KEY = Deno.env.get(\"OPENAI_API_KEY\");\nif (!OPENAI_API_KEY) {\n throw new Error(\"🔴 OpenAI API key not configured\");\n}\n\nexport function makeHeaders(contentType?: string) {\n const obj: Record = {\n Authorization: `Bearer ${OPENAI_API_KEY}`,\n };\n if (contentType) obj[\"Content-Type\"] = contentType;\n return obj;\n}\n\nexport interface TranscriptionConfig {\n model?: \"gpt-4o-transcribe\" | \"gpt-4o-mini-transcribe\" | \"whisper-1\";\n language?: string;\n prompt?: string;\n enableVAD?: boolean;\n noiseReduction?: \"near_field\" | \"far_field\" | null;\n includeLogprobs?: boolean;\n}\n\nexport function makeTranscriptionSession(config: TranscriptionConfig = {}) {\n const {\n model = \"gpt-4o-transcribe\",\n language = \"en\",\n prompt = \"\",\n enableVAD = true,\n noiseReduction = \"near_field\",\n includeLogprobs = false\n } = config;\n\n // Create a basic realtime session first\n const session: any = {\n type: \"realtime\",\n model: \"gpt-realtime\",\n instructions: \"You are a transcription service. Transcribe what the user says without responding. Do not generate any audio responses.\"\n };\n\n // Configure VAD if enabled\n if (enableVAD) {\n session.turn_detection = {\n type: \"server_vad\",\n threshold: 0.5,\n prefix_padding_ms: 300,\n silence_duration_ms: 500\n };\n } else {\n session.turn_detection = null;\n }\n\n return session;\n}\n\nexport function makeSessionUpdateMessage(config: TranscriptionConfig = {}) {\n const {\n model = \"gpt-4o-transcribe\",\n language = \"en\",\n prompt = \"\",\n enableVAD = true,\n noiseReduction = \"near_field\",\n includeLogprobs = false\n } = config;\n\n // Create session update message to enable transcription\n const update: any = {\n type: \"session.update\",\n input_audio_transcription: {\n model,\n prompt,\n language\n }\n };\n\n // Add include for logprobs if requested\n if (includeLogprobs) {\n update.include = [\"item.input_audio_transcription.logprobs\"];\n }\n\n return update;\n}",[],[186,252],"1a4d460c-8830-11f0-bfd2-0224a6c84d84","rtc.ts","01990bce-c638-714e-97a4-07196c7003e4",5,["D",1756840062844],"import { Hono } from \"npm:hono\";\nimport { makeHeaders, makeTranscriptionSession } from \"./utils.ts\";\n\nconst rtc = new Hono();\n\n// POST /rtc : create a new transcription call\nrtc.post(\"/\", async (c) => {\n // Get transcription configuration from query params\n const model = c.req.query(\"model\") as any || \"gpt-4o-transcribe\";\n const language = c.req.query(\"language\") || \"en\";\n const vad = c.req.query(\"vad\") !== \"false\"; // default true\n const logprobs = c.req.query(\"logprobs\") === \"true\"; // default false\n \n // Create the transcription session config\n const sessionConfig = makeTranscriptionSession({\n model,\n language,\n enableVAD: vad,\n includeLogprobs: logprobs\n });\n\n // Create the call with transcription session\n const url = \"https://api.openai.com/v1/realtime/calls\";\n const headers = makeHeaders();\n const fd = new FormData();\n fd.set(\"sdp\", await c.req.text());\n fd.set(\"session\", JSON.stringify(sessionConfig));\n \n const opts = { method: \"POST\", headers, body: fd };\n const resp = await fetch(url, opts);\n \n if (!resp.ok) {\n const errText = await resp.text().catch(() => \"\");\n console.error(`🔴 start transcription call failed: ${resp.status} ${errText}`);\n return c.text(\"Internal error\", 500);\n }\n\n const contentType = resp.headers.get(\"Content-Type\");\n const location = resp.headers.get(\"Location\");\n const callId = location?.split(\"/\").pop();\n console.log(\"✅ transcription call created:\", callId);\n\n // Kick off observer in the background to monitor transcription events\n const selfUrl = new URL(c.req.url);\n fetch(`${selfUrl.origin}/observer/${callId}`, { method: \"POST\" })\n .catch(() => {}); // catch to not crash the process\n\n return c.newResponse(resp.body, {\n headers: { \"Content-Type\": contentType },\n });\n});\n\nexport default rtc;",[],[186,262],"1a514ed2-8830-11f0-8241-0224a6c84d84","observer.ts","01990bce-c638-714e-97a4-029a75bebd75",6,["D",1756840062935],"import { Hono } from \"npm:hono\";\nimport { makeHeaders } from \"./utils.ts\";\n\nconst observer = new Hono();\n\n// POST /observer/:callId : establish websocket to observe transcription events\nobserver.post(\"/:callId\", async (c) => {\n const callId = c.req.param(\"callId\");\n console.log(`📡 Starting observer for transcription call: ${callId}`);\n \n // Connect to OpenAI WebSocket for this call\n const wsUrl = `wss://api.openai.com/v1/realtime?call_id=${callId}`;\n const ws = new WebSocket(wsUrl, {\n headers: makeHeaders(),\n });\n\n ws.onopen = () => {\n console.log(`✅ Observer connected for call: ${callId}`);\n };\n\n ws.onmessage = (event) => {\n try {\n const data = JSON.parse(event.data);\n \n // Log transcription-specific events\n if (data.type === \"conversation.item.input_audio_transcription.delta\") {\n console.log(`📝 Transcription delta [${data.item_id}]: \"${data.delta}\"`);\n } else if (data.type === \"conversation.item.input_audio_transcription.completed\") {\n console.log(`✅ Transcription completed [${data.item_id}]: \"${data.transcript}\"`);\n } else if (data.type === \"input_audio_buffer.committed\") {\n console.log(`🎤 Audio buffer committed [${data.item_id}]`);\n } else if (data.type === \"error\") {\n console.error(`❌ Error: ${data.error?.message || JSON.stringify(data.error)}`);\n }\n \n // Log other important events at debug level\n if (data.type?.startsWith(\"transcription_session\")) {\n console.log(`🔧 Session event: ${data.type}`);\n }\n } catch (e) {\n console.error(\"Failed to parse WebSocket message:\", e);\n }\n };\n\n ws.onerror = (error) => {\n console.error(`❌ WebSocket error for call ${callId}:`, error);\n };\n\n ws.onclose = () => {\n console.log(`👋 Observer disconnected for call: ${callId}`);\n };\n\n // Return immediately (observer runs in background)\n return c.text(\"Observer started\", 200);\n});\n\nexport default observer;",[],[186,270],["D",1756839724543],[227,228,229],[186],"c3a04d42-8837-11f0-b042-0224a6c84d84","index.html","01990bce-c637-779b-aded-90c65c11e441",20,["D",1756843353462],4,"\n\n \n \n \n OpenAI Realtime Transcription\n \n \n \n \n\n

Transcription

\n Start speaking to see transcription here...\n

Event Logs

\n\n

\n\n \n \n",[],[192,281],["D",1756839724543],[225],[192],"3907d908-8935-11f0-9ad7-0224a6c84d84","realtime-webrtc.md","3907d9a8-8935-11f0-9ad7-0224a6c84d84","38edc7a2-8935-11f0-98af-0224a6c84d84",25,["D",1756952213152],"Realtime API with WebRTC\n========================\n\nConnect to the Realtime API using WebRTC.\n\n[WebRTC](https://webrtc.org/) is a powerful set of standard interfaces for building real-time applications. The OpenAI Realtime API supports connecting to realtime models through a WebRTC peer connection.\n\nFor browser-based speech-to-speech voice applications, we recommend starting with the [Agents SDK for TypeScript](https://openai.github.io/openai-agents-js/guides/voice-agents/quickstart/), which provides higher-level helpers and APIs for managing Realtime sessions. The WebRTC interface is powerful and flexible, but lower level than the Agents SDK.\n\nWhen connecting to a Realtime model from the client (like a web browser or mobile device), we recommend using WebRTC rather than WebSocket for more consistent performance.\n\nOverview\n--------\n\nConnecting to the Realtime API via WebRTC requires a backend server that can generate an ephemeral API key that can be safely used in a client-side environment (like a browser).\n\nThe process for initializing a WebRTC connection using an ephemeral API key is as follows:\n\n1. A browser makes a request to a developer-controlled server to mint an ephemeral API key.\n2. The developer's server uses a [standard API key](/settings/organization/api-keys) to request an ephemeral key from the [OpenAI REST API](/docs/api-reference/realtime-sessions), and returns that new key to the browser.\n3. The browser uses the ephemeral key to authenticate a session directly with the OpenAI Realtime API as a [WebRTC peer connection](https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection).\n\n![connect to realtime via WebRTC](https://openaidevs.retool.com/api/file/55b47800-9aaf-48b9-90d5-793ab227ddd3)\n\nInitializing a peer connection\n------------------------------\n\nIn the browser, you can use standard WebRTC APIs to initialize a peer connection object. The code below requires a server-side endpoint that generates an ephemeral API token (which we'll see code for in a moment).\n\n```javascript\n// Get a session token for OpenAI Realtime API\nconst tokenResponse = await fetch(\"/token\");\nconst data = await tokenResponse.json();\nconst EPHEMERAL_KEY = data.value;\n\n// Create a peer connection\nconst pc = new RTCPeerConnection();\n\n// Set up to play remote audio from the model\naudioElement.current = document.createElement(\"audio\");\naudioElement.current.autoplay = true;\npc.ontrack = (e) => (audioElement.current.srcObject = e.streams[0]);\n\n// Add local audio track for microphone input in the browser\nconst ms = await navigator.mediaDevices.getUserMedia({\n audio: true,\n});\npc.addTrack(ms.getTracks()[0]);\n\n// Set up data channel for sending and receiving events\nconst dc = pc.createDataChannel(\"oai-events\");\n\n// Start the session using the Session Description Protocol (SDP)\nconst offer = await pc.createOffer();\nawait pc.setLocalDescription(offer);\n\nconst baseUrl = \"https://api.openai.com/v1/realtime/calls\";\nconst model = \"gpt-realtime\";\nconst sdpResponse = await fetch(`${baseUrl}?model=${model}`, {\n method: \"POST\",\n body: offer.sdp,\n headers: {\n Authorization: `Bearer ${EPHEMERAL_KEY}`,\n \"Content-Type\": \"application/sdp\",\n },\n});\n\nconst answer = {\n type: \"answer\",\n sdp: await sdpResponse.text(),\n};\nawait pc.setRemoteDescription(answer);\n```\n\nThe WebRTC APIs provide rich controls for handling media streams and input devices. For more guidance on building user interfaces on top of WebRTC, [refer to the docs on MDN](https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API).\n\nSee the next section for details on implementing the server-side component that powers the `/token` endpoint used in the code above.\n\nCreating an ephemeral token\n---------------------------\n\nTo create an ephemeral token to use on the client-side, you will need to build a small server-side application (or integrate with an existing one) to make an [OpenAI REST API](/docs/api-reference/realtime-sessions) request for an ephemeral key. You will use a [standard API key](/settings/organization/api-keys) to authenticate this request on your backend server.\n\nBelow is an example of a simple Node.js [express](https://expressjs.com/) server which mints an ephemeral API key using the REST API:\n\n```javascript\nimport express from \"express\";\n\nconst app = express();\n\nconst sessionConfig = JSON.stringify({\n session: {\n type: \"realtime\",\n model: \"gpt-realtime\",\n audio: {\n output: {\n voice: \"marin\",\n },\n },\n },\n});\n\n// An endpoint which would work with the client code above - it returns\n// the contents of a REST API request to this protected endpoint\napp.get(\"/token\", async (req, res) => {\n try {\n const response = await fetch(\n \"https://api.openai.com/v1/realtime/client_secrets\",\n {\n method: \"POST\",\n headers: {\n Authorization: `Bearer ${apiKey}`,\n \"Content-Type\": \"application/json\",\n },\n body: sessionConfig,\n }\n );\n\n const data = await response.json();\n res.json(data);\n } catch (error) {\n console.error(\"Token generation error:\", error);\n res.status(500).json({ error: \"Failed to generate token\" });\n }\n});\n\napp.listen(3000);\n```\n\nYou can create a server endpoint like this one on any platform that can send and receive HTTP requests. Just ensure that **you only use standard OpenAI API keys on the server, not in the browser.**\n\nSending and receiving events\n----------------------------\n\nRealtime API sessions are managed using a combination of [client-sent events](/docs/api-reference/realtime_client_events/session) emitted by you as the developer, and [server-sent events](/docs/api-reference/realtime_server_events/error) created by the Realtime API to indicate session lifecycle events.\n\nWhen connecting to a Realtime model via WebRTC, you will not have to handle audio events to hear model responses in the same granular way you must with [WebSockets](/docs/guides/realtime-websocket). The WebRTC peer connection object, if configured as above, will do much of that work for you.\n\nTo send and receive other client and server events, you will use the WebRTC peer connection's [data channel](https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API/Using_data_channels).\n\n```javascript\n// This is the data channel set up in the browser code above...\nconst dc = pc.createDataChannel(\"oai-events\");\n\n// Listen for server events\ndc.addEventListener(\"message\", (e) => {\n const event = JSON.parse(e.data);\n console.log(event);\n});\n\n// Send client events\nconst event = {\n type: \"conversation.item.create\",\n item: {\n type: \"message\",\n role: \"user\",\n content: [\n {\n type: \"input_text\",\n text: \"hello there!\",\n },\n ],\n },\n};\ndc.send(JSON.stringify(event));\n```\n\nTo learn more about managing Realtime conversations, refer to the [Realtime conversations guide](/docs/guides/realtime-conversations).\n\n[\n\nRealtime Console\n\nCheck out the WebRTC Realtime API in this light weight example app.\n\n/Users/emcho/devel/openai-realtime-console\n\nWas this page useful?",[],[197,301,293],"guides","397729b6-8935-11f0-9096-0224a6c84d84",["D",1756952213233],[],[197,301,32],"38edc6ee-8935-11f0-98af-0224a6c84d84",["D",1756952212978],[222,223],[197,301],["D",1756952212884],[221],[197],"390439f6-8935-11f0-939c-0224a6c84d84","settings.local.json","39043ac8-8935-11f0-939c-0224a6c84d84",24,["D",1756952213071],"{\n \"permissions\": {\n \"allow\": [\n \"Read(//Users/emcho/devel/juberti-demos/realtime/**)\"\n ],\n \"deny\": [],\n \"ask\": []\n }\n}",[],[203,314],["D",1756952212789],[219],[203],["D",1756839724543],{"_97":96,"_104":100},{"_97":96,"_98":-5,"_99":100,"_18":52,"_101":102,"_103":24},{"_13":83,"_31":84,"_41":40,"_85":42,"_86":87,"_88":89,"_90":48,"_91":92,"_93":62,"_94":43,"_95":96},"ctx",{"_76":-5,"_126":361,"_362":363,"_364":365,"_366":24,"_367":24,"_368":24,"_369":56,"_370":-5,"_371":-7,"_372":-7,"_373":60,"_374":24,"_375":24,"_376":-5,"_377":24,"_378":24,"_379":24,"_380":-5},"ENV",{"_332":333,"_334":335,"_336":84,"_337":338,"_339":340,"_341":342,"_343":344,"_345":346,"_347":348,"_349":350,"_351":352,"_353":354,"_355":356,"_357":358,"_359":360},"REACT_APP_SENTRY_DSN","https://586020d86f05489db63a554690acf1e9@o1398359.ingest.sentry.io/6724499","RENDER_GIT_COMMIT","e87eb04e707cf64e7e532c8e8ec395f209e17f15","RENDER_GIT_BRANCH","POSTHOG_API_TOKEN","phc_nmoyuAkStEZuUp69rYlg1DZgrIt19Gx0Y4Hwf8tGgJP","POSTHOG_API_HOST","https://wart.val.town","WEB_API_TEMPLATE","https://{val}.web.val.run","APP_URL","https://www.val.town","MODULE_URL","https://esm.town","VANITY_WEB_API_TEMPLATE","https://{label}.val.run/","DENOLS_URL","https://lsp.val.town","NODE_ENV","production","CLOUDFLARE_TURNSTILE_SITE_KEY","0x4AAAAAACtaIZJH7s9rBMZ6","GOOGLE_CLIENT_ID","875989696899-jtsqkhlbdv5jrlsk8631p7ipjg3c6o62.apps.googleusercontent.com","STRIPE_PUBLISHABLE_KEY","pk_live_51N2zvWEpZTTLipIkVEwXrUG9AT0yeOOEYxszarZT1jpsZV58Ly6wlg3SltZpHN26CUalOBmUSY7adQRQiSkCZz2500qSXNkf8Y","API_URL","https://api.val.town",[],"preferencesKeybindings","default","preferencesTheme","light","preferencesFixers","preferencesCopilot","preferencesSelectionAi","preferencesCloseBrackets","preferencesChatPanelState","preferencesDprint","preferencesDenofmt","notificationCount","flagDenols","isBusinessOrgMember","businessOrgHandle","isBusinessOrgAdmin","isHigherTownieTier","navCollapsed","currentOrgId"]

Diff · v26