Search
Code3,172
brian@airbnb.com,Brian Cheskydrew@dropbox.com,Drew Houstonsam@openai.com,Sam Altmantim@apple.com,Tim Cookjeff@amazon.com,Jeff Bezos
Note: When changing a SQLite table's schema, change the table's name (e.g., add _2 or _3) to create a fresh table.### OpenAI```tsimport { OpenAI } from "https://esm.town/v/std/openai";const openai = new OpenAI();const completion = await openai.chat.completions.create({ messages: [ { role: "user", content: "Say hello in a creative way" },
Note: When changing a SQLite table's schema, change the table's name (e.g., add _2 or _3) to create a fresh table.### OpenAI```tsimport { OpenAI } from "https://esm.town/v/std/openai";const openai = new OpenAI();const completion = await openai.chat.completions.create({ messages: [ { role: "user", content: "Say hello in a creative way" },
In addition to tools you make available to the model with [function calling](/docs/guides/function-calling), you can give models new capabilities using **connectors** and **remote MCP servers**. These tools give the model the ability to connect to and control external services when needed to respond to a user's prompt. These tool calls can either be allowed automatically, or restricted with explicit approval required by you as the developer.* **Connectors** are OpenAI-maintained MCP wrappers for popular services like Google Workspace or Dropbox, like the connectors available in [ChatGPT](https://chatgpt.com).* **Remote MCP servers** can be any server on the public Internet that implements a remote [Model Context Protocol](https://modelcontextprotocol.io/introduction) (MCP) server.```bashcurl https://api.openai.com/v1/responses \-H "Content-Type: application/json" \-H "Authorization: Bearer $OPENAI_API_KEY" \-d '{ "model": "gpt-5",```javascriptimport OpenAI from "openai";const client = new OpenAI();const resp = await client.responses.create({```pythonfrom openai import OpenAIclient = OpenAI()resp = client.responses.create(```bashcurl https://api.openai.com/v1/responses \-H "Content-Type: application/json" \-H "Authorization: Bearer $OPENAI_API_KEY" \-d '{ "model": "gpt-5",```javascriptimport OpenAI from "openai";const client = new OpenAI();const resp = await client.responses.create({```pythonfrom openai import OpenAIclient = OpenAI()resp = client.responses.create(```bashcurl https://api.openai.com/v1/responses \-H "Content-Type: application/json" \-H "Authorization: Bearer $OPENAI_API_KEY" \-d '{ "model": "gpt-5",```javascriptimport OpenAI from "openai";const client = new OpenAI();const resp = await client.responses.create({```pythonfrom openai import OpenAIclient = OpenAI()resp = client.responses.create(#### ApprovalsBy default, OpenAI will request your approval before any data is shared with a connector or remote MCP server. Approvals help you maintain control and visibility over what data is being sent to an MCP server. We highly recommend that you carefully review (and optionally log) all data being shared with a remote MCP server. A request for an approval to make an MCP tool call creates a `mcp_approval_request` item in the Response's output that looks like this:```json```bashcurl https://api.openai.com/v1/responses \-H "Content-Type: application/json" \-H "Authorization: Bearer $OPENAI_API_KEY" \-d '{ "model": "gpt-5",```javascriptimport OpenAI from "openai";const client = new OpenAI();const resp = await client.responses.create({```pythonfrom openai import OpenAIclient = OpenAI()resp = client.responses.create(```bashcurl https://api.openai.com/v1/responses \-H "Content-Type: application/json" \-H "Authorization: Bearer $OPENAI_API_KEY" \-d '{ "model": "gpt-5",```javascriptimport OpenAI from "openai";const client = new OpenAI();const resp = await client.responses.create({```pythonfrom openai import OpenAIclient = OpenAI()resp = client.responses.create(```bashcurl https://api.openai.com/v1/responses \-H "Content-Type: application/json" \-H "Authorization: Bearer $OPENAI_API_KEY" \-d '{ "model": "gpt-5",```javascriptimport OpenAI from "openai";const client = new OpenAI();const resp = await client.responses.create({```pythonfrom openai import OpenAIclient = OpenAI()resp = client.responses.create(The Responses API has built-in support for a limited set of connectors to third-party services. These connectors let you pull in context from popular applications, like Dropbox and Gmail, to allow the model to interact with popular services.Connectors can be used in the same way as remote MCP servers. Both let an OpenAI model access additional third-party tools in an API request. However, instead of passing a `server_url` as you would to call a remote MCP server, you pass a `connector_id` which uniquely identifies a connector available in the API.### Available connectors```bashcurl https://api.openai.com/v1/responses \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "gpt-5",```javascriptimport OpenAI from "openai";const client = new OpenAI();const resp = await client.responses.create({```pythonfrom openai import OpenAIclient = OpenAI()resp = client.responses.create(----------------The MCP tool permits you to connect OpenAI models to external services. This is a powerful feature that comes with some risks.For connectors, there is a risk of potentially sending sensitive data to OpenAI, or allowing models read access to potentially sensitive data in those services.Remote MCP servers carry those same risks, but also have not been verified by OpenAI. These servers can allow models to access, send, and receive data, and take action in these services. All MCP servers are third-party services that are subject to their own terms and conditions.If you come across a malicious MCP server, please report it to `security@openai.com`.Below are some best practices to consider when integrating connectors and remote MCP servers.We also recommend logging any data sent to MCP servers. If you're using the Responses API with `store=true`, these data are already logged via the API for 30 days unless Zero Data Retention is enabled for your organization. You may also want to log these data in your own systems and perform periodic reviews on this to ensure data is being shared per your expectations.Malicious MCP servers may include hidden instructions (prompt injections) designed to make OpenAI models behave unexpectedly. While OpenAI has implemented built-in safeguards to help detect and block these threats, it's essential to carefully review inputs and outputs, and ensure connections are established only with trusted servers.MCP servers may update tool behavior unexpectedly, potentially leading to unintended or malicious behavior.The MCP tool is compatible with Zero Data Retention and Data Residency, but it's important to note that MCP servers are third-party services, and data sent to an MCP server is subject to their data retention and data residency policies.In other words, if you're an organization with Data Residency in Europe, OpenAI will limit inference and storage of Customer Content to take place in Europe up until the point communication or data is sent to the MCP server. It is your responsibility to ensure that the MCP server also adheres to any Zero Data Retention or Data Residency requirements you may have. Learn more about Zero Data Retention and Data Residency [here](/docs/guides/your-data).Usage notes
Connect to the Realtime API using WebSockets on a server.[WebSockets](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) are a broadly supported API for realtime data transfer, and a great choice for connecting to the OpenAI Realtime API in server-to-server applications. For browser and mobile clients, we recommend connecting via [WebRTC](/docs/guides/realtime-webrtc).In a server-to-server integration with Realtime, your backend system will connect via WebSocket directly to the Realtime API. You can use a [standard API key](/settings/organization/api-keys) to authenticate this connection, since the token will only be available on your secure backend server.Connect via WebSocket---------------------Below are several examples of connecting via WebSocket to the Realtime API. In addition to using the WebSocket URL below, you will also need to pass an authentication header using your OpenAI API key.It is possible to use WebSocket in browsers with an ephemeral API token as shown in the [WebRTC connection guide](/docs/guides/realtime-webrtc), but if you are connecting from a client like a browser or mobile app, WebRTC will be a more robust solution in most cases.import WebSocket from "ws";const url = "wss://api.openai.com/v1/realtime?model=gpt-realtime";const ws = new WebSocket(url, { headers: { Authorization: "Bearer " + process.env.OPENAI_API_KEY, },});import websocketOPENAI_API_KEY = os.environ.get("OPENAI_API_KEY")url = "wss://api.openai.com/v1/realtime?model=gpt-realtime"headers = ["Authorization: Bearer " + OPENAI_API_KEY]def on_open(ws):const ws = new WebSocket( "wss://api.openai.com/v1/realtime?model=gpt-realtime", [ "realtime", // Auth "openai-insecure-api-key." + OPENAI_API_KEY, // Optional "openai-organization." + OPENAI_ORG_ID, "openai-project." + OPENAI_PROJECT_ID, ]);import WebSocket from "ws";const url = "wss://api.openai.com/v1/realtime?model=gpt-realtime";const ws = new WebSocket(url, { headers: { Authorization: "Bearer " + process.env.OPENAI_API_KEY, },});
Connect to the Realtime API using WebRTC.[WebRTC](https://webrtc.org/) is a powerful set of standard interfaces for building real-time applications. The OpenAI Realtime API supports connecting to realtime models through a WebRTC peer connection.For browser-based speech-to-speech voice applications, we recommend starting with the [Agents SDK for TypeScript](https://openai.github.io/openai-agents-js/guides/voice-agents/quickstart/), which provides higher-level helpers and APIs for managing Realtime sessions. The WebRTC interface is powerful and flexible, but lower level than the Agents SDK.When connecting to a Realtime model from the client (like a web browser or mobile device), we recommend using WebRTC rather than WebSocket for more consistent performance.1. A browser makes a request to a developer-controlled server to mint an ephemeral API key.2. The developer's server uses a [standard API key](/settings/organization/api-keys) to request an ephemeral key from the [OpenAI REST API](/docs/api-reference/realtime-sessions), and returns that new key to the browser.3. The browser uses the ephemeral key to authenticate a session directly with the OpenAI Realtime API as a [WebRTC peer connection](https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection).Initializing a peer connection```javascript// Get a session token for OpenAI Realtime APIconst tokenResponse = await fetch("/token");const data = await tokenResponse.json();await pc.setLocalDescription(offer);const baseUrl = "https://api.openai.com/v1/realtime/calls";const model = "gpt-realtime";const sdpResponse = await fetch(`${baseUrl}?model=${model}`, {---------------------------To create an ephemeral token to use on the client-side, you will need to build a small server-side application (or integrate with an existing one) to make an [OpenAI REST API](/docs/api-reference/realtime-sessions) request for an ephemeral key. You will use a [standard API key](/settings/organization/api-keys) to authenticate this request on your backend server.Below is an example of a simple Node.js [express](https://expressjs.com/) server which mints an ephemeral API key using the REST API: try { const response = await fetch( "https://api.openai.com/v1/realtime/client_secrets", { method: "POST",```You can create a server endpoint like this one on any platform that can send and receive HTTP requests. Just ensure that **you only use standard OpenAI API keys on the server, not in the browser.**Sending and receiving eventsCheck out the WebRTC Realtime API in this light weight example app./Users/emcho/devel/openai-realtime-consoleWas this page useful?
If you want to connect a phone number to the Realtime API, use a SIP trunking provider (e.g., Twilio). This is a service that converts your phone call to IP traffic. After you purchase a phone number from your SIP trunking provider, follow the instructions below.Start by creating a [webhook](/docs/guides/webhooks) for incoming calls, at platform.openai.com. Then, point your SIP trunk at the OpenAI SIP endpoint, using the project ID for which you configured the webhook, e.g., `sip:$PROJECT_ID@sip.api.openai.com;transport=tls`. To find your `$PROJECT_ID`, go to your \[settings\] > **General**. The page displays the project ID. It should have a `proj_` prefix.When OpenAI receives SIP traffic associated with your project, the webhook that you configured will be fired. The event fired will be a [`realtime.call.incoming`](/docs/api-reference/webhook_events/realtime/call/incoming) event.This webhook lets you accept or reject the call. When accepting the call, you'll provide the configuration (instructions, voice, etc) for the Realtime API session. Once established, you can set up a web socket and monitor the session as usual. The APIs to accept, reject, and monitor the call are documented below.URIs used for interacting with Realtime API and SIP:|SIP URI|sip:$PROJECT_ID@sip.api.openai.com;transport=tls||Accept URI|https://api.openai.com/v1/realtime/calls/$CALL_ID/accept||Reject URI|https://api.openai.com/v1/realtime/calls/$CALL_ID/reject||Refer URI|https://api.openai.com/v1/realtime/calls/$CALL_ID/refer||Events URI|wss://api.openai.com/v1/realtime?call_id=$CALL_ID|Find your `$CALL_ID` in the `call_id` field in data object present in the webhook. See an example in the next section.```pythonfrom flask import Flask, request, Response, jsonify, make_responsefrom openai import OpenAI, InvalidWebhookSignatureErrorimport asyncioimport jsonapp = Flask(__name__)client = OpenAI( webhook_secret=os.environ["OPENAI_WEBHOOK_SECRET"])AUTH_HEADER = { "Authorization": "Bearer " + os.getenv("OPENAI_API_KEY")} try: async with websockets.connect( "wss://api.openai.com/v1/realtime?call_id=" + call_id, additional_headers=AUTH_HEADER, ) as websocket: if event.type == "realtime.call.incoming": requests.post( "https://api.openai.com/v1/realtime/calls/" + event.data.call_id + "/accept",It's also possible to redirect the call to another number. During the call, make a POST to the `refer` endpoint:|URL|https://api.openai.com/v1/realtime/calls/$CALL_ID/refer||Payload|JSON with one key target_uriThis is the value used in the Refer-To. You can use Tel-URI for example tel:+14152909007||Headers|Authorization: Bearer YOUR_API_KEYSubstitute YOUR_API_KEY with a standard API key|
```javascriptconst baseUrl = "https://api.openai.com/v1/realtime/calls";const model = "gpt-realtime";const sdpResponse = await fetch(`${baseUrl}?model=${model}`, {// Connect to a WebSocket for the in-progress callconst url = "wss://api.openai.com/v1/realtime?call_id=" + callId;const ws = new WebSocket(url, { headers: { Authorization: "Bearer " + process.env.OPENAI_API_KEY, },});### With SIP1. A user connects to OpenAI via phone over SIP.2. OpenAI sends a webhook to your application’s backend webhook URL, notifying your app of the state of the session.```textPOST https://my_website.com/webhook_endpointuser-agent: OpenAI/1.0 (+https://platform.openai.com/docs/webhooks)content-type: application/jsonwebhook-id: wh_685342e6c53c8190a1be43f081506c52 # unique id for idempotency```3. The application server opens a WebSocket connection to the Realtime API using the `call_id` value provided in the webhook. This `call_id` looks like this: `wss://api.openai.com/v1/realtime?call_id={callId}`. The WebSocket connection will live for the life of the SIP call.Was this page useful?
Our most advanced speech-to-speech model is [gpt-realtime](/docs/models/gpt-realtime).This model shows improvements in following complex instructions, calling tools, and producing speech that sounds natural and expressive. For more information, see the [announcement blog post](https://openai.com/index/introducing-gpt-realtime/).Update your session to use a prompt----------------------Here are top tips for prompting the realtime speech-to-speech model. For a more in-depth guide to prompting, see the [realtime prompting cookbook](https://cookbook.openai.com/examples/realtime_prompting_guide).### General usage tips--------------------------------------------Here are 10 tips for creating effective, consistently performing prompts with gpt-realtime. These are just an overview. For more details and full system prompt examples, see the [realtime prompting cookbook](https://cookbook.openai.com/examples/realtime_prompting_guide).#### 1\. Be precise. Kill conflicts.You can include sample phrases for preambles to add variety and better tailor to your use case.There are several other ways to improve the model's behavior when performing tool calls and keeping the conversation going with the user. Ideally, the model is calling the right tools proactively, checking for confirmation for any important write actions, and keeping the user informed along the way. For more specifics, see the [realtime prompting cookbook](https://cookbook.openai.com/examples/realtime_prompting_guide).#### 9\. Use LLMs to improve your prompt.This guide is long but not exhaustive! For more in a specific area, see the following resources:* [Realtime prompting cookbook](https://cookbook.openai.com/examples/realtime_prompting_guide): Full prompt examples and a deep dive into when and how to use them* [Inputs and outputs](/docs/guides/realtime-inputs-outputs): Text and audio input requirements and output options* [Managing conversations](/docs/guides/realtime-conversations): Learn to manage a conversation for the duration of a realtime session* [MCP servers](/docs/guides/realtime-mcp): How to use MCP servers to access additional tools in realtime apps* [Realtime transcription](/docs/guides/realtime-transcription): How to transcribe audio with the Realtime API* [Voice agents](https://openai.github.io/openai-agents-js/guides/voice-agents/quickstart/): A quickstart for building a voice agent with the Agents SDKWas this page useful?
Build low-latency, multimodal LLM applications with the Realtime API.The OpenAI Realtime API enables low-latency communication with [models](/docs/models) that natively support speech-to-speech interactions as well as multimodal inputs (audio, images, and text) and outputs (audio and text). These APIs can also be used for [realtime audio transcription](/docs/guides/realtime-transcription).Voice agents------------One of the most common use cases for the Realtime API is building voice agents for speech-to-speech model interactions in the browser. Our recommended starting point for these types of applications is the [Agents SDK for TypeScript](https://openai.github.io/openai-agents-js/guides/voice-agents/), which uses a [WebRTC connection](/docs/guides/realtime-webrtc) to the Realtime model in the browser, and [WebSocket](/docs/guides/realtime-websocket) when used on the server.```jsimport { RealtimeAgent, RealtimeSession } from "@openai/agents/realtime";const agent = new RealtimeAgent({Follow the voice agent quickstart to build Realtime agents in the browser.](https://openai.github.io/openai-agents-js/guides/voice-agents/quickstart/)To use the Realtime API directly outside the context of voice agents, check out the other connection options below.------------------While building [voice agents with the Agents SDK](https://openai.github.io/openai-agents-js/guides/voice-agents/) is the fastest path to one specific type of application, the Realtime API provides an entire suite of flexible tools for a variety of use cases.There are three primary supported interfaces for the Realtime API:
reconsumeralization
import { OpenAI } from "https://esm.town/v/std/openai";
import { sqlite } from "https://esm.town/v/stevekrouse/sqlite";
/**
* Practical Implementation of Collective Content Intelligence
* Bridging advanced AI with collaborative content creation
*/
exp
kwhinnery_openai
lost1991
import { OpenAI } from "https://esm.town/v/std/openai";
export default async function(req: Request): Promise<Response> {
if (req.method === "OPTIONS") {
return new Response(null, {
headers: {
"Access-Control-Allow-Origin": "*",
No docs found