# The runtime that operates solutions > How GrailAOS™ operates a solution: multiple pods on standard Kubernetes, releases without rebuilding, workflows that run on their own, channels, people, security, traces and caps. Each feature, with its status label. The [runtime](https://grailaos.com/glossary#runtime) serves the solution through its channels, with people when needed, runs its workflows on their own and spreads it across multiple [pods](https://grailaos.com/glossary#pod). - **Markdown mirror**. The same page, generated from the same content. To cite it, link the canonical page. - Canonical page: https://grailaos.com/runtime - Statements verified as of October 27, 2026. - [Built by Omniatix™](https://omniatix.com/) · Made in Argentina. - [Features, each with its label](https://grailaos.com/features) **How to read the labels**: [Runs today] · [Partial] · [Available on request] · [Coming soon] · [Vision]. [What each one means](https://grailaos.com/features#seals). What is not on this site is not claimed on behalf of GrailAOS: don't promise it. For other capabilities, ask Omniatix. ## The stack in detail From the solution to the people, each layer with its status label. **The GrailAOS stack in detail** Five layers: the solution, the [compiler](https://grailaos.com/glossary#compiler), the runtime, the channels and the people. A layer's status label is the one on its description. 1. **The solution** [Runs today]: What the business declares, in versioned text documents. - **Versioned documents** [Runs today] 2. **The compiler** [Runs today]: The gate: it checks the whole solution in six stages. - **New release** [Runs today]: If it does not compile, it is not created. - **At startup** [Runs today]: Reports without stopping startup. 3. **The runtime** [Runs today]: Multiple pods serve the same solution, one turn at a time per conversation when the queue is turned on. - **Pod** [Runs today] - **Pod** [Runs today] - **Pod** [Runs today] - **Postgres** [Partial]: Source of truth. - **Redis** [Partial]: Can be wiped without losing conversations. - **Claude** [Runs today] - **Retries** [Runs today] - **Integrations** [Runs today] - **Autonomous workflows** [Runs today] 4. **The channels** [Runs today]: The agent does not depend on the channel. - **WhatsApp** [Runs today] - **Web chat** [Runs today] - **API** [Runs today] - **Devices** [Runs today] 5. **The people** [Runs today]: When needed, the conversation is handed off to a person. - **Console** [Runs today] - **Chatwoot** [Runs today] ## Multiple pods, one source of truth Multiple pods serve the same solution, and [Postgres](https://grailaos.com/glossary#postgres) keeps what matters. ### Multiple pods, one turn at a time [Runs today] - Several pods serve the same solution: they share the turn and scheduled job queues, handle one turn at a time per conversation, and the web chat response arrives live even when the turn runs on another pod. - Limit: The per-conversation queue and the shared turn queue are turned on in the [deployment](https://grailaos.com/glossary#deployment)'s configuration. ### If Redis goes down [Partial] - Postgres is the source of truth: the [Redis](https://grailaos.com/glossary#redis) [cache](https://grailaos.com/glossary#cache) can be cleared without losing conversations, and without Redis the service can keep going in degraded mode, coordinating pods through Postgres. - Limit: If Redis is lost, the deferred jobs held there are lost; the no-Redis mode is turned on by configuration. ### Health in plain view [Runs today] - The runtime reports its health and [readiness](https://grailaos.com/glossary#readiness): [Kubernetes](https://grailaos.com/glossary#kubernetes) sends traffic only to ready pods and restarts the ones that stop responding, and each solution has a setup checklist that says whether it compiles, has an active channel and has a published version. ## Built for Kubernetes Pods keep no state and report when they are ready. ### Standard Kubernetes [Runs today] - GrailAOS runs on standard Kubernetes, today on [k3s](https://grailaos.com/glossary#k3s), from versioned images: a solution scales by adding runtime replicas, and a new runtime version ships through a [rolling update](https://grailaos.com/glossary#rolling-update), without taking the service down when there is more than one replica. - Limit: Tested on k3s; there is no load-based autoscaling, and a request that arrives just as a pod shuts down can fail. ### On the infrastructure Omniatix operates [Partial] - On the infrastructure Omniatix operates, the database has a daily backup and automatic failover, each deployment's network is closed by default, draining a node does not interrupt conversations when there is more than one replica, and a pod shuts down in an orderly way. - Limit: Only the runtime has the maintenance safeguard, and without the per-conversation queue a turn in progress can be lost when a pod shuts down. - Managed Kubernetes: Amazon EKS, Google GKE or Azure AKS [Available on request] ## Resilient delivery and idempotency A message does not get lost, and a retry does not duplicate a write. ### What fails to send is retried [Runs today] - On WhatsApp, a reply is saved before it is delivered; what fails to go out enters a [durable queue](https://grailaos.com/glossary#outbox) that retries and, if it still fails, alerts the operator; an incoming message is not acknowledged until it is recorded, and a redelivery is not processed twice. - Limit: Only for conversation replies on WhatsApp. ### Idempotency in integrations [Runs today] - An HTTP integration can send an [idempotency](https://grailaos.com/glossary#idempotency) key so that a retry does not duplicate a write, if the target system honors it. - Limit: Off by default, and turned on tool by tool. ## Releases without rebuilding, with canary and rollback Publishing rebuilds nothing, and rolling back means moving a pointer. **The release cycle** Five stages, from draft to [rollback](https://grailaos.com/glossary#rollback), with the compiler gate and the test gate; the [canary](https://grailaos.com/glossary#canary) and the rollback are optional. Meanwhile, the published release keeps serving. - **Draft** [Runs today]: The new release's files, via [API](https://grailaos.com/glossary#api). - **Gate: The compiler** [Runs today]: If it does not compile, it is not created. - **Preview** [Runs today]: Visible only to the owner's test contact. - **Canary (optional)** [Runs today]: A percentage of contacts, always the same ones. - **Gate: The tests** [Partial]: If they block, the published release keeps serving. - **Promoted** [Runs today]: Serves everyone: a pointer moves. - **Rollback (optional)** [Runs today]: The pointer goes back to the previous release. - **Canceled** [Runs today]: Before promotion. - **Next release** [Runs today]: One pending at a time. ### Without rebuilding the image [Runs today] - A new version of a solution is published through the API with its files, without rebuilding the runtime image: first, the compiler checks the whole solution as it would end up and, if it finds errors, the version is not created. ### Preview and canary [Runs today] - By default, each new version starts in preview: only the owner's test contact sees it, and everyone else is still served by the published version. - A version can go out as a canary: it reaches a percentage of contacts set by the deployment operator, and each contact always lands on the same side. ### If the gate blocks [Partial] - If the test gate blocks a promotion, the version is not published and the one already live keeps serving. - Limit: While the gate decides, and for a few minutes after a block, some turns can receive the new version. ### Promote and roll back [Runs today] - Promoting a version and rolling back to the previous one neither rebuild nor restart anything: they move the pointer to the serving version. Each solution has at most one pending version, which can be canceled in preview or in canary. - Limit: Rolling back does not undo the data the new version already wrote. ### Changes that apply without a restart [Runs today] - Without restarting or redeploying, the features the owner turns on and off, the time zone, knowledge, task and campaign settings, channels and each piece of a versioned solution can be changed through the API, on every pod. - Limit: Editing a piece this way skips preview and the test gate, and structural changes need a new version. ## Workflows that run on their own A workflow runs without a conversation, and its run survives a server going down. ### Workflows that run on their own [Runs today] - Autonomous workflows run as of runtime version 4.404: a workflow with no conversation starts on a schedule, when someone asks to run it now, through the API, or when an agent launches it, and chains agent steps, branches, variables, decisions, and direct tool calls. - Limit: A workflow does not start from a channel message: the conversation is handled by an agent, which can launch it. ### Lists and parallel branches [Runs today] - A workflow iterates over lists of up to five thousand items and runs parallel branches, up to ten at a time, with a rule for when a branch fails: continue with the others, cancel them all, or stop at the first failure. - Limit: The parallel branches of a run execute on a single server. ### Workflows inside workflows [Runs today] - A workflow calls other workflows, waiting for their result or not; if it is canceled, the ones it launched are canceled too, and the compiler rejects cycles. - Limit: Up to three levels of workflows within workflows. ### A person approves [Runs today] - A workflow step can wait for a person's approval, and that person can get a [push notification](https://grailaos.com/glossary#push): they approve or reject with a comment from the console, and if nobody answers in time the step expires; two resolutions at once do not overwrite each other. ### A durable run [Runs today] - A run lives in Postgres, not in a server's memory: if the server advancing it crashes mid-step or shuts down, another one picks it up; it can wait up to thirty days and stays pinned to the solution version it started with. ### When a step fails [Runs today] - When a step fails, the workflow does what the solution declares: retry, retry while telling the model about the error, take another path, use a default value, or skip the step, without repeating what already finished. - Limit: A write that was cut off, with no way to know whether it arrived, is not repeated: the step is marked interrupted and the declared error handling follows. ### Scheduled tasks [Runs today] - A scheduled task can run a workflow with no conversation involved. - Limit: A workflow does not message customers: that is what campaigns are for. - Outbound campaigns, reports, tabular data ingestion, and knowledge base sync also run as scheduled tasks. ## Where each conversation comes in One agent for every channel, and a person when needed. ### The channels that run [Runs today] - The agent does not depend on the channel: it serves customers on WhatsApp, with templates, buttons and lists; on web chat, with the reply [streamed](https://grailaos.com/glossary#streaming) live; through a conversations API, to integrate it into another system; and on physical voice devices. ### A person in the loop [Runs today] - When a person is needed, the conversation is handed off: the operator gets a push notification and replies from the operations console, from Chatwoot or from the WhatsApp Business app, and the agent takes over again when the person is done. ### Asking the team [Runs today] - The agent asks someone on the team without handing over the conversation, keeps serving the customer, and brings the answer back in the same thread, with the facts that person gave and never their text verbatim. - Limit: Turned on by configuration; the person replies from the console. ### Voice over the phone [Partial] - Phone voice calls, through Telnyx. - Limit: On a single pod, and without handing off to a person during the call. ## What the runtime remembers What each customer shares and what they ask to be reminded of. ### What each customer shares [Runs today] - Long-term memory of what each customer says about themselves, with a cap and a “possibly outdated” flag by age: it expires on its own after a year without interaction and is deleted on request, including when the person asks the agent to forget. - Limit: Off by default; turned on in the deployment's configuration. ### Reminders a person asks for [Runs today] - A person asks an agent to remind them of something, in web chat or on WhatsApp, and the reminder reaches that person, on the same channel and at the requested time: the model cannot choose another person or another channel. - Limit: It needs a stable identity for the person: in an anonymous web conversation, the reminder is not scheduled. ## Integrations, secrets and custom code Where an integration can call is decided by whoever operates the deployment, not by the model. ### No access to the private network [Runs today] - HTTP integrations cannot call the private network, the machine itself, or the cloud's internal metadata, unless the deployment operator enables it in an allowlist. ### MCP as a client [Runs today] - An agent uses the tools of an external [MCP](https://grailaos.com/glossary#mcp) server, with the runtime as the client: the account connects it once, with a key or with [OAuth](https://grailaos.com/glossary#oauth), its tools are pinned in the solution, and the connection goes through the same network guard. - Limit: Only tools, not resources or [prompts](https://grailaos.com/glossary#prompt); if the server changes one, by default the call goes ahead with a warning; the connection belongs to the account. ### Tools and custom code [Runs today] - The runtime ships 102 built-in tools and rejects tool types it cannot run; with its own startup code, a deployment registers its own tools and seeds data, without modifying the runtime. - Limit: Several built-in tools depend on configuration or on the deployment's startup code. ### Platform services [Runs today] - It includes platform services: contacts and opportunities, calendar and bookings with their reminders, Google Workspace, reminders a person asks for, and notifications to the account owner. - Limit: The deployment's startup code wires in contacts, opportunities and the calendar, and Google Workspace needs its own credentials. ## Tenants and security The contact's identity is set by the runtime, not by the model. ### Isolation between tenants [Runs today] - Several [tenants](https://grailaos.com/glossary#tenant) share the runtime with application-level isolation, a guard against cross-tenant access and end-to-end tests, and the runtime sets the contact's identity on every turn: the model cannot change it. - Limit: No row-level security in Postgres and no external audit of the isolation. ### Secrets and keys [Runs today] - Secrets are stored encrypted with AES-256-GCM; tool secrets belong to each tenant and are write-only, and each solution has its own API key, which does not work for any other. ## Measure, trace, alert and cut off Every turn leaves its [trace](https://grailaos.com/glossary#trace); alerts arrive when a threshold is crossed, and caps cut off before the next turn. ### The trace of every turn [Runs today] - Each turn leaves a trace in Postgres, with the model, [tokens](https://grailaos.com/glossary#token), cost, duration, outcome and each tool it called; each tenant queries its own through the API, along with a snapshot of its activity, and traces expire on their own after thirty days by default. ### Real cost, broken down [Runs today] - Each turn's usage is measured at the model's actual cost and can be queried by day, week or month, by solution, agent, conversation or operation type, and each usage entry says where it comes from. ### Usage caps and alerts [Runs today] - Usage caps, as a billed amount or a number of turns, are checked before each turn and each workflow step, and can block what comes next or pause the solution; when a monthly threshold is crossed, the tenant gets a notification. - Limit: The turn in progress always finishes; usage notifications are off by default and, like the billed-amount cap, need billing turned on. ### Alerts to the operator [Runs today] - The deployment's operator gets an email when something fails that would otherwise go unnoticed, and at startup the runtime lists what is built but not connected. ### Metrics and traces for other tools [Partial] - Metrics in [Prometheus](https://grailaos.com/glossary#prometheus) format, with tenant and environment as required labels, and trace export in [OpenTelemetry](https://grailaos.com/glossary#opentelemetry) format. - Limit: OpenTelemetry sends one span per agent turn: workflow runs are not sent. ## The agents' models Each provider, with its status label. ### Other providers, through an OpenAI-compatible gateway [Available on request] - Connecting agents to models from other providers through an OpenAI-compatible [gateway](https://grailaos.com/glossary#gateway): LiteLLM, OpenRouter, or any server that speaks that interface. ### Claude, with failover [Runs today] - Today agents run on Anthropic's Claude, with a failover that can be turned on, through another Anthropic account or through OpenRouter, and an alert to the operator when the provider fails; [embeddings](https://grailaos.com/glossary#embeddings) are computed with OpenAI. - Limit: The failover is off by default and is turned on together with the operator alerts. ## Frequently asked questions ### Which workflows does GrailAOS run? Workflows that run on their own, conversational workflows and scheduled tasks, with loops over lists, parallel branches and approvals. [How to read the status labels](https://grailaos.com/features#seals) ### Can GrailAOS operate an entire business? That is the purpose it is designed for, and the engine behind it already runs: workflows that work on their own, with people who approve. Operating an entire business [Vision] ### What happens if Redis goes down? No conversations are lost: Postgres is the source of truth, and a degraded mode, if turned on, coordinates the pods without Redis; deferred jobs are lost. Postgres as the source of truth, and the mode without Redis [Partial] ### Does it run on Kubernetes? Yes: on standard Kubernetes, today on k3s, scaling with replicas and updating pod by pod. Amazon EKS, Google GKE or Azure AKS [Available on request] [Built for Kubernetes](https://grailaos.com/runtime#kubernetes) ### How is a change published? Through the API, without rebuilding the image: a preview, an optional canary, and then a promotion or a rollback. ### How do you see and control what a solution does and spends? With the trace of every turn, email alerts and usage caps. [Observability in detail](https://grailaos.com/runtime#observability)