About Deploying a TEN Framework with Enterprise AI Capabilities
Two of the biggest hurdles to the wide adoption of human-to-AI voice conversation are latency (or delay) and wireless last mile challenges such as rapidly varying bandwidth and high packet loss. Agora’s Software-Defined Real-Time Network (SD-RTN), a real time overlay network for the internet, is built with intelligent routing and last mile optimizations to ensure the highest quality and lowest latency. Applying Agora’s real-time network infrastructure to voice-powered conversational AI enables humans to interact with AI in the same way they would with a human.
The TEN Framework provides real-time network infrastructure. The TEN Agent is licensed under the Apache 2.0 open-source license, while the TEN Framework uses a hybrid open-source license. Agora, as a primary supporter of TEN, offers conversational AI services built with the framework. OCI can provide the compute power and Oracle Autonomous Database Select AI with support for RAG features enable you to use LLMs to gain insight or generate innovative content based on your private data using natural language prompts.
Architecture
This architecture shows how you can create a multimodal voice agent model that uses Oracle and Agora's AI technologies for the AI infrastructure.
The following diagram is an example of a cascaded multimodal AI voice agent model:
Description of the illustration multimodal-voice-agents.png
The model shows how audio and video is processed by the large language model (LLM) and includes the following components:
- LLM deployed on OCI VM
- Speech-to-text (STT)
- Text-to-speech (TTS)
The AI infrastructure includes the underlying hardware and software system designed specifically to support the demanding computational needs of AI tasks. You can deploy an AI infrastructure in a single OCI region or deploy a multicloud architecture where processing power is spread across multiple geographically dispersed data centers, allowing for flexible deployment of applications and data.
The following diagram is a single OCI Region with a TEN (Transformative Extensions Network) Agent installed on a VM.
Description of the illustration oci-ten-agent-deploy.png
oci-ten-agent-deploy-oracle.zip
For faster processing and greater scalability for large datasets, deploy a multicloud infrastructure to run AI algorithms across multiple computing nodes. You can use a distributed AI architecture with OCI regions or in a multicloud architecture. A distributed cloud leverages the power of a multicloud to split large AI tasks into smaller parts and execute them across multiple nodes simultaneously. It also allows for faster training of complex AI models on massive datasets. However, it presents challenges in managing data consistency and coordinating computations across different nodes.
The following architecture uses a multicloud solution with Google Cloud and OCI where Google Kubernetes Engine (GKE) orchestrates the overall training and inferencing process while it offloads the computationally intensive part to OCI AI Infrastructure on demand.
Description of the illustration oci-google-multiregion.png
oci-google-multiregion-oracle.zip
The architectures support the following components:
- Region
An Oracle Cloud Infrastructure region is a localized geographic area that contains one or more data centers, called availability domains. Regions are independent of other regions, and vast distances can separate them (across countries or even continents).
- Availability domain
Availability domains are standalone, independent data centers within a region. The physical resources in each availability domain are isolated from the resources in the other availability domains, which provides fault tolerance. Availability domains don’t share infrastructure such as power or cooling, or the internal availability domain network. So, a failure at one availability domain shouldn't affect the other availability domains in the region.
- Virtual cloud network (VCN) and subnet
A VCN is a customizable, software-defined network that you set up in an Oracle Cloud Infrastructure region. Like traditional data center networks, VCNs give you control over your network environment. A VCN can have multiple non-overlapping CIDR blocks that you can change after you create the VCN. You can segment a VCN into subnets, which can be scoped to a region or to an availability domain. Each subnet consists of a contiguous range of addresses that don't overlap with the other subnets in the VCN. You can change the size of a subnet after creation. A subnet can be public or private.
- Fault domain
A fault domain is a grouping of hardware and infrastructure within an availability domain. Each availability domain has three fault domains with independent power and hardware. When you distribute resources across multiple fault domains, your applications can tolerate physical server failure, system maintenance, and power failures inside a fault domain.
- Load balancer
Oracle Cloud Infrastructure Load Balancing provides automated traffic distribution from a single entry point to multiple servers.
- Autonomous Database
Oracle Autonomous Database is a fully managed, preconfigured database environment that you can use for transaction processing and data warehousing workloads. You do not need to configure or manage any hardware, or install any software. Oracle Cloud Infrastructure handles creating, backing up, patching, upgrading, and tuning the database.
- Identity and Access Management
Oracle Cloud Infrastructure Identity and Access Management (IAM) provides user access control for Oracle Cloud Infrastructure (OCI) and Oracle Cloud Applications. The IAM API and the user interface enable you to manage identity domains and the resources within them. Each OCI IAM identity domain represents a standalone identity and access management solution or a different user population.
- Policy
An Oracle Cloud Infrastructure Identity and Access Management policy specifies who can access which resources, and how. Access is granted at the group and compartment level, which means you can write a policy that gives a group a specific type of access within a specific compartment, or to the tenancy.
- Audit
The Oracle Cloud Infrastructure Audit service automatically records calls to all supported Oracle Cloud Infrastructure public application programming interface (API) endpoints as log events. All OCI services support logging by Oracle Cloud Infrastructure Audit.
- LoggingOracle Cloud Infrastructure Logging is a highly-scalable and fully managed service that provides access to the following types of logs from your resources in the cloud:
- Audit logs: Logs related to events produced by OCI Audit.
- Service logs: Logs published by individual services such as OCI API Gateway, OCI Events, OCI Functions, OCI Load Balancing, OCI Object Storage, and VCN flow logs.
- Custom logs: Logs that contain diagnostic information from custom applications, other cloud providers, or an on-premises environment.
- Object storage
Oracle Cloud Infrastructure Object Storage provides quick access to large amounts of structured and unstructured data of any content type, including database backups, analytic data, and rich content such as images and videos. You can safely and securely store and then retrieve data directly from the internet or from within the cloud platform. You can scale storage without experiencing any degradation in performance or service reliability. Use standard storage for "hot" storage that you need to access quickly, immediately, and frequently. Use archive storage for "cold" storage that you retain for long periods of time and seldom or rarely access.
- Internet gateway
An internet gateway allows traffic between the public subnets in a VCN and the public internet.
- Service gateway
The service gateway provides access from a VCN to other services, such as Oracle Cloud Infrastructure Object Storage. The traffic from the VCN to the Oracle service travels over the Oracle network fabric and does not traverse the internet.
- Web Application Firewall (WAF)
Oracle Cloud Infrastructure Web Application Firewall (WAF) is a payment card industry (PCI) compliant, regional-based and edge enforcement service that is attached to an enforcement point, such as a load balancer or a web application domain name. WAF protects applications from malicious and unwanted internet traffic. WAF can protect any internet facing endpoint, providing consistent rule enforcement across a customer's applications.
- FastConnect
Oracle Cloud Infrastructure FastConnect creates a dedicated, private connection between your data center and Oracle Cloud Infrastructure. FastConnect provides higher-bandwidth options and a more reliable networking experience when compared with internet-based connections.
The following third-party components:
- TEN Framework
The TEN (Transformative Extensions Network) is an open-source framework that enables developers to quickly build real-time multimodal agents (voice, video, data stream, image and text), making it easy for developers to experiment, integrate large language models, and create reusable extensions.
- TEN Agent
The TEN Agent is installed on a virtual machine. You can use TEN to build agents supporting use cases like voice chatbots, AI generated meeting minutes, language tutors, simultaneous translators, virtual companions, counseling and a lot more. Developers can leverage a diverse set of AI services and extensions and have the full flexibility to build, test and roll out the next-gen AI agents, which can think, listen, see, and interact as humans do in real-time.
- Llama 3.2
Llama 3.2 is a lightweight version of Meta AI's Llama large language model (LLM). Llama 3.2 is a multimodal auto-regressive language model that you can use in tasks requiring image recognition and language processing.
- Deepgram
Speech-to-text (STT) component.
- Fishaudio
Text-to-Speech (TTS) component.
Recommendations
- VCN
When you create a VCN, determine the number of CIDR blocks required and the size of each block based on the number of resources that you plan to attach to subnets in the VCN. Use CIDR blocks that are within the standard private IP address space.
Select CIDR blocks that don't overlap with any other network (in Oracle Cloud Infrastructure, your on-premises data center, or another cloud provider) to which you intend to set up private connections.
After you create a VCN, you can change, add, and remove its CIDR blocks.
When you design the subnets, consider your traffic flow and security requirements. Attach all the resources within a specific tier or role to the same subnet, which can serve as a security boundary.
- Security lists
Use security lists to define ingress and egress rules that apply to the entire subnet.
- Network security groups (NSGs)
You can use NSGs to define a set of ingress and egress rules that apply to specific VNICs. We recommend using NSGs rather than security lists, because NSGs enable you to separate the VCN's subnet architecture from the security requirements of your application.
- Cloud Guard
Clone and customize the default recipes provided by Oracle to create custom detector and responder recipes. These recipes enable you to specify what type of security violations generate a warning and what actions are allowed to be performed on them. For example, you might want to detect Object Storage buckets that have visibility set to public.
Apply Cloud Guard at the tenancy level to cover the broadest scope and to reduce the administrative burden of maintaining multiple configurations.
You can also use the Managed List feature to apply certain configurations to detectors.
- Security Zones
For resources that require maximum security, Oracle recommends that you use security zones. A security zone is a compartment associated with an Oracle-defined recipe of security policies that are based on best practices. For example, the resources in a security zone must not be accessible from the public internet and they must be encrypted using customer-managed keys. When you create and update resources in a security zone, Oracle Cloud Infrastructure validates the operations against the policies in the security-zone recipe, and denies operations that violate any of the policies.
- Load balancer bandwidth
While creating the load balancer, you can either select a predefined shape that provides a fixed bandwidth, or specify a custom (flexible) shape where you set a bandwidth range and let the service scale the bandwidth automatically based on traffic patterns. With either approach, you can change the shape at any time after creating the load balancer.
Considerations
When implementing a multimodal TEN Framework, consider the following:
- Network connectivity
Requires robust network connectivity to manage distributed computing resources effectively.
- GPUs
AI infrastructure includes high-performance computing clusters with specialized hardware, such as GPUs, to accelerate AI computations. It often leverages distributed storage systems to handle large datasets efficiently and may involve specialized AI frameworks and libraries for training and deploying AI models. Use OCI Supercluster to scale up to 32,768 GPUs.
- Oracle Database 23ai
Leverage Oracle Database 23ai technologies and assisted by AI provides the following benefits:
- Reduce the risk of hallucinations: Leverage RAG and AI Vector Search to provide more precise responses to natural language questions when using LLMs with enterprise data.
- Oracle Autonomous Database NVIDIA GPU support: Access NVIDIA GPUs to accelerate performance of certain AI data operations without having to worry about provisioning or managing GPU servers. You can take advantage of Oracle Machine Learning Notebooks that use GPU-enabled Python packages for resource-intensive workloads, such as generating vector embeddings using transformer models and building deep learning models.
- Broader support for LLMs: Helps organizations gain more value from generative AI with built-in integration from Oracle Autonomous Database to additional LLMs: Google Gemini, Anthropic Claude, and Hugging Face. Autonomous Database integrates with 35 different LLMs across seven providers to give you a wide choice in building GenDev applications.
- Data Studio AI enhancements: Prepare and load data using natural language, as well as use a visual “drag and drop” tool to create AI pipelines with text and image vector embeddings.
- Graph Studio enhancements: Build Operational Property Graph models without code, new in Oracle Database 23ai, using the built-in self-service tool.
- Autonomous Database for Developers: Access the rich set of features and tools provided by Oracle Autonomous Database at a flat hourly rate. This provides a lower and more predictable entry point for development use cases with a simple upgrade path to production deployment.
- Multicloud
OCI's ability to work in a multicloud architecture provides the following benefits:
- Enables you to deploy applications and data across various locations, including on-premises, edge devices, and public clouds.
- Offers improved performance, resilience, and data locality depending on where the user is located.
- OCI is a hyperscaler capable of delivering more than 150 cloud services across any environment, anywhere. Or, get Oracle Database services from AWS, Microsoft Azure, and Google Cloud partner regions.
Explore More
Review these additional resources to learn more about the features of this reference architecture.
- Oracle Cloud Infrastructure Documentation
- OCI Generative AI documentation
- OCI Generative AI
- Announcing Select AI with Retrieval Augmented Generation (RAG) on Autonomous Database (blog)
- TEN Framework
Review the following OCI resources: