BDD supports many different cluster configurations. The following sections describe three suitable for demonstration, development, and production environments, and their possible variations.
Note that you aren't limited to these examples and can install in any configuration suited to your resources and data processing needs.
Single-node demo environment
You can install BDD in a demo environment running on a single physical or virtual machine. This configuration can only handle a limited amount of data, so it is recommended solely for demonstrating the product's functionality with a small sample database.
In a single-node deployment, all BDD and required Hadoop components are hosted on the same node, and the Dgraph databases are stored on the local filesystem.
For single-node installations, BDD provides a QuickStart option that enables you to install quickly with default configuration. For more information, see QuickStart Installation.
Two-node development environment
You can install BDD in a development environment running on two nodes. This configuration can handle a slightly larger database than a single-node deployment, but still has limited processing capacity and doesn't provide high availability for any BDD components.
In a two-node configuration, all Hadoop components and the Data Processing libraries are hosted on one node, while the remaining BDD components are hosted on the other. Although the diagram above shows the Dgraph databases stored on the local file system, they could also be stored in HDFS on Node 1.
Multi-node production environment
For a production environment, BDD should be installed on a multi-node cluster. The size of your cluster depends on the amount of data you plan on processing and the number of end users expected to be querying that data at any given time; however, a minimum of six nodes ensures high availability for all components.
A typical BDD cluster will consist of nodes that look similar to the following:
- Node 1 is running the Data Processing libraries, along with the YARN NodeManager service, Spark on YARN, and HDFS, all of which Data Processing requires to function. The number of Data Processing nodes your cluster should contain depends on the amount of data you have and its size, although a minimum of three ensures high availability.
- Node 2 is running the Dgraph, the Dgraph HDFS Agent, the DP CLI, and the HDFS DataNode service. The Dgraph databases are stored in HDFS, which is recommended for production environments. (Note that they could also be stored on an NFS, in which case the DataNode service wouldn't be required.) A typical cluster would contain two or more Dgraph nodes.
- Node 3 is running Studio and the Dgraph Gateway inside a WebLogic Managed Server container; the Transform Service and the Workflow Manager service, each inside a Jetty container; the DP CLI; and the Hive Table Detector. A typical cluster would contain one or more Studio nodes, depending on the number of users making concurrent queries. Note that in a cluster with multiple Studio nodes, the Workflow Manager Service and Hive Table Detector would each only be installed on one of them. Additionally, one Studio node in the cluster must serve as the Admin Server.
Co-locating components
One way to configure your BDD cluster is to co-locate different components on the same nodes. This is a more efficient use of your hardware, since you don't have to devote an entire node to any specific component.
Be aware, however, that the co-located components will compete for memory and other resources, which can have a negative impact on performance. The decision to host different components on the same nodes depends on your site's production requirements and your hardware's capacity.
- You shouldn't co-locate the Dgraph with Hadoop components, other than the HDFS DataNode service. In particular, you shouldn't host it on the same nodes as Spark, as both require a lot of memory. If you have to do this, you should use cgroups ensure each has access to sufficient resources. For more information, see Setting up cgroups.
- Similarly, you shouldn't co-locate the Dgraph with the Transform Service, which also requires a lot of memory.
- While you can co-locate Managed Servers with either the Dgraph or any Hadoop components, you should limit the amount of memory WebLogic Server can consume to ensure the other components have access to the resources they require.
