Working with Apache Kafka

This section guides you through setting up, configuring, and managing the Apache Kafka integration.

Prerequisites

OpsRamp Classic Gateway (Linux) 15.0.0 and above.
OpsRamp Nextgen Gateway 15.0.0 and above.
Upgrade to the latest Gateway version to ensure comprehensive coverage of recent bug fixes, feature enhancements, and so on.
OpsRamp integrated the Aapche Kafka application by using CLI and JMX calls. The following are the needed priviliges.
- Ensure that the provided Publisher or Subscriber IP address or host name, along with the credentials, are valid and accessible for successful integration (verify from the OpsRamp Gateway).
  - To check network connectivity from the gateway, use the following command:
    ping <IP Address/Host Name>
  - To check the SSH port accessibility, use the following command:
    telnet <IP Address/Host Name> <Port>
- JMX -
  - Apache Kafka Brokers must be started with JMX remote enabled (usually enabled by default).
  - The JMX port (default port - 1099) must be open and accessible from the monitoring host.
  - JMX username and password must be configured and available for use, and the user must have permission (at least readonly or monitor access) to connect to the JMX server and read MBean attributes.
  - All Borkers should use same JMX port and same JMX credentials
  - All the brokers should be reachable from OpsRamp gateway, using ipAddress or hostName that configured Kafka Cluster.

Note

Ip Address is most preferable for brokerName instead of dns name, it should be resolve from Opsramp gateway.

Configure and Install the Apache Kafka Integration

From All Clients, select a client.
Go to Setup > Account.
Select the Integrations and Apps tab.
The INSTALLED INTEGRATIONS page, where all the installed applications are displayed.
Note: If there are no installed applications, it will navigate to the Available Integrations and Apps page.
Click + ADD on the INSTALLED INTEGRATIONS page. The AVAILABLE INTEGRATIONS AND APPS page displays all the available applications along with the newly created application with the version.
Note: You can even search for the application using the search option available. Also you can use the All Categories option to search.
Click ADD in the Apache Kafka application.
In the Configurations page, click + ADD. The Add Configuration page appears.
Enter the following BASIC INFORMATION:


Field Name	Description	Field Type
Name	A user-defined display name for the integration instance	String
Kafka BootStrap IPAddress	Enter the Apache Kafka BootStrap IP address. You must be able to connect to this node using SSH and able to execute CLI commands to get Kafka Resource information.	String
SSH Port	Enter the SSH Port, that must be open and accessible from the gateway. Typically, the default is 22. Note: By default SSH Port is 22	String
SSH Credentialas	Select the SSH credentials to authenticate with the Apache Kafka system. User should have permission to execute CLI command to get the Kafka Resource information. Select existing or else, click +ADD to create credentials. The ADD CREDENTIAL window is displayed. Enter the following information: Name: Enter the credential name. Description: Enter the brief description of the credential. User Name: Enter the Username. Password: Enter the password. Confirm Password: Confirm your password. Port: Select the port number from the dropdown. Connection Timeout (ms): Specify the connection timeout in milliseconds.	Dropdown
JMX Port	Enter the JMX Port, that must be open and accessible from the monitoring host. Typically, the default is 1099.	String
JMX Credentials	Select the JMX credentials to authenticate with the Apache Kafka brokers. JMX username and password must be configured and available for use, and the user must have permission (at least readonly or monitor access) to connect to the JMX server and read MBean attributes. Select existing or else, click +ADD to create credentials. The ADD CREDENTIAL window is displayed. Enter the following information: Name: Enter the credential name. Description: Enter the brief description of the credential. User Name: Enter the Username. Password: Enter the password. Confirm Password: Confirm your password. Port: Select the port number from the dropdown. Connection Timeout (ms): Specify the connection timeout in milliseconds.	Dropdown
Kafka Port	Enter the Kafka Port, that must be open and accessible from the monitoring host. Typically, the default is 9092.	String
Kafka Path	Enter the Kafka Path where we configured Kafka and the scripts available to get the Kafka Resource information. Ex: /opt/kafka	String
App Failure Notifications	When selected, you will be notified in case of an application failure such as Connectivity Exception, Authentication Exception.	Checkbox
Connection Timeout	Adds the feasibility to configure SSH/JMX connection timeouts in seconds (ex: 60, 50).	Integer
Read Timeout	Adds the feasibility to configure SSH/JMX Read timeouts in seconds (ex: 60, 50).	Integer
Connection Idle Timeout	Adds the feasibility to configure JMX connection timeouts in seconds (ex: 60, 50).	Integer

Select the below mentioned Custom Attribute:


Field Name	Field Type	Description
Custom Attribute	Dropdown	Select the custom attribute from the dropdown.
Value	Dropdown	Select the value from the dropdown.

Note: The custom attribute that you add here will be assigned to all the resources that are created by the integration. You can add a maximum of five custom attributes (key and value pair).

In the RESOURCE TYPE section, select:
- ALL: All the existing and future resources will be discovered.
- SELECT: You can select one or multiple resources to be discovered.
In the DISCOVERY SCHEDULE section, select Recurrence Pattern to add one of the following patterns:
- Minutes
- Hourly
- Daily
- Weekly
- Monthly
Click ADD.

Note

Sample values shown in the configuration fields are for illustration purposes only. Use your actual integration details when configuring.

Now the configuration is saved and displayed on the configurations page after you save it.
Note: From the same page, you may Edit and Remove the created configuration.

Click Next.
Below are the optional steps you can perform on the Installation page.

Under the ADVANCED SETTINGS, Select the Bypass Resource Reconciliation option, if you wish to bypass resource reconciliation when encountering the same resources discovered by multiple applications.
Note: If two different applications provide identical discovery attributes, two separate resources will be generated with those respective attributes from the individual discoveries.

Click +ADD to create a new collector by providing a name or use the pre-populated name.

Select an existing registered profile.

Click FINISH.

The application is now installed and displayed on the INSTALLED INTEGRATION page. Use the search field to find the installed application.

Modify the Configuration

See Modify an Installed Integration or Application article.
Note: Select the Apache Kafka application.

View the Apache Kafka Details

Navigate to Infrastructure > Search > MIDDLEWARE > Apache Kafka.
Select the application on the Apache Kafka page.
The RESOURCE page appears from the right.
Click the ellipsis (…) on the top right and select View Details.

View resource metrics

To confirm Apache Kafka monitoring, review the following:

Metric graphs: A graph is plotted for each metric that is enabled in the configuration.
Alerts: Alerts are generated for metrics that are configured as defined for integration.