Streaming Data into HCP

The first step in adding a new data source telemetry is to stream all raw events from the telemetry data source into its own Kafka topic.

Note

Although HCP includes parsers for several data sources (for example, Bro, Snort, and YAF), you must still stream the raw data into HCP through a Kafka topic.

By default, the Snort parser is configured to use ZoneId.systemDefault() for the source `timeZone` for the incoming data and MM/dd/yy-HH:mm:ss.SSSSSS as the default `dateFormat`. Valid timezones are per Java's ZoneId.getAvailableZoneIds(). DateFormats should be valid per the options defined in https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html. Below is a sample configuration with the `dateFormat` and `timeZone` explicitly set in the parser config.

"parserConfig": {
"dateFormat" : "MM/dd/yy-HH:mm:ss.SSSSSS",
 "timeZone" : "America/New_York"

Note

When you install and configure Snort, you must configure Snort to include the year in the timestamp by modifying the snort.conf file as follows:

# Configure Snort to show year in timestamps
config show_year

This is important for the proper functioning of indexing and analytics.

Depending on the type of data you are streaming into HCP, you can use one of the following methods:

NiFi

This type of streaming method works for most types of data sources. For information on installing NiFi, see the NiFi documentation.

	Important
	NiFi cannot be installed on top of HDP, so you must install NiFi manually to use it with HCP.

	Note
	Ensure that the NiFi web application is using port 8089.

Performant network ingestion probes

This type of streaming method is ideal for streaming high volume packet data. See Setting up pcap to View Your Raw Data for more information.

Real-time and batch threat intelligence feed loaders

This type of streaming method is used for real-time and batch threat intelligence feed loaders. For more information see Using Threat Intelligence Feeds.

​Streaming Data into HCP

Streaming Data into HCP