How to get data into Splunk

Getting data into Splunk involves taking data from inputs, and then indexing that data by transforming it into individual events that contain searchable fields. Here's a brief overview of how it all works.

Data inputs

A data input is a source of incoming event data. Splunk can index data from the following types of inputs:

• Files and directories—the contents of files and directories of files. You can upload a file for one-time indexing (a oneshot input), monitor for new data, or monitor for file system changes (events are generated when the directory undergoes a change). Files and directories can be included using whitelists, and excluded using blacklists.
• Network events—data that is received over network Transmission Control Protocol (TCP) and User Datagram Protocol (UDP) ports, such as data that is sent from a Splunk forwarder from a remote computer. TCP inputs are separated into raw (unprocessed) and cooked (processed) inputs, with SSL as an option for either type.
• Windows data—data from Windows computers, which includes:
• Windows event log data
• Windows Registry data
• Windows Management Instrumentation (WMI) data
• Active Directory data
• Performance monitoring (perfmon) data
• Other data sources—data from custom apps, FIFO queues, scripts that get data from APIs, and other remote data interfaces and message queues.

Your data inputs and their configurations are saved in the inputs.conf configuration file.

Indexes

The index stores compressed, raw event data. When receiving data from your inputs, Splunk parses the data into events and then indexes them, as follows:

• During parsing, Splunk extracts default fields, configures character-set encoding, identifies line termination, identifies timestamps (creating them if they aren't there), masks sensitive or private data, and can apply custom metadata. Parsing can be done by heavy forwarders. Universal forwarders do minimal parsing.
• During indexing, Splunk breaks events into segments, builds the index data structures, and writes the raw data and index files to disk.

Splunk can usually determine the data type and handle the data accordingly. But when setting up new inputs, you might consider sending data to a test index first to make sure everything is configured the way you want. You can delete the indexed data (clean the index) and start over as needed. Event processing rules are set in the props.conf configuration file, which you'll need to modify directly if you want to reconfigure how events are processed.

Each index is stored as a collection of database directories (also known as buckets) in the file system, located in $SPLUNK_HOME/var/lib/splunk. Buckets are organized by age:

• Hot buckets are searchable, actively being written to, one per index. Hot buckets roll to warm at a certain size or when splunkd is restarted, then a new hot bucket is created.
• Warm buckets are searchable. Oldest warm buckets roll to cold when the number of warm buckets reaches a number limit.
• Cold buckets are searchable. After a set period of time, cold buckets roll to frozen.
• Frozen buckets are not searchable. These buckets are archived or deleted.

You can configure aspects such as the path configuration for your buckets. For example, keep the hot and warm buckets on a local computer for quick access, and put the cold and frozen buckets on a separate disks for long-term storage. You can also set the storage size.

By default, data is stored in the main index, but you can add more indexes for different data inputs. You might want multiple indexes to:

• Control user access. Users can search only in indexes they are allowed to by their assigned role.
• Accommodate varying retention policies. Set a different archive or retention policy by index.
• Speed searches in certain situations. Create dedicated indexes for each data source, search just in the index you want.

The input and index APIs

The classes for working with data inputs are:

• The InputCollection class for working with the collection of data inputs.
• The Input class is the base class for the typed data inputs:
• The MonitorInput class for monitoring files and directories.
• The ScriptInput class for script inputs.
• The TcpInput class for raw TCP inputs.
• The TcpSplunkInput class for cooked TCP inputs.
• The UdpInput class for UDP inputs.
• The WindowsActiveDirectoryInput class for Active Directory inputs.
• The WindowsEventLogInput class for Windows event log inputs.
• The WindowsPerfmonInput class for Windows performance monitor inputs.
• The WindowsRegistryInput class for Windows Registry inputs.
• The WindowsWmiInput class for WMI inputs.
• The InputKind class enumerates the types of data inputs.
• The TcpConnections class for managing TCP connections.
• The UdpConnections class for managing UDP connections.

The classes for working with indexes are:

• The Index class for working with an individual index.
• The IndexCollection class for working with the collection of indexes.
• The IndexCollectionArgs class for retrieving a collection of indexes.

Access these classes through an instance of the Service class. Retrieve a collection, and from there you can access individual items in the collection and create new ones. For example, here's a simplified program for getting a collection of inputs and creating a new one:

// Connect to Splunk
Service service = Service.connect(connectArgs);

// Retrieve the collection of data inputs
InputCollection myInputs = service.getInputs();

// Create a TCP input
TcpInput tcpInput = myInputs.create(port, InputKind.Tcp);

Code examples

This section provides examples of how to use the index and input APIs, assuming you first connect to a Splunk instance:

• To list data inputs
• To create a new data input
• To view and modify the properties of a data input
• To list indexes
• To create a new index
• To view and modify the properties of an index
• To clean events from an index
• To add data directly to an index
• To add data directly to a TCP input
• To add data directly to a UDP input

Here are the available parameters for inputs and indexes:

• Collection parameters
• Input parameters
• Index parameters

To list data inputs

This example shows how to retrieve and list the collection of data inputs that have been configured for Splunk.

When retrieving a collection, you can set additional parameters to specify how many entities to return, the sorting order, and so on. For more, see Collection parameters.

Note  To be able to list inputs, the user's role must include those capabilites. For a list of available capabilities, see Capabilities.

// Get the collection of data inputs
InputCollection myInputs = service.getInputs();

// Iterate and list the collection of inputs        
System.out.println("There are " + myInputs.size() + " data inputs:\n");
for (Input entity: myInputs.values()) {
    System.out.println("  " + entity.getName() + " (" + entity.getKind() + ")");
}

To create a new data input

This example shows how to create a data input. You'll need to provide an input kind and a name. The input kind is a member of the InputKind enumeration. The name you specify depends on the type of input you are creating:

Note  Oneshot inputs are created differently (see To add data directly to an index).

You can set additional properties for the data input in different ways:

• Use the setter methods. The setter methods are the easiest way to set and modify properties, but they aren't available until after the data input has been created. See the next section for more about modifying a data input.
• Create an argument map of key-value pairs. Creating an argument map is the only way to set properties at the same time you create a data input, but it requires a little more work to look up properties and provide values in the correct format. For a list of possible properties, see Input parameters.

This example shows how to create a monitor data input.

Note  To be able to create and modify inputs, the user's role must include those capabilites. For a list of available capabilities, see Capabilities.

// Get the collection of data inputs
InputCollection myInputs = service.getInputs(); 

// Create a new Monitor data input
String monitor_filepath = "/Applications/splunk/readme-splunk.txt";
MonitorInput monitorInput= myInputs.create(monitor_filepath, InputKind.Monitor);

To view and modify the properties of a data input

This example continues from the previous example―it displays the properties of the new monitor input, then modifies a few of them using the setter methods, which set the properties on your local, cached copy of the object. To make these changes to the server, call the Entity.update method. For more about the properties you can set for different types of data inputs, see Input parameters.

Note  To be able to create and modify inputs, the user's role must include those capabilites. For a list of available capabilities, see Capabilities.

// Retrieve the new input
String testinput = "/Applications/splunk/readme-splunk.txt";
MonitorInput monitorInput = (MonitorInput) service.getInputs().get(testinput);

// Retrieve and display some properties for the new input
System.out.println("Name:      " + monitorInput.getName());
System.out.println("Kind:      " + monitorInput.getKind());
System.out.println("Path:      " + monitorInput.getPath());
System.out.println("Index:     " + monitorInput.getIndex());
System.out.println("Whitelist: " + monitorInput.getWhitelist());

// Modify some properties and update the server
System.out.println("\nSet some properties\n");
monitorInput.setIndex("test_index");
monitorInput.setWhitelist("phonyregex*2");
monitorInput.update();

// Display the changed properties again to show the change
System.out.println("Index:     " + monitorInput.getIndex());
System.out.println("Whitelist: " + monitorInput.getWhitelist());

To list indexes

This example shows how to retrieve and list the indexes that have been configured for Splunk, along with the number of events contained in each. The collection is also sorted by using some additional parameters.

When retrieving a collection, you can set additional parameters to specify how many entities to return, the sorting order, and so on. For more, see Collection parameters.

// Retrieve the collection of indexes, sorted by number of events
IndexCollectionArgs indexcollArgs = new IndexCollectionArgs();
indexcollArgs.setSortKey("totalEventCount");
indexcollArgs.setSortDirection(IndexCollectionArgs.SortDirection.DESC);
IndexCollection myIndexes = service.getIndexes(indexcollArgs);

// List the indexes and their event counts
System.out.println("There are " + myIndexes.size() + " indexes:\n");
for (Index entity: myIndexes.values()) {
    System.out.println("  " + entity.getName() + " (events: " 
            + entity.getTotalEventCount() + ")");
}

To create a new index

When you create an index, all you need to specify is a name. As usual, you can set additional properties when you create the index by also creating an argument map of key-value pairs. Or, you can wait and use the setter methods after the index has been created, as described in the next example.

Note  To be able to create and modify an index, the user's role must include those capabilites. For a list of available capabilities, see Capabilities.

This example shows how to create a new index.

Note  If you are using a version of Splunk earlier than 5.0, you can't delete indexes using the SDK or the REST API—something to be aware of before creating lots of test indexes.

//Get the collection of indexes
IndexCollection myIndexes = service.getIndexes();

//Create a new index
Index myIndex = myIndexes.create("test_index");

To view and modify the properties of an index

This example shows how to view the properties of the index created in the previous example and modify its properties. For more about the properties available for indexes, see Index parameters.

Note  To be able to create and modify an index, the user's role must include those capabilites. For a list of available capabilities, see Capabilities.

// Retrieve the index that was created earlier
Index myIndex = service.getIndexes().get("test_index");

// Retrieve properties      
System.out.println("Name:                " + myIndex.getName());
System.out.println("Current DB size:     " + myIndex.getCurrentDBSizeMB() + "MB");
System.out.println("Max hot buckets:     " + myIndex.getMaxHotBuckets());
System.out.println("# of hot buckets:    " + myIndex.getNumHotBuckets());
System.out.println("# of warm buckets:   " + myIndex.getNumWarmBuckets());
System.out.println("Max data size:       " + myIndex.getMaxDataSize());
System.out.println("Max total data size: " + myIndex.getMaxTotalDataSizeMB() + "MB");

// Modify a property and update the server
myIndex.setMaxTotalDataSizeMB(myIndex.getMaxTotalDataSizeMB()-1);
myIndex.update();
System.out.println("Max total data size: " + myIndex.getMaxTotalDataSizeMB() + "MB");

To clean events from an index

This example shows how to clean an index, which removes the events from it.

// Retrieve the index that was created earlier
Index myIndex = service.getIndexes().get("test_index");

// Clean events from the index, printing the before-and-after size
System.out.println("Current DB size:     " + myIndex.getCurrentDBSizeMB() + "MB");
myIndex.clean(180);
System.out.println("Current DB size:     " + myIndex.getCurrentDBSizeMB() + "MB");

To add data directly to an index

There are different ways to add data directly to an index, without configuring a data input. First, retrieve an index using the Index class, and then use one of the following methods:

• Use the upload method to upload a single file as an event stream for one-time indexing, which corresponds to a oneshot data input. You'll need to specify the file and path to upload, too.
• Use the submit method to send an event over HTTP. You'll need to provide the event as a string, and can specify values to apply to the event (host, source, and sourcetype).
• Use the attach and attachWith methods to send events over a writeable socket. You can also specify the values to apply to these events (host, source, and sourcetype).

Here is an example of uploading a single file:

// Retrieve the index for the data
Index myIndex = service.getIndexes().get("test_index");

// Specify a file and upload it
String uploadme = "/Applications/splunk/readme-splunk.txt";
myIndex.upload(uploadme);

Here is an example of using the submit method to submit single events directly to the index over HTTP:

// Retrieve the index for the data
Index myIndex = service.getIndexes().get("test_index");

// Specify  values to apply to the event
Args eventArgs = new Args();
eventArgs.put("sourcetype", "access_combined.log");
eventArgs.put("host", "local");

// Submit an event over HTTP
myIndex.submit(eventArgs, "This is my HTTP event");

The submit method opens a new connection for each event, so if you have many events to send, this method may not perform well. In this case, use the attach method get an open socket to Splunk that you can write your events to. Here is an example of sending an event (with a timestamp) to a socket opened by the attach method:

// Additional imports
import java.io.*;
import java.net.Socket;
import java.text.SimpleDateFormat;
import java.util.Date;

...

// Retrieve the index for the data
Index myIndex = service.getIndexes().get("test_index");

// Set up a timestamp
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ssZ");
String date = sdf.format(new Date());

// Open a socket and stream
Socket socket = myIndex.attach();
try {
     OutputStream ostream = socket.getOutputStream();
     Writer out = new OutputStreamWriter(ostream, "UTF8");

     // Send events to the socket then close it
     out.write(date + "Event one!\r\n");
     out.write(date + "Event two!\r\n");
     out.flush();
} finally {
     socket.close();
}

You should always wrap a socket created with attach in a try block, and the socket should be closed in a finally clause. Because this is easy to forget, the Splunk SDK for Java provides the attachWith method to handle it for you. The attachWith method takes the body of the try block as an anonymous implementation of the ReceiverBehavior interface, and handles the set up and tear down around it automatically. For example:

// Additional imports
import java.io.*;
import java.net.Socket;
import java.text.SimpleDateFormat;
import java.util.Date;

...

// Retrieve the index
Index myIndex = service.getIndexes().get("test_index");

// Open the socket and stream, set up a timestamp
myIndex.attachWith(new ReceiverBehavior() {
     public void run(OutputStream stream) throws IOException {
         DateFormat dateFormat = new SimpleDateFormat("yyyy/MM/dd-HH:mm:ss");
         String date = dateFormat.format(new Date());
         String eventText = date + " Boris the mad baboon!\r\n";
         stream.write(eventText.getBytes("UTF8"));
     }
});

Note that any local variables from the outer scope that you use in the block must be declared final.

The attachWith method creates the socket and a stream attached to it, then calls the run method that you implement with that stream. When run finishes, whether from returning or throwing an exception, the socket is closed. Unless you have a good reason to do otherwise, you should use attachWith instead of attach.

To add data directly to a TCP input

There are different ways to add data directly to a TCP input. First, retrieve a TCP input using the TCPInput class, and then use one of the following methods to send data:

• Use the submit method to send a single event over its own connection.
• Use attachWith to get an open output stream to the TCP input that you can write an arbitrary number of events to.
• Use attach if you need to get a raw socket to the TCP input without the protections provided by attachWith.

Here is an example of sending a single event to a TCP input with submit:

// Retrieve the input
TcpInput myInput = (TcpInput)service.getInputs().get("10000");

// Send a single event to the input
myInput.submit("This is my event.");

The submit method opens a socket, sends the event, and closes the socket. You can get an open socket directly and write events to it by calling the attach method:

// Additional imports
import java.io.*;
import java.net.Socket;
import java.text.SimpleDateFormat;
import java.util.Date;

...

// Retrieve the input
TcpInput myInput = (TcpInput)service.getInputs().get("10000");

// Open a socket
Socket socket = myInput.attach();

// Wrap the socket in a try block so we can close it in case of an error
try {
     OutputStream ostream = socket.getOutputStream();
     Writer out = new OutputStreamWriter(ostream, "UTF8");

     DateFormat dateFormat = new SimpleDateFormat("yyyy/MM/dd-HH:mm:ss");
     String date = dateFormat.format(new Date());

     // Send events to the socket then close it
     out.write(date + "Event one!\r\n");
     out.write(date + "Event two!\r\n");
     out.flush();
} finally {
     socket.close();
}

Calling attach directly requires you to manage cleaning up the socket in the presence of errors. Because this is easy to forget, the Splunk SDK for Java provides the attachWith method to handle it for you. The attachWith method takes the body of the try block as an anonymous implementation of the ReceiverBehavior interface, and handles the set up and tear down around it automatically. For example:

// Additional imports
import java.io.*;
import java.net.Socket;
import java.text.SimpleDateFormat;
import java.util.Date;

...

// Retrieve the input
TcpInput myInput = (TcpInput)service.getInputs().get("10000");

// Open the socket and stream, set up the timestamp
myInput.attachWith(new ReceiverBehavior() {
     public void run(OutputStream stream) throws IOException {
         DateFormat dateFormat = new SimpleDateFormat("yyyy/MM/dd-HH:mm:ss");
         String date = dateFormat.format(new Date());
         String eventText = date + " Boris the mad baboon!\r\n";
         stream.write(eventText.getBytes("UTF8"));
     }
});

Note that any local variables from the outer scope that you use in the block must be declared final.

The attachWith method creates the socket and a stream attached to it, then calls the run method that you implement with that stream. When run finishes, whether from returning or throwing an exception, the socket is closed. Unless you have a good reason to do otherwise, you should use attachWith instead of attach.

To add data directly to a UDP input

UDP is an unreliable protocol—you can dispatch data using it, but with no guarantees of arrival. Because there is no notion of a connection, UDP inputs have a single method, submit, for sending events, as shown in the following example:

// Get a UDP input to send to
UdpInput myInput = (UdpInput)service.getInputs().get("9999");

// Send an event via a UDP datagram
myInput.submit("This is my event.");

Collection parameters

By default, all entities are returned when you retrieve a collection. Using the parameters below, you can specify the number of entities to return, how to sort them, and so on. Set these parameters using setters from the CollectionArgs class for inputs, the IndexCollectionArgs class for indexes, or create a generic Args map.

Input parameters

The properties that are available for each type of data input corresponds to the parameters for the following REST API endpoints:

• Active Directory: POST data/inputs/ad endpoint
• files and directories (monitor): POST data/inputs/monitor endpoint
• files and directories (oneshot): POST data/inputs/oneshot endpoint
• scripts: POST data/inputs/script endpoint
• TCP, cooked: POST data/inputs/tcp/cooked endpoint
• TCP, raw: POST data/inputs/tcp/raw endpoint
• TCP, SSL: POST data/inputs/tcp/ssl endpoint
• UDP: POST data/inputs/udp endpoint
• Windows event log: POST data/inputs/win-event-log-collections endpoint
• Windows performance monitor: POST data/inputs/win-perfmon endpoint
• Windows Registry: POST data/inputs/registry endpoint
• WMI: POST data/inputs/win-wmi-collections endpoint

This table summarizes the available parameters for different types of inputs:

Index parameters

The parameters you can use for working with indexes correspond to the parameters for the data/indexes endpoint in the REST API.

The following parameters are available for indexes: