Research and plan your ITSI module

While it might be tempting to begin developing an ITSI module right away, taking the time to research and plan before writing any code or creating a prototype is essential. Research and planning not only helps you determine what to build, but whether what you have in mind is the right thing to build.

Start conceptualizing your ITSI module by answering some basic questions, beginning at a high level and then working down to the details.


What is the module going to do?

ITSI modules allow you to customize ITSI to work with a particular domain and one or more associated technologies. You can then create modules based on solutions, such as a module that monitors popular software suites and incorporates multiple technologies.

Consider the domain you want your module to target, the key functions to monitor, and the technologies that your domain supports. The technology can also indicate the type of data you collect.

Things to consider:

  • The scope of the domain.
  • The key technologies that are associated with the domain.
  • The size of the organization that uses each technology, which determines the amount of data that can be collected.

Here are some examples of domains and their technologies to give you some idea of how to conceptualize the scope and purpose of your module:

  • The Operating System domain deals with technologies that run processes and manage base memory, disk, processing and networks. Technologies include Windows and Linux.
  • The Virtualization domain deals with technologies that run virtual machines or containers on a host operating system. Technologies include VMware and Hyper-V.
  • The Relational Database domain deals with technologies related to relational storage and retrieval scenarios. Technologies include SQL Server and Oracle.

How and where are you getting data for the module?

Here are some considerations for the data that your module will work with:

  • How are you getting your data? Will you work with an existing add-on, or do you need to build an add-on to collect data specifically for your domain?
  • If you need to build an add-on, see the Splunk Add-on Builder User Guide.

  • Do you own the data? Or will you need to normalize the data you receive from different vendors or domains?
  • Do you need to use a data model?
  • A data model provides a hierarchical structure and view of your data, and applies tags and event types to your data for easier searches. However, for ITSI modules, using searches based on data models is not recommended. For details, see About data models in the Knowledge Manager Manual.


What are the entities for the module?

What do you want to monitor? Identify the entities in your domain, which correspond to infrastructure components with attributes and are typically something that can be tracked using a metric.

Examples of entities include servers, hosts, applications, or even users that are part of an Active Directory (AD) environment.


What are the KPIs for the module?

What do you want to track? Once you know what your entities are, determine what you want to measure using key performance indicators, or KPIs. A KPI is a performance or operational indicator that shows how an entity is performing and has:

  • A resource counter to be queried or collected.
  • A collection or query frequency.
  • A computational method to aggregate data over the frequency.
  • Optionally, a set of generally-accepted performance thresholds.

To determine which KPIs are important for your module, consider doing the following:

  • Talk to existing domain experts and interview their customers.
  • Explore support forums.
  • Read product documentation.
  • Explore competitor offerings.

What are the services for the module?

Which KPIs and entities do you want to monitor together as a group? An ITSI service is a collection of KPIs and entities, and when combined can provide a window into the state of the group. For example, a group of KPIs can provide a health score to determine the health of a service―calculated based on the importance and status of all KPIs according to a predetermined frequency.

Services could be a collection of hosts in a data center, an external API that is critical for your application delivery, load-balancing groups from across your infrastructure, and database tiers that supply information to your applications.


What else do you need?

Do you anticipate needing to drill down into any KPI data for deeper investigation? Will you need any visualizations to display data in a specific way?


Example: Planning the Web Server Module

Here's an example of how the Splunk team planned the Web Server Module for Splunk IT Service Intelligence. This exercise answers the following questions:

  • What do we want to monitor?
  • What are the entities for our domain, and how do they relate to each other?
  • What are the metrics for these entities?
  • What other details do we know about the entities?
  • What are the searches needed to produce these metrics?
  • What kind of drilldown information should we display?

By gathering detailed information in this way, you'll make the development of your own ITSI module much easier.

    Note  This exercise is an example and is not meant to accurately describe the final Web Server Module.

Targeted Technologies

ApacheHTTP Web ServerSplunk Add-on for Apache Web Server
MicrosoftIIS Web ServerSplunk Add-on for Microsoft IIS

Entity Types

Web ServerThe web server host and site

Web Server Inventory

TitleThe unique title of the entity, the host name.string
VendorThe company/group that produces the web server.string
ProductThe web server package.string
VersionThe version number of the product.string
IPThe server IP address.IP
HostThe host name of the web server.string
FQDNThe full-qualified domain name.string
CPU CoresThe number of CPU cores for the server.count
MemoryThe total amount of memory on the server.GB
StorageThe total amount of storage for the server.GB

KPIs and Drill Down

NameDescriptionUnitApplies toThresholdDrill DownSearch query
4XX ErrorsThe count of 4xx response statusescountWeb Server, ApplicationAdaptive: 7 DaysBreakdown by 4xx codes, count by 401, 402, 404TBD*
5XX ErrorsThe count of 4xx response statusescountWeb Server, ApplicationAdaptive: 7 DaysBreakdown by 4xx codes, count by 401, 402, 404
Percent Error CodesThe percentage of total responses that resulted in a 4xx or 5xx code%Web Server, ApplicationH: 10
M: 5-10
N: 0-5
Time chart showing max values over time
Response TimesThe average response time for all requestsmsWeb Server, ApplicationAdaptive: 7 DaysResponse time graph of top 10 slowest requests by URI and response time
Hits Per MinuteNumber of requests handled per minute (RPM) by the web serverRPMWeb Server, ApplicationAdaptive: 7 DaysGraph of throughput for web server
Bytes InSum of bytes from requests to the server by secondKBWeb ServerAdaptive: 7 DaysGraph of bytes in
Bytes OutSum of bytes sent out from the server by secondKBWeb ServerAdaptive: 7 DaysGraph of bytes out
Status by Client AgentThe count of responses by status code and user agent (browser, client)countWeb Server, ApplicationNoneGraph the count of response status by client agent
Status by Client IPThe count of responses by status code and client IPcountWeb Server, ApplicationNoneGraph the count of response status by client IP
Application TableTable of all sites/applications served by server with info about 4XX/5XX errors, response times(multiple)Web ServerN/ATab and table showing the site/application name, average response time,
average error count
Total ErrorsCount of events in error logcountWeb Server, ApplicationAdaptive: 7 DaysHistogram of all error log growth
Error Log EventsList of all error log events for this server/application(table)Web Server, ApplicationN/AErrors for this particular web server or application
OS LinkLink to the OS Host ModuleURLWeb ServerN/ALink to OS Host Module
AvailabilityPercent of successful transactions out of total transactionsbinaryWeb ServerNoneShow UP/DOWN


    * Be sure to fill these searches in later. For details, see Explore your data and create searches.