Ken Gross
Mainframe Software Specialist
With year after year exponential growth of Big Data, organizations continue to experience the challenges of how to analyze this information to make better decisions. The data is now arriving from everywhere, for example sensors used to gather shopper information, supply-chain data, logging data across the enterprise, smart appliance data, purchase transactions and cell phone GPS signals, just to name a few. Following are two questions that are being asked.
1. How do my data scientists create predictive models that provide for more accurate decision-making?
2. How do I utilize analytic solutions, predictive analytics or even machine learning to allow the corporation to make better decisions?
In today’s article, I would like to highlight a very useful solution from IBM called the IBM® Common Data Provider for z Systems™; and specifically, to narrow the focus to mainframe operational analytics, as well as mainframe z logging data.
IBM® Common Data Provider for z Systems® is a collection of software programs that rapidly centralize IBM z/OS® operational data and feed it to analytic platforms. It accesses and distributes data in near real- time and batch modes. It also defines multiple data collection sources (140+ data sources are supported, including all standard SMF records plus syslog and applications log types like CICS®, IMS™ and more). A couple examples of this are Splunk and the Elastic Stack (formerly ELK Stack).
IBM® Common Data Provider for z Systems™ automatically monitors z/OS log data and SMF data, and then forwards it to the configured destination. In each logical partition (LPAR) from which you want to analyze z/OS log data or SMF data, a unique instance of IBM Common Data Provider for z Systems must be installed and configured to specify the type of data to be gathered. The destination for that data is called a subscriber.
Let’s discuss a bit about the analytics platform. An analytics platform is a software program, or group of dedicated systems and software packages, that are configured to receive, store, and analyze large volumes of operational data.
Analytics platforms:
- IBM® Db2 Analytics Accelerator for z/OS® – a database application that provides query-based reporting
- IBM Operations Analytics for z Systems™ – an on-premises product that can receive large volumes of operational data for analysis and can provide insights and recommended actions to the system owners, which are based on expert knowledge about z Systems and applications
- Platforms, such as Elasticsearch, Apache Hadoop, and Splunk, can receive and store operational data for analysis. These platforms do not include expert knowledge about z Systems and applications, but users can create or import their own analytics to run against the data.
Tip
- Apache Kafka is an open-source, high-performance, distributed streaming platform that can be used to publish and subscribe to streams of records. It can be used as a path to data consumers such as Apache Hadoop, Apache Spark Streaming, and others.
A deeper view of how the operational data will flow to your choice of analytics platform:
As illustrated in Figure 1, operational data (such as SMF record type 30 or z/OS SYSLOG data) is gathered by data gatherers, such as the System Data Engine or the Log Forwarder, and can be streamed to multiple subscribers. The data gatherers send the data to the Data Streamer, which transforms the data before it sends the data to the subscribers.
The flow of data is controlled by a policy that you define in the IBM Common Data Provider for z Systems Configuration Tool.
Policy
In IBM Common Data Provider for z Systems, Policy is a set of rules that define what operational data to collect and where to send that data.
Subscriber
In the IBM Common Data Provider for z Systems configuration, Subscriber is the software that you define to receive operational data. You can have both on-platform and off-platform subscribers.
On-Platform Subscriber
A subscriber that is on the same z/OS system, or in the same logical partition (LPAR), as the source from which the operational data originates is called an on-platform subscriber.
Off-Platform Subscriber
A subscriber that is not on the same z/OS system, or in the same logical partition (LPAR), as the source from which the operational data originates is called an off-platform subscriber.
Figure 1. Flow of operational data among IBM Common Data Provider for z Systems components to multiple analytics platforms
This is a high-level overview of what is possible. You can now improve the flexibility of your analytics and gain more consumable options. With this understood, there is still a lot that can be explored with your technical teams. If you want greater insights and details on how IBM® Common Data Provider for z Systems® can benefit you, please reach out to your Mainline Account Executive directly, or click here to contact us with questions on this topic or any other.