If you’re a mainframe engineer, you’ve probably had this conversation at least once: Someone from the operations team wants a dashboard of job abend trends, RACF activity spikes or workload performance data across LPARs, and the answer is always the same: “We’ll need to route SMF records through Splunk.”

You route them. Then you wait for the Splunk team to ingest the data, build a parser and validate the pipeline. Weeks pass. The license bill grows. The dashboard shows up three months later, looking nothing like what was originally asked for.

The problem isn’t Splunk specifically. It’s the assumption that your SMF data needs to travel through a third-party SIEM to become useful. It doesn’t. Your z/OS system has been writing structured, timestamped, typed operational records since before most observability tools existed. The data is there. What’s been missing is a way to surface it without building a custom parser for every SMF record type or routing everything offboard.

AI changes that math. Specifically, retrieval-augmented generation (RAG) applied directly to your SMF corpus gives you something no dashboard tool alone can — answers in plain English, grounded in the actual records your system wrote, without ingesting everything into a downstream platform first.

The Real Cost of Routing SMF Through Splunk

Here’s what the architecture usually looks like: SMF records get offloaded from z/OS, converted to something Splunk can parse, indexed and then queried. Each step in that chain introduces latency, schema complexity and cost.

A single production LPAR generates millions of SMF records per day. During an incident, that volume can spike by an order of magnitude. You’re not just paying for storage, you’re paying for ingestion bandwidth, index capacity and Splunk license tiers that scale with data volume.

You still need someone who knows which SMF record types to query. SMF type 30 captures workload accounting. Type 80 captures RACF security events. Types 70 through 79 hold RMF performance data. Type 92 tracks UNIX file system activity. Unless your Splunk queries already know this and your field extractions handle the binary record formats correctly, you get dashboards full of null values and engineers who don’t trust the data.

The deeper issue is that Splunk was built for distributed, unstructured log streams. SMF records are structured, binary, domain-specific and deeply typed. The schema mismatch creates ongoing maintenance overhead that compounds every time IBM ships a new z/OS release.

A Different Architecture: AI Directly Over SMF

The alternative is to build a retrieval layer that sits between your SMF archive and your operations team, without requiring a SIEM in the middle.

Here’s how it works in practice. You extract SMF records using standard IBM utilities, SMFDUMP or equivalent and convert them into normalized text representations. Each record type gets its own transformation: Type 30 records become job-level accounting summaries with timestamps, CPU consumption and class names; type 80 records become structured event descriptions with the user ID, resource name and access decision. The binary fields are resolved into human-readable output.

Then that normalized text gets chunked, embedded and indexed using a hybrid retrieval approach: A BM25 sparse index for exact-match queries (useful when an operator types a specific abend code or job name) and a dense vector index for semantic queries (useful when someone asks, “Which jobs had unusual I/O wait last night?” without knowing the exact SMF field names).

That index becomes the backbone of your dashboard layer. Operators submit natural language queries. The system retrieves the most relevant SMF summaries, passes them to a language model with source attribution and returns a grounded answer that cites the actual record. No more guessing which Splunk query to write. No more chasing field names across 200 pages of IBM documentation.

The dashboard itself can be as simple as a query interface or as complex as a multi-panel view of the top SMF record categories, updated on a rolling 24-hour basis. The key difference from Splunk is that the intelligence lives in the retrieval layer, not in the visualization tool. You can swap the front end without rebuilding the underlying pipeline.

What You Can Actually Monitor

This architecture covers a lot of ground your Splunk dashboards probably handle today, and it does so without the ingestion cost.

For performance operations, SMF types 70 through 79 give you RMF interval data: CPU utilization by LPAR, memory frames, I/O rates by channel path and WLM service class activity.

A RAG query such as, “Which LPARs exceeded 85% CPU utilization between 2 a.m. and 4 a.m. last night?” returns a direct answer with the supporting records, not a raw table you have to filter yourself.

For security auditing, type 80 records contain every RACF decision: Successful and failed access events, authority changes and dataset profile modifications. If your security team currently pulls these through Splunk for compliance reporting, you can replace that pipeline with a local index that answers, “Show me all failed access attempts on PAYROLL datasets in the last seven days,” without routing sensitive operational data off the mainframe.

For capacity planning, type 30 workload accounting records hold the raw material for charge-back reports and trend analysis. An AI layer over these records can surface job-level anomalies, such as a batch job that used three times its normal CPU allocation, without requiring you to write a custom parser every quarter.

Where Splunk Still Makes Sense

This isn’t a case for ripping everything out. Splunk makes sense when you need to correlate mainframe events with distributed system logs, cloud infrastructure or endpoint telemetry. If your SOC monitors z/OS RACF events alongside AWS CloudTrail and Microsoft Active Directory in the same incident timeline, that’s a legitimate cross-platform use case, and a dedicated SIEM earns its cost there.

However, for day-to-day operational visibility, capacity monitoring and performance trend analysis, where all the data lives on the mainframe anyway, you don’t need to route it offboard. Build the retrieval layer close to the source, keep sensitive SMF data inside your security boundary and use AI to make that data queryable without a custom dashboard for every question someone can think to ask.

Your SMF records have been telling you everything your operations team wants to know. The data has been there for decades. You just needed a way to ask it something.