Name: IBM® OMEGAMON®: Powered by Legends, Built for the Future Virtual Summit
Uploaded: 2025-12-03T17:33:16.524Z
Duration: 2 h 46 min 8 s
Description: IBM® OMEGAMON®: Powered by Legends, Built for the Future Virtual Summit

Transcript for "IBM® OMEGAMON®: Powered by Legends, Built for the Future Virtual Summit": Yeah. Welcome to our panel session that is scheduled for our Omega Month Summit this year. And, we from Rocket and from IBM, we are very happy to have two customers showing up today and talking a little bit about what they, did with the or what they are still doing with the Omega one family or, some kind of tips that they just figured out with using the tool and, again. So we are happy to to welcome first Claus Mikkelsen. He works for no Nordea Nordea. And, he is not an special, is what you said. He is the guy who is responsible for performance in his shop, and he made some very good experience. And, but I'd like to pass it to to Claus, and please introduce yourself a little bit, and then you can start what you figured out with your performance issues. Thank you. Yeah. Thank you, Thomas. Yes. I work for, as Thomas said, I work for Nordea, And, I call myself the policeman of Nordea because, if anyone in Nordea uses too much CPU on the mainframe, they will hear from me, sometimes, faster and sometimes slower slower than that. Then back in 2023, I think, we found out that, OmegaMon was using a hell of a lot of CPU. And we start thinking about, what should we do about that? And, I actually then learned a lot about, Umairment. And, I tried different things, and none none of these things worked. That was, specifically with these, situations that I worked with, trying to to optimize those. But, I did a lot of different things, and, none of those helped. So I actually ended up reaching out to to Rocket. So I think that was, both an introduction of me and, my part of this, Omega one. I'm, Thomas Oroson. I work as a systems programmer or a tech lead here at the Handelsbanken. And my team, we handle all the operating system, disk subsystem, tape, EDPs, and and a lot of other products and and also performance and monitoring of some kind with with OmegaMon as well, the configuration. And and, I'm gonna talk a little about how how we, actually, migrated from from PowerMian to to a configuration manager and how that has helped us a lot and save a lot of time to make made our, actually, our job a lot easier having a run putting a service and and making sure that everything around smoothly. Okay. Thanks for the short introduction. And, okay. So we continue with, with Klaus again. And, so, Klaus, if you can go into such, more details about what you were just mentioned in your introduction. Yes. What we did was, we reached out to to Rocket, as I said, and, then we had a meeting with a lot of, experts and peers from, from Rocket. And we were, then we found out that we should send a lot of information to Rocket about, what we saw and also the different logs. And then we then Rocket made a very extensive report to us about all the different findings that they had. It was a really, really good and detailed work, which we appreciated a lot. And the the the bottom of it the bottom of it, was that, there was, a lot of different things that we could do. And then we started doing a lot of those, but, none of it really, really helped. That was really, really strange. And then we had to, to run some, some traces. And one of the problems was that this problem that we had actually mostly showed up Wednesday morning, which was really, really strange. Why why why Wednesday morning? And then where there was a lot of IP traffic going on between the, the the temp top and the and the mainframe. So when, then we sent all these traces to, to Rocket, and, and they came back and said, yeah, it looks like you are doing some kind of, what's it called? You're doing some data warehousing things. It was primarily the MQ the MQ monitoring task and this, OMIC DSST task that is responsible for the communication between the mainframe and, and the hub that was, using a load sub CPU. And then we could see from from the, from these traces that there was a huge amount of, of data that was sent from, the MQ tasks to to the, to the data warehouse, and then we started thinking, oh, that was strange. We have a very, very old, OmegaMon installation that has been set up, at that time, maybe fifteen to twenty years ago. And then one of the persons from Nordea that was participating in in in all of this, tray tracking, he, he had actually been involved in the initial installation, and he remembered that they have had a lot of discussions going back and forth about, this with the Stata warehousing. Should we use it or should we not use it? And then they ended up saying, okay. We will use it. And in the meantime, we have had, installed a lot of more in queue stuff. And, and that has simply made this, data warehousing explode, but we weren't actually using the data for anything. So it was a really easy easy, decision to say, okay. We will simply turn it off because we are not using this data. We actually didn't know that we were collecting them anyway, and everybody had forgotten about it. So that was that was our first step, and that helped, tremendously on our MQ, performance. And then, we continued with this Omid PSST task was actually still using loads of CPU every Wednesday morning. So we dig further into that trace, and it was it was hard because it was not every Wednesday. So sometimes we woke up in the morning at 06:00 logged on and did all the tracing, but nothing happened. Then we were really disappointed. But finally, we succeeded in a Wednesday morning with a really good trace. And, and then we could see that, the number of nodes that was, connected to to the temp shop from all our different servers outside the mainframe, they were going offline and online, Wednesday morning, probably because of some kind of service going on. And that was primarily this was only in our test environments that we saw all those huge spikes. So we, of course, have a huge amount of test servers, and they were all connected to the temp shop. And then the design thing in Omega One is that, Omega One thinks that everybody, when something goes on or off in the in the that is connected to the temp sub, then everybody else should know about it. So that's why the Omega task in our test environment, it got totally overloaded with all these data being sent to to the mainframe. Now this node has, gone offline now. And and now this node has gone online again. We used loads of CPU for that. So and we thought, okay, how can we solve this? And we decided that we would, use, a separate, tempestruck for the mainframe so that we actually set all these all the different servers that was running on the temp shop in the distributed environment outside the mainframe, they would have their own temp shop. Then, all the only and then we would have a separate one for the mainframe so that they would not interfere with the mainframe because we didn't care about that they were going on offline and online on the mainframe anyway. Or should we use that information? That was actually that was what I was thinking, and, that has not, and we have been satisfied with that. So we varied that. So we did set up this separate environment, temp shop environment for our mainframe in our test environments. And now I can show you some results here. So the end result was that, we saved more than 50 of the CPU on Omega monitor on the mainframe in general. So and for the Omid DSST task, we saved more than 35% of the total CPU consumption for the OmidiMon task. And even better for the OmidiM2 task with this data warehousing going on, there we saved more than 71%. So that was really, really good results that we saw because of this. So those numbers look very impressive. So, Klaus, and, this was a result of the changes you did and, in the work together with with the Rocket team. And, and what I'd like to point out here is for all the listeners here to our call today is that every customer has this option or can take this option to get in contact with our team. And, what we are doing is we're just asking you for some data, and that's what we did with with CLOS and at Nordea. And our team analyzes those data and is coming back with some suggestions and with some findings. And, and then, usually, we work together with with our clients. And what Klaus did is he he took over these suggestions, and he changed the system. And I saw very impressive results we we got in your environment. And, yeah. So I think it's very impressive. Yeah. Thank you. Yeah. I would say to, to all the listeners here that, please please please use this opportunity to reach out to Rocket. I think, you have been extremely helpful. And, yeah, so that's that's really great. You're really listening to to what we are saying, and, and you also give a really competent, advice back. So that's really good. Okay. I am a very satisfied customer. Oh, thanks. Thanks, Claus. Okay. So, Thomas, would like move over to you and, talk a little bit about what your experiences are with with Omega One. Yeah. So, yeah, Humboldt's banking, not little but us, but quite an old bank and one of the big banks in Nordics. We're well known for our low risk, low credit score. But yeah, that's over to our pain points with running, with, with Parhamgan. There was we actually had a few people who who could do it, but it was, many jobs that need to be run-in the correct sequence. That's why we also yeah. And those are not always clear which job we need to run when we do that. So depending on how much experience you have when doing a reconfiguration with new new parameters or when servers put on, it could be easier. It could be very hard depending. And, since the way we roll out service, it must be planned directly after IPL to do this usually. And, also, we needed to verify that the activation has worked with the new pair parameters or or in and also we will need that or or if it was incorrect, we need to stop Omega one again and do a new reconfiguration and yeah. Then maybe, if somebody forgot to run a job or something to copy a dataset. That's that's quite a big pain point, actually. So then come this new configuration manager that we then decided to to try out and and actually found out that was really good. A much easier way to to do a configuration. So there is a migrate option where you put in your old and your new. So you just you can just run it and and then just take out the parameters you have today and put it in a new dataset so you can easily migrate from Power BI into to Configuration Manager. And this way also when we did this, we could also verify that all the parameters that we needed and wanted for from the different subsystems or or were actually taking over. We also have to change the qualifiers, but that was a restriction in length that that is not an issue anymore for for with new cuts and drillers. And we did this to make the configuration process easier and with with less errors with more more automatic. And that, yeah. And and this was also the to minimize the downtime when when doing actually changes. Only one job need to be run, and and we saved about up to thirty minutes per helper every time we did this. So how how did we do this stuff? Yeah. Yeah. If we need new changes, we can just go in in the definitions and and change them ahead of our, of the IPL, or or if somebody asks for a new change. We just put them in and and then it will be activated during the IPL. That's quite quite convenient, actually. You'll have to add the changes in the round jobs because we we do this. Like, if if there's a new new fixes that we need to do that requires you to run the job, the configuration job, we will we have a intimate kind of automated way to doing this during the IPL so we we don't have to even get up in the morning when we're doing our IPLs. Let's see. Yeah. So just describe a little bit how how we deploy our code. We we have one Sysplex where we do the installation, and then we have a shared disk where we dump and restore the volumes to to the target. Like, this in the production Sysplex. This is an example. So it's kind of fully automated. When when we had done the service, we just run a job group, and it deploys this new code to a different Sysplexus, and it's ready to IPL. And we use GDPS to and and workflow scheduler to to do the IPL. It's fully automated. It's it's just that the operators are monitoring and and are also starting the IPLs because we want to make sure that when the IPL starts, we don't want to have any issues in the Sysplex, but, otherwise, it could be fully automated, so on time. The first thing we do then in in, to to, do this configuration on the MegaMon. First, when we, after we have taken down and moved and make sure the variable correct things to do the IPL, we first, we stop the MegaMon. But although MegaMon will be stopped in that l part, and directly after that we start the IPL. Afterwards, that stop order system automation. Then they just do the stop in a stop command to GDPS, stop to LPAR. And, you will be adding a new configuration job to the s two. So when a new, dialer comes up with new service, this job will then automatically run when, all the things are available, like, yeah, just two and so on in in for that helper. So it would run as fast as it's possible to to run the job in in that new help or the new configuration or or the new code. So we can take advantage of services and and changes that needs to be implemented. And also, if you get a return code of four or more, you can actually halt the whole IPL and then somebody can call us and then we can fix it. It kind of never happens, but we can do this depending on the environment actually. For For some environments in the production, if it's a return code form, we have operators have to call us and then we verify before we continue taking up the otherwise, in some some sense, flexibility just, yeah, we'll just we'll get notified and then fix it. Well, we'll fix that when we get in the morning instead so we don't have to be woken up. And we also have SDA enabled, so so all new updates are propagated to clients that need them so as well. The hub console we usually do it manually since it's not usually that much that needs to be done for that one. It's only special cases when we're doing an upgrade of the new version. We do the hub temps first before we are doing the other agents. And I can also show you an example of how I how simple this actually this configuration of configuration actually look it's like just like this. We also have an, action generates with the RTD name and and the high level qualifier of your where you have your omega one. So it's and it's, like, it's it's fully automated, so it runs during IPL applications. We we do not involve only when if if but it hasn't almost ever it has I think it happened that once we got where the full dataset or something. But otherwise, we haven't had any problems at all running this. So it's run automatically at every IPL, so we don't forget to add new parameters or when parameters from codes are added to that. You need to run the configuration job to to run it in the way you know. And it saves a lot of time and a lot of you don't need to pay full experience with Power BI and then make sure that you run the correct jobs or otherwise you would miss something. So this is I think it's a really good good step forward using configuration manager, actually. Yeah, that's kind of all all from me. Okay, thanks Thomas and, just like as a as a repeat of what you did is, the configuration manager for all the listeners here in our call today is it's it's the new way of how to install, how to maintain, how to work with OmegaMon instead of using Palm Gen. It doesn't mean that Palm Gen is is the way. Now you can use it if you like. You can stay on Palm Gen. But much better way of doing it is, as we have seen in in what Thomas was presenting today, is using configuration manager. And, as I remember from our discussions we had earlier last week is that, the way to get into was very simple. Or was it complicated to to get into the configuration manager logic and No. Just you, like, you run a migrate job. Yeah. And then and then you you verify the output that all the configuration parameters are as you as you want them to be. But but yeah, we had no issues with with migrating. It was a few years ago now, but it's it just worked and then, yeah, all that up has to be run with configuration management. That's a very simple way of migrating and then, yes, configuration every time you need it. Yeah. And what I also like to mention here in our call today is that everyone who is listening today and he doesn't hear anything about configuration manager, please feel free to contact us later today. We have another session where we are introducing our concierge program. And the concierge program is is a part of offering what we at Rocket Software have for our customers. And the configuration manager could be part of this of this program, but you will see this later today. And, yeah, some more questions from Thomas to Claus. Maybe you you like to interchange something by yourself. If not so feel free to to ask everybody. And, if not, I'd like to thank you both and for the very good insights you gave us and, your experience you made and also with the help we we were offering because we we like, if you like it, what we are doing. And so that is what we like to point out here. And to all of our listeners today, please feel free to to use our service. And, and, also, if you like to show up here, like Claus or Thomas, to talk a little bit about what you are doing in your shop, feel free to contact us. And, we'd like to welcome you, maybe, in one of our next summits. And, again, thank you, Claus. And it was very interesting to see what you are doing for performance in your shop and, also, many, many thanks to you, Thomas. And, with this, I'd like to close the panel for today. Thank you. Thank you. Yeah. You too. Thank you. Bye bye. Bye. This segment of the demo will show the OmegaMon WebUI sample dashboards for OmegaMon DB2. We start with the enterprise overview dashboard that's part of the sample dashboards. You can jump to the landing page of Omega on DB2 by clicking the DB2 link and it will show what available data sharing groups are being monitored by the respective data source which essentially is the hub temps running on zOS. For each group you will then also see the respective members of this group and in addition standalone subsystems that are not part of the data sharing group. From these entries you can then jump into more detail level views. For example a data sharing group threat activity will open in a separate tab in your browser and here you see all active threats and you can page through the threats look at the details and also take action. For example, if you want to cancel a specific thread, you take the respective action and you can cancel the thread. The sample dashboards have additional link to other information that Omega one d v two can show. For example subsystem statistics data. On this screen you see a mix of real time but also near term historical data that is collected for the respective subsystems using the PDS collection. On the top you have threat utilization parameters, buffer pull hit ratios, but you also see active threats in context. Another interesting part of it could be for example accelerator statistics. So let's select a system that has accelerators connected to it and you will see an overview of the accelerators connected or paired with the respective subsystem PCB2 and RSC2. There are five accelerators. You see the product IDs, they have different versions, the number of processors, whether they are online or not, whether they are currently executing queries, what the buffer pool iterations are and so forth. From here you can then build dashboards to go into more details about your accelerator. Another interesting aspect is to look at history data from this dashboard. So let's look at the, threat history data. The threat history data workspace that we deliver is based on the same information that you see in enhanced thirty seventy threat history. So you have an overview of the data store, how many records are in it, what's sold is the newest time stamp, what's the status of your history data sets, what accounting classes are turned on for collection. You can then select the threat that has terminated and drill down into the details. You can filter by the threat type, so I filter by RSAF in this case. I drill down into the details of the threat in history and can look at the class one two breakdown in more detail. I can look at the class three wait times to see what's going on on the waiting side and to jump back to the overview screen and also look at things like buffer pool details for the specific threat that has terminated. This concludes the demo of the sample dashboards for OmegaMond DB2. In this segment, I will show how you can use the streaming capabilities of OmegaMon data provider in the context of OmegaMon AI for DB two six ten. In this case, we will stream accounting file or statistic file data to an Elasticsearch stack. What we see on this stream is the sample job provided to you in TKO2 sample library to stream the data. You see the reference to the ODP library up here and in the steplib besides the ODP library also reference the OmegaMon DB2 product library tkandmodp. As an input statement, you specify the, datasets, in this case an accounting file and a set of statistic file datasets. As part of the SysIn, you specify the broker name and the store which the data is being streamed to and optionally you can specify a table name. If you do not specify a table name, all tables contained in the file or SafeConvert datasets are streamed. The batch size parameter is an optimization, so you will stream a thousand records at a time. So if you submit the job, you can look at the job output and you will see what messages are being displayed. In this case, the Sysprint is the DD you want to look at. You will be echoed with the tables and the input parameters and then you will see a list of records that are being streamed and inserted into the store and you see the different tables that are used. The naming convention we use is the same as for the performance database. For example, DGOAFGE for accounting file general or DGO SACC for statistics aggregated accounting. So let's see how this looks like in the Elasticsearch index management panel. So you see all the indexes that are being or data streams that are being created and filled and if you filter for let's say DGA file general, you will see the data stream. You can click on the data stream and see that this is data until November 1. So let's look at how the data is displayed in Grafana in this case. I built a sample dashboard that uses the Elasticsearch as the data source and I built some dashboarding for the data that we just streamed. This is accounting file data and you can for example draw a time series for DB2 class two, three breakdown for the respective workloads or for class three suspensions by specific main packs. You can also look at commits per main pack or top 10 main packs by commit and rollback and you can further drill down and go down to the main pack level and understand the class one, two, three distribution and zoom into class three by looking at the largest ledge time consumers for this specific time period. Finally, you can also draw the top five prim off over time. You can then use the Grafana capabilities to zoom into specific time periods to annotate specific time periods. If you want to detail on specific things, look at this and then pass this link to someone else or augment it in your existing problem determination. On the statistics file side, you can do similar things. In this case, I'm looking at DML activities or average row activities for a specific time period. So this is a stacked line chart. You see also aggregated accounting information over this time period. Lots of DDF workload here followed by Kicks workload. And then you can also look at not accounted times over time, drill into the respective time periods, look at the data more closely and you will see that based on the on the history, you will see the change the data changing accordingly. In summary, this capability opens up a whole new possibility to analyze the patch report data more graphically and you can integrate this into the same visualization platform, in this case Grafana, but with the power of a MegaMon data provider you can also stream this data to other platforms like Splunk for example. In this segment we will show the AI capabilities of MegaMon AI insights in the context of Db2. The data that you see is ingested by the aggregated accounting statistic attribute group coming from OmegaMon AI for DB two six one zero. You can select the data sharing group that you want to observe on the upper side of the panel and you can also detect the connection type there. Based on what you select, the respective anomalies are clustered into high, low, and medium anomalies and visualized on this first screen. You will also get further drill down into what we call overconsumption that the model has calculated based on the data it learned. In this case, for the observation period, the model detected an overconsumption of forty five thousand nine hundred twenty six CPU seconds. The main persona of this UI is the so called site reliability engineer that oversees the respective environment that is being monitored and can do some initial problem investigation and then hand over the issues to other more qualified persons called subject matter experts SMEs. So what will the SRE see on this dashboard? The primary metric that our model is trained on is the indivi two CPU time and what the side reliability engineer will see here is the evolution of the individ two CPU time and, the real values as depicted in blue. Again, this data is aggregated at the fifteen minute interval. What we see here is that starting at the August, there was some anomalies that have been detected and these have been annotated by the SRE using the Grafana notation capabilities. The real alert on the system when things started to really get bad was towards the August and we see this by this high spike of high anomalies which I have the red dots. The problem continued for a few days and then it was fixed by the customer. This data is based on real customer data so we ingested the aggregated accounting data from from history, meaning we ingested SMF 100 data, we trained the model, and we validated with the customer that the anomalies detected here are true anomalies. Let's look at how the SRE can use this dashboard to get an initial understanding of the problem. We looked at the primary KPI which is the indivi two CPU time and it shows unexpected spikes over the specific time period. The dashboard also has so called secondary metrics which are also coming from the aggregated accounting statistic data that contain things like transaction count, class three suspension times or class two not accounted times. These values can be useful to understand what the problem is and what the root cause is. You can further drill down into the secondary metrics, for example, by detailing on the suspension times and breaking them down in things like database IO, lock write IO, or other write IO, or into more latch log oriented values that this data provides. So let's look at one specific time period by drilling into this period and try to understand what happened. So what we see here is an increased transaction amount, but also increased suspension and not accounted time. So the issue might be even outside of DB2. So how would an SME look at this data? For this purpose, we used the PDB, the batch streaming capabilities to stream accounting safe data to an Elasticsearch and we can show this data in context to the existing aggregated accounting data. This is done by selecting the SME dashboard in this case it's called DB2 SME dashboard which will take us to this data and we can browse through that data for the specific time period which was around 02:00 that day. So by 02:15 since we aggregate the data we see the data in in the dashboard. So this is we start with more plan level information by class two three breakdowns. We can aggregate by by package and in this case we see that the Sysl h 200 package is the one that contributes the most to the anomaly. So we have a a good indication that this might be the pack the offending package, also confirmed by the large suspension times of that package. And if we break this down, we see, okay, there is a lot of lock and latch, but also AO activity going on for that specific package for the time period that we look at, which is the exceptional spike. We can then in theory drill down to the prim auth. We did not do that because we aggregated all to a single prim auth. However, by looking at the package level information, we see that there has been an extraordinary amount of rollbacks going on besides the commits which might be normal. And, at the end, we can also drill down to the package level information, and this all confirms that the Sysl h 200, which is a Java package, is causing this issue. This is confirmed by the customer and in fact it was a Java application that caused the issue at the end. So what are the potential options for an SRE once anomaly is detected? In this case, we set up our sample dashboard to use an integration with Teams so that we can create create an alert ticket. You can of course integrate with other tools like ServiceNow or NetCool to get your alerting into your company wide alerting systems. So let's create an alert and we'll see on the team's channel that the new message is being posted to this channel and from this message Fabien who is the person that should look closer at this, can link directly to the dashboard that is shown on the screen. Alternatively, you can also create an alert that informs an SME by pointing the SME to a specific dashboard as well. In this case we have set up another webhook that will send another Teams message to our OM AI demos channel. In this case, Matthias, that's me, is the SME that would have to look closer at this and we can again share the dashboard with the respective person. So in summary, I hope you have seen the capabilities of OmegaMon AI insights for the DB2 specific data are very useful to do proactive monitoring. In this specific case, the customer would have detected things that are not going well already at the August and could have saved all the trouble, that he incurred in the August December thus saving a lot of CPU cycles. And if you combine this capability with the capability to stream battery port or generated data into Elasticsearch and augment your problem determination by building additional dashboards to do root cause analysis. You have a pretty comprehensive solution to attack the text and the tech problems that you discover.