Wide-ranging utilization, from cloud infrastructure to RPAs and networks
Escaping vendor lock-in, promoting in-house monitoring systems
Adopt scalable monitoring infrastructure using Zabbix, advance internal development. Expand applications, support further speed and flexibility
Functionality to monitor AWS Auto Scaling feature
In order to switch from ordering from external partners to in-house solutions, create a robust organization capable of in-house development
Significantly cut running costs by switching monitoring from a commercial product to Zabbix
This migration not only solved issues with monitoring, statistics, and analysis, but also added value by allowing for faster response to faults
Mitsubishi UFJ Information Technology, which provides IT support for the financial services at each company in the Mitsubishi UFJ Financial Group (MUFG), in conjunction with its migration to cloud computing, selected Zabbix as a scalable monitoring platform and promoted internal development. Applications have further expanded to RPA, ATM, and network monitoring.
As the word “FinTech” implies, nowadays digital technologies and financial services are inextricably linked. At Mitsubishi UFJ Information Technology (MUIT), they provide IT support for the financial services of Mitsubishi UFJ Bank and the other companies of the Mitsubishi UFJ Financial Group (MUFG).
The first things needed for banking and other financial service systems are stability and security, but in recent years speed and flexibility are also required. Being able to meet market needs by swiftly providing new financial services that combine smartphones and Web services is indispensable from a strategic point of view.
To meet these demands, MUIT has been gradually undertaking reforms in terms of both system platforms and its own organization, with a particular focus on internal development. “Constantly ordering from external partners leaves us with less maneuverability, and costs remain high. To get away from this, we are aggressively utilizing open-source software (OSS) while expanding activities that aim to create a robust organization capable of doing our own development.” (Mr. Motohiro Katsuno, Senior Professional, Business Platforms No. 4 Department, Business Platforms Group, Mitsubishi UFJ Information Technology, Ltd.)
As part of these efforts, they have actively pursued a migration from on-site to cloud computing, starting with Amazon Web Services (AWS). The superiority of the cloud, with its excellent scalability, is clear when it comes to swiftly developing new services to meet the needs of the business side as well as clients. Determined to lead the financial industry, the MUFG Group has also been working on the migration of its infrastructure platforms, flying the “Cloud First” banner.
They ran into challenges, however, with how to implement monitoring. “AWS has Auto Scaling functions that can scale resources to meet the load. We needed the functionality to monitor these, but that was sometimes difficult with the monitoring product we were using at the time.” (Kazuma Sugiura, Professional, IT Control Service Department, Group Common Platforms Group, Mitsubishi UFJ Information Technology Ltd.) Simple instance monitoring can be implemented with tools like the monitoring tools provided by AWS, but when it came to monitoring that was a match for Auto Scaling and the other unique use cases of AWS, it was difficult for them to find the right fit.
It was at that point where, among all the OSS and other options that they explored, Zabbix emerged as the leading candidate. Compared to the commercial monitoring tools they were using in parallel, Zabbix was flexible, there was no vendor lock-in, and it had the unique advantage of OSS— a wide variety of information put on the internet by volunteers, in addition to the official documentation. For those reasons, they decided to adopt it, first starting monitoring operations with Zabbix with the help of their partners, then gradually switching over to doing things in-house. The wealth of information available really came in handy during that transition. “In just the three years I’ve been handling it, the scale of monitoring has increased four-fold, not only on AWS but also the on-premises environment. This required some tuning of things like value cache size and InnoDB’s buffer pool size, but the variety of information available in the official documentation and Zabbix user group forums allowed us to accomplish this ourselves instead of relying on vendors” (Mr. Sugiura).
Since this is a large-scale system that processes thousands to tens of thousands of triggers per day, they’re also working to improve efficiency. They utilized the Zabbix API to link Zabbix to their workflow, and automated the monitoring configuration settings by having content entered in the API format by each person-in-charge reflected via that workflow. They don’t need to have each individual person-in-charge touch the Zabbix GUI.
As a result, “Although the scale of monitoring has increased four-fold and the number of monitored servers has doubled, we’re able to operate with essentially no change in maintenance personnel” (Mr. Sugiura). Although automation mechanisms have also been created in other tools, with the Zabbix API you can check for differences with existing items and triggers and ensure consistency while updating them, leading to more stable operation. Another advantage of Zabbix is that it allows the management side to centrally manage monitoring definitions, rather than forcing this responsibility on the subject. Mr. Sugiura aggregates the results of monitoring by Zabbix and those from other monitoring tools and manages them centrally, and in the process, he uses Zabbix macros, which allow him to configure post-alert detection operations and processes with a high degree of freedom. “At this point in time, we’re utilizing only the usual monitoring functions, but since macros, etc., are highly scalable, I think there are a lot of areas where we can make improvements going forward” (Mr. Sugiura).
Mr. Katsuno, who was tasked with detailed operational monitoring of each business system and application while adapting to their specific needs, was faced with another challenge: ensuring stable operation of the ever-increasing number of RPAs.
Bank and financial service operations include document review and a variety of other workflows, many of which need a human touchpoint. At MUFG Group, however, in parallel with the migration to AWS, since around 2014 they’ve been focusing on RPAs and promoting the automation and optimization of operations.
The problem was how to achieve the stable operation of RPAs. “Unlike our key systems, which run on servers, RPAs don’t operate on the information systems side—they are developed by end users and run on fat clients. When there weren’t enough resources or if a fault occurred, the client would ask us what was happening, but it was difficult to understand what was going on” (Mr. Katsuno). Furthermore, as the role of RPAs in operations expanded, external audits began to require monitoring and assurance of stable operation.
However, as opposed to new services and systems that directly benefit customers, it was difficult to put a lot of funds toward RPA monitoring. At that point, “aiming for something that we could start small with, we evaluated a lot of different products, and selected Zabbix” (Mr. Katsuno). They added a series of RPAs to Windows and Linux and ran them on Oracle, SQL Server, and a variety of other platforms. The breadth of platforms that Zabbix runs on, and, as Mr. Sugiura also touched on, the amount of related information and documentation available, were reassuring.
At present, using Zabbix as a separate system from their platform's monitoring system, they have created a system that gives them a bird's-eye view of the overall operating status of the approximately 450 fat clients and the RPAs running on them. The management interface has been modified to allow not only the information system side but also the users of RPAs to use it like an application to directly check what is currently happening.
RPA utilization scenarios at MUFG continue to expand, and it’s not uncommon to suddenly realize that there are now 10 or 20 more. For these kinds of “shadow RPAs” as well, they use Ansible for automated discovery, they’ve developed a mechanism for inputting monitoring settings, and they’re able to monitor things like, “Is the robot running correctly?” and “Are there any resource shortages due to load concentration?” They have also developed a mechanism for automatically removing monitoring settings from RPAs that they can determine haven’t been used for a set period of time.
Once they created those mechanisms, they started getting a variety of other requests from the field, things like, “I want to know how many users are logging in,” and “I want to monitor whether there are changes in the installed versions of Excel and PowerPoint.” There were a variety of needs, and being able to flexibly handle those also gave them a sense of Zabbix’s merit. “In monitoring development as well, we start with the range of what we can do and proceed in an Agile-like way while taking into account the requests we receive” (Mr. Katsuno).
In addition to monitoring infrastructure platforms and RPAs, the applications for Zabbix in MUFG Group IT systems have been expanding more and more—it has now also been adopted for use in monitoring a portion of the ATMs installed at their branches throughout Japan, for example. Recently, the monitoring platform for the network connecting sales offices, unmanned branches, overseas sites, and data centers has also just been migrated to Zabbix. This is a large-scale private network with more than 114,000 IP addresses, and being able to reduce its running costs by tens of millions of JPY by switching over monitoring from a commercial product to Zabbix played a big part in that decision.
While there are also a lot of benefits to using a commercial product, vendor lock-in is a major constraint. It caused the work of the MUIT team to be weighted toward planning and project management, and they faced an issue in that the number of people who could actually roll up their sleeves and do the work was declining. However, through a series of initiatives— migrating to AWS, adopting the Zabbix OSS, and bringing development and operations in-house—they’re gradually changing this. MUIT will continue to support next-gen financial services through initiatives to create flexible IT platforms, systems, and applications that meet the needs of a continually shifting market, optimizing running costs along the way.
Mitsubishi UFJ Information Technology is a team of financial and IT professionals that provides IT support for the financial services of Mitsubishi UFJ Bank and the other companies of the Mitsubishi UFJ Financial Group (hereinafter, “MUFG”). They use cutting-edge technology to provide peace of mind, stability, and security for the financial systems that are such critical social infrastructure in the company’s four domains: the Headquarters Sector; the Banking Business Sector, which leads system development for Mitsubishi UFJ Bank; the Group Business Sector, which handles system infrastructure and IT services for each MUFG company, including the Bank; and the Chance Business Sector, which provides key system packages to regional banks.
Fill out this Questionnaire Oder contact our Marketing Team für weitere Unterstützung.
Zabbix ist Open-Source-Software und damit kostenlos. Ganz ohne Einschränkungen oder versteckte Kosten.