Before discussing how to recognize when your system has been hacked, it’s critical to first acknowledge that hacking an IBM i server is even possible. For many, the term "hack" conjures up images of hardened cyber-criminals covertly circumventing perimeter defenses; however, the reality is often distinctly easier and far less dramatic. In my opinion, hacking describes any form of unauthorized access to data and programs, including when that access originates from inside the firewall, not just the exploitation of an inherent flaw in the operating environment.
IBM i and its predecessors, i5/OS and OS/400, have each enjoyed a reputation as some of most secure operating systems available. This strategic benefit is touted by IBM and is widely acknowledged by administrators, auditors, and even some security experts. However, the integrity of any device, application, or operating system depends not only on the existence of robust security controls but also on the correct configuration of those controls. Risk is elevated whenever either of these elements is lacking.
Cybersecurity Depends on Correct IBM i Configuration
Integrity of the IBM i operating system is partly assured by several unique protection mechanisms. These include restrictions that ensure system objects are provided and modified only by IBM and that server functions are accessible only via IBM-supplied interfaces, such as commands and application programming interfaces (APIs). However, these mechanisms are not enforced unless the security level system value (QSECURITY) is set to level 40 or 50.
Many years ago, IBM established level 40 as the default security level but, according the State of IBM i Security Study, an alarming 35 percent of servers running IBM i fail to meet this requirement. This is largely a result of system values being restored from earlier generations of the server.
While the OS mechanisms are valuable, they are ineffective against unauthorized use of user objects, such as database files and application programs, which rely on the appropriate object permissions being established. User profile configuration is another critically important consideration. For example, a user with All Object (*ALLOBJ) special authority—and the study reported an average of 78 exists on each server—can override object security settings and issue commands that will disable the operating system and destroy the database!
Users Pose Real Risks to Data
Why would a legitimate user wish to wreak havoc by exploiting poor configuration? Well, it could be malicious or it could be accidental. During a Security Scan, I discovered a programmer had scheduled a destructive function using the IBM i job scheduler. The job was being rescheduled repeatedly to prevent execution and payload delivery at a future date—presumably until the programmer was no longer on the payroll to reschedule it. This blatant and extreme example was a deliberate act that threatened the integrity of the entire environment. There are many documented examples of legitimate users accessing data in ways that contravene business expectations, such as performing unauthorized downloads to a PC via non-traditional interfaces such as ODBC or FTP.
Although no less problematic, accidents are even more likely. I learned about "multi-member physical files" the hard way when, in the early days of my programming career, I innocently deleted a handful of my client's database tables that appeared devoid of data. Unfortunately, I was unaware that the data resided in secondary members! As foolish as that act now seems, most people in IT can recall an instant when they wished they knew then what they know now or when the Enter key was less obedient. The risk of unfortunate events occurring can be reduced through careful configuration of object-level authorities as well as appropriately assigned user privileges. Either way, a forensics trail is required by most regulatory mandates.
The correct configuration of IBM i and user applications is beyond the scope of this article; however, there are many sources of education, including security sessions at COMMON and on-demand webinars. I always counsel organizations to begin a security project with an assessment to document the starting point—which helps to measure subsequent progress—and prioritize mitigation steps for maximum risk reduction.
Use the QAUDJRN
Hacking of IBM i can originate from a compromised profile or from a legitimate user, and it can be either a malicious act or the result of an accidental misstep. Fortunately, integrated user and event auditing is one of the many value statements of IBM i and should always be configured to track when something unexpected happens. IBM i logs events to a special journal (*JRN) object named QAUDJRN.
The benefit of a journal is that it's tamper-resistant, guarding the receiver entries so they cannot be altered or removed, a unique benefit of IBM i and a regulatory must-have. As with database journals, the data is funneled into a journal receiver where it is stored. The journal receiver can reside in any library, and I recommend altering the QGPL default to a library created specifically to house these objects. Once a receiver is full—a threshold defined via the journal's property settings—the system will automatically create another in a named sequence. Two system values, audit control (QAUDCTL) and audit level (QAUDLVL), oversee the main auditing functions and define what gets audited at a system level. Objects and user profiles also possess auditing properties, enabling closer observation.
Rely on IBM i’s Intrusion Detection System
Enterprise-class servers, such as those running IBM i, have higher likelihood of suffering exploitation at the hands of the trusted user rather than the anonymous attacker. However, operating within the confines of a corporate firewall does not eliminate the risk of being hacked from outside, or even the possibility of the host being used as the source of an extrusion attack. Firewalls, like any software, are susceptible to misconfiguration and can be compromised easily using network credentials that were cracked, stolen, or socially engineered.
This possibility reinforces the benefits of utilizing a defense-in-depth model, whereby security is applied in multiple, independent layers. Fortunately, and as a further statement about the security capabilities inherent to the platform, IBM i has an integrated Intrusion Detection System (IDS). In my opinion, the IDS moniker undersells the total value of this feature as the detection is complemented by prevention capabilities, as well as the ability to monitor for both intrusion and extrusion events. Available at no additional charge, this component of the operating system is designed to detect—and even react to—anomalous TCP/IP activities, such as port scanning or distributed denial-of-service (DDoS) attacks like a "Syn Flood" or "Smurf" attack.
Activate this feature using the GUI interface provided in IBM Navigator for i, and then deploy a default policy, or define custom policies, to oversee the detection and response to attacks, scans, and traffic irregularities. Starting in IBM i 6.1, reliance on the Quality of Service (QoS) function was removed, making the IDS even easier to configure. Events that contravene policy are logged in the audit journal like any other type of suspicious or unauthorized behavior. They can also be brought to the attention of interested parties using built-in email notifications. Of course, success depends on the function being activated and properly configured.
Plan for the Worst-Case Security Scenario
After auditing is initiated, it's important to establish a review process, at least for the most critical entries. While this sounds obvious, many organizations collect audit data without any form of security- or compliance-motivated review. This often includes those using high-availability (HA) solutions that utilize auditing to detect system changes to be replicated to a backup system. Commercial HA applications typically purge the audit data once it's no longer required for that purpose, often within 48 hours of collection. This retention window is woefully inadequate for security forensics and does not meet any modern regulatory statute.
Other organizations have no intention of reviewing the data and therefore believe there's no business benefit from collecting it. In that scenario, I introduce the analogy of closed-circuit TV cameras in a bank: while the live video may not be monitored constantly, recording the feed remains imperative so that it may be "rewound" if an incident does occur. Even without ongoing review, IBM i audit data should still be captured so that it can be analyzed retroactively if any form of red flag is observed, akin to an airplane "black box" recorder.
Reviewing thousands or even millions of individual audit entries manually is an arduous and error-prone task. The operating system contains the Copy Audit Journal (CPYAUDJRN) command to extract entries from the receivers into database files. Selection is based on a two-character audit-record entry type, meaning that the challenge of dealing with a single, huge repository (possibly containing more than a hundred different entry types) is supplanted by the challenge of dealing with more than one hundred individual database files, each containing a subset of the full record set. It's certainly no wonder that the requirement to review the audit information is typically met with disdain!
Consider Tools that Automate Security Tasks
Commercial-grade forensics tools are designed to minimize the human effort associated with gathering, filtering, and sorting the data. Sending log entries to a Security Event and Information Manager (SEIM) in real time is also becoming more commonplace thanks to the growing popularity of these solutions; however, there is risk of security events becoming obfuscated from the server's own administrators. Some audit events are more critical than others, so focusing attention on reviewing a subset of the events can make a review more manageable.
In the absence of any automated tools, I still recommend collecting all the events as commercial solutions can always be purchased and utilized to search the log if a breach is discovered. Budgets tend to become unlimited in this scenario! At this point, I feel I need to remind readers that certain entries, such as the feared "AF" authority failure event, will be logged only if the security configuration restricts certain events. If, for example, every profile has all object (*ALLOBJ) privileges, or if objects are all set with a *PUBLIC authority of *ALL, then an authority failure event can technically never occur.
Common sense dictates how critical it is that unauthorized activity on the network or the server be logged. Less frequently acknowledged—but equally important—is to proactively establish an incident response plan. Note that I said "proactive," which suggests it should not be created in a state of panic when an incident has already occurred! Planning for worst-case scenarios before they happen helps to ensure that any reaction is officially sanctioned, timely, and decisive. But remember: a plan is beneficial only if staff are aware of it and educated on how to use it, and are taught how to manage a breach scenario.
Summary
In summary, protection of an IBM i environment relies heavily upon the integrated security controls, the comprehensive configuration of those controls, and auditing the actions of users. In the absence of auditing, the question of when you will know of a breach translates more accurately to if you will ever know of a breach.
We have to stop ignoring that a server running IBM i most certainly can be hacked and, while it may not come from a blatant flaw in the operating system, common configuration weaknesses such as a sub-par password policy with passwords that match the user name, uncontrolled TCP services (e.g., FTP, DDM, and ODBC), and reliance on the IBM-supplied default for object authority leave data vulnerable to unauthorized access.
