Cyber threats are becoming more ferocious than ever. What are IBM i pros doing in 2020 to keep up?
Watch this on-demand webinar for new data on the IBM i security controls deployed at organizations around the globe.
Security experts Robin Tatam and Sandi Moore will reveal intelligence gathered from hundreds of IBM i security audits conducted over the past year. You’ll see impartial data around seven configuration categories:
• Network-initiated commands & data access
• Server-level security controls
• Profile and password settings
• Administrative capabilities
• Public accessibility to corporate data
• System event auditing
• Antivirus protection
You’ll get expert analysis of current IBM i security trends, along with predictions for the future of the OS and practical tips for improving IBM i security at your organization.
Introduction to The 2020 State of IBM i Security: Threats, Trends, and Predictions
Hi everybody. Welcome to today's session. We're going to be talking about the new 2020 State of IBM i Security Study. You are getting a view of this at almost the point where it hits the website. The ink is still wet. We're excited to share this with you here today. I'm going to be assisted along the way by Sandi. You've heard her voice already, but Sandi, why don't you say hi.
Hi everybody. Sandi Moore here, excited to share this information. Robin's going to have some interesting things for you to ponder.
Sandi is going to be kind of the MC of the event, so if you have questions, feel free to fire those into the question panel, and she can monitor that if you have any challenges. If they are audio challenges, obviously there's some strains on bandwidth in the world at the moment, so sometimes disconnecting and reconnecting can help. If I have an issue, Sandi will have to take over, and we'll try to keep that to a minimum. But we are recording, so again if there's any challenges hopefully that will be addressed there.
My name is Robin Tatam. I'm the Director of Security Technologies over here at HelpSystems, which is just a fancy way of saying that I evangelize IBM i security. I have the pleasure and honor of speaking at events, as does Sandi, around the world. We do a security tour process whereby Sandi and I travel out and speak about IBM i security, which is a really fun thing to do. I'm also one of the three IBM Champions here at HelpSystems, we have two in security and one in overall automation. But, the main role that I have is really to work with customers and do security scans and help with the education.
Purpose of the IBM i Security Study
Let’s talk for a moment here Sandi because I like people to understand how and why we do this study, and I think it just gives some background on the purpose here.
There’s really three main bullets that I speak to when I talk about the purpose of the study and that really starts with the reason that we get in here which is to:
- Help IT management to be able to understand what, if any, IBM i security exposures exists.
- There's definitely a reputation that the platform is secure, we know that has some nuances. There is some I guess justification of disclaimers associated with that, and we want IT management to be cognizant of that, so that they can make good decisions.
- We want to focus on the top areas of concern.
- We want to zero in on the things that are most impactful. We want to make sure that we are spending the limited time and resources that we have in a day to be able to make meaningful changes.
- Then as we develop the plans to address and mitigate these items, we want to make sure we're focusing on the high-risk items first.
- That is the most important thing
I liken it to we're doing some work on the deck of the ship, but we don't realize there's a hole in the side, and we want to make sure that we recognize that we're taking on water and that we're working on that immediate need before we worry about some of the other things.
You know Robin too is that this actually can help once you've identified all these areas is to start building budget for it to because knowing about it, then you can actually start planning for it, like you said.
For sure. Cost justification is always an important consideration in building a business case and by using the study, we are hopeful that that is a beneficial process and that it helps give some reinforcements behind what it is people are asking to do.
The actual data itself is pulled in through the process that we do that we refer to as Security Scan. We do hundreds of these. I know Sandi you probably do as many of these as I do, if maybe not more at this point, throughout the year, and it's a fun process. I've already done two of them just today, and I know you were on one earlier.
It's a process whereby we are looking for common areas of vulnerability within the IBM i environment. The customer that runs that scan has full visibility to what is revealed. They are optionally then invited to share that data with us for use in an anonymous aggregated form within the study. They don't have to do that, although we appreciate of course when people are willing to share, but even if they don't, the data is still available to them. So, that's the process, whereby we're looking at live active systems. The first study that we did all the way back in 2004, actually before even my tenure with HelpSystems, but I was actually involved with it previous to my work here at HelpSystems over the last 11 years because I used to run these scans back in a prior life. Now that we've done it for so long, we've got thousands of servers that we can build this insight over, giving us great credibility as far as baselines, the technologies, and of course the gaps that exist in people's security. One thing, I always try to be transparent about the companies that are running the scan are self-selected. It's people such as yourself that said, hey we want to do this, and one could argue that that makes people perhaps less security aware because they need a scan to tell them what their security is, or it could make them more security aware because they've asked somebody to actually step in and look at their system. There's no right or wrong answer there. I'll let you make up your mind.
But, certainly I will offer that even the scans that we're not able to include really do fall in line with a lot of the things that we see, so it's not like we cherry pick the worst ones to make the study look worse or likewise look better than it actually is. This is extremely representative based on the real-world examples that we do each and every day.
IBM i Security Study: Settings
I have noted here that since we've been doing the Marketplace Survey here at HelpSystems, which is a survey-based report and some of the other outside studies that are survey-based, we do notice some interesting disparities between what people perceive they have in place or the controls that they are reliant on and the reality of what we see actually on the system. And so we like to remind that the Security Study is pulling data specifically from actual settings versus somebody's awareness or interpretation of whether those settings exist or not.
There are seven areas that we look at ranging from privileged accounts to PC interfaces to password policy to system values, of course and even antivirus control. So, there is a number of different categories that we look at, and that's where we're going to get into today.
First thing I'd like to do is I'm going to open up a very quick poll because I'm interested. We put a lot of time, effort, and money in a way into providing this study to the IBM i Community. I'm interested in whether reading this perhaps in the past has influenced any interest that you have in security for IBM i. It's nice to know that that you've benefited from it, but likewise if this is new to you and you haven't yet taken a look at it or made changes based on the results, then this will be a good opportunity for people to get that experience. It looks like about 80 percent of you say yes, and twenty percent say no, so hopefully you've just not seen it before versus not thinking it's valuable. I know marketing does a fantastic job of taking my chicken scratchings and putting them into something more meaningful, and we're excited for you to have that, so thank you appreciate that feedback.
About the State of IBM i Security Study
All right, Sandi, so the 2020 State of Security Study, let's talk about what went into it. 249 companies this year submitted data on a total of 255 Power Systems Servers. Some companies did run this across multiple partitions or multiple physical frames, but in most instances, we saw that there was a pretty close companionship between the number of companies and the number of power servers. This number goes up and down each year depending on how focused people are, whether they're feeling generous and want to share the data, but this is this pretty average. It's up from last year, I believe and definitely a good representation of the community.
When we look at, you know, the sources of the scans, we certainly have an emphasis on North America, over eighty percent of the scans that we receive data on come from North America. Now, there are different regulations in different parts of the world that perhaps prohibit sharing of this type of information, so it's not necessarily indicative that we're running more scans, just that there is more submission of the data. EMEA is behind on thirteen percent, Latin America at six percent. We certainly encourage people in these geographies to take advantage of the scan and hopefully that way we can level this out. But definitely we appreciate those of you in North America that have shared this with us.
Some other statistics that we pull from the data, these are self-reported, so customers have given us an indication that most of the IBM i Systems for 2020 came from the manufacturing sector. We also saw a lot in government and quite a few in transportation, as well. The bottom line is we have a pretty good cross reference of IBM i servers in different industry verticals.
We also see a lot of different sizes of organizations. There were more large companies last year then small, and we've seen that level set a little bit this year. We definitely still have some large organizations submit data, people up to 5,000 profiles accommodating or accounting for twenty-six percent of the data. But, when we get down to slightly smaller systems, still a good portion, almost 30% that have between 250 and 1,000 employees contributing data, as well. So good representation of the different bread and butter type servers for the IBM i community.
Sandi, do you realize we looked at 274,000 profiles as part of the 2020 study.
That’s a lot!
Yes, it is indeed, and we had to analyze all of them. That broke down to an average of a little over 1,000 profiles per server. So, if you're looking for an idea of how the average compares to your internal shop, then that's the comparison that you can use. From a library perspective, over 141,000 libraries were analyzed to determine the public permissions on those libraries and the default object permissions that come from the library, that breaks down to an average of 557 libraries on each and every system, which is a pretty impressive number.
The operating system isn't something that a lot of people think of in terms of security, but when it comes to regulations, as well as best practices, we want to make sure we're staying on supported versions of the operating system. We want to make sure that the patches are coming, that they're being installed in a timely manner.
The good news is the lion's share of you now are on version 7.3. That is a supported operating system with no end of life, as of yet. A few of you have crept onto the latest and greatest 7.4. There's still a good portion of you, about 30%, that are on 7.2. 7.2 is still supported but has an end of life, although April 2021 may be extended, I don't know based on the fact that obviously there's a lot of projects and initiatives that are being delayed at the moment. If you are part of the 6.1 or 7.1 crowd, the red or the orange slices, you are already unsupported. You should be making plans to move to a supported OS if you can. If it's a restriction based on the hardware you're on, then obviously staying up on the latest hardware can have a lot of cost benefits, as well.
The bottom line here, Sandi, is we just want people to make sure that they have somebody they can call if there is a problem, right? And they are getting those patches installed in a timely fashion. Usually the impacts we run on, typically once a quarter, if there are critical patches, a lot of the regulations talk about having them installed within 30 days or less. So, you want to look to any regulatory bodies if there are any. If it's just best practices, that would be a good starting point for you.
Basic System Security: QSECURITY Levels
Getting into the security controls. The security enforcement level is something that is controlled by a system value. We lovingly refer often to as QSECURITY, this controls the overall health of the system from a security perspective, and the good news here, a lot of you are firing on all cylinders. You are at or above the minimum recommended level.
75% of the systems are accomplishing that goal, which is a very positive note, and it has increased gradually over the years, and that's fantastic. However, we have to acknowledge that the security enforcement level is influenced by a lot of other controls. I had a customer not too long ago that was a security level 50. They were very excited about seeing that green check mark when they ran the scan, but then we discovered 97% of their profiles had All-Object Special Authority (*ALLOBJ), and I had to break the news to them that they were equivalent of security level 20, a full three levels down. So, there are other influencing factors. A lot of people arguably put a little too much emphasis on QSECURITY, but it does have a role to play.
As I said, this has improved over the years. We still have a number of people down all the way at level 20, even at level 30, which is has some well-documented and well-known vulnerabilities associated with it, and so being at level 40 is an objective that you should have on your radar. The good news is you can emulate level 40 when you're still at level 30, so you can do some careful planning and do that migration at a point where you're already comfortable that you're not going to run into issues. Don't do what some customers do, which is change the value, reboot the box, and then scramble when your application doesn't function. It's rare that we see that, but when it does happen, it's very disruptive, and so doing the migration carefully is something that will eliminate that risk.
Although there's not too many of them flying at the moment, systems like this are obviously monitored on a real-time, ongoing basis, and the intent with sensors and feedback from an airliners computer systems enable the flight crew to make adjustments and do so in a real-time format so that they can maintain one of the safest modes of transportation, and that's very beneficial. In the case where there is a catastrophe, the flight data recorders obviously are available to the safety boards to be able to determine what went wrong, and there's just a fantastic approach there where that data is shared amongst the different airlines and ultimately is used to improve and further increase the safety of air travel.
Now, I mentioned that because IBM i does have a flight data recorder, and the good news is 80% of you have created it, which is a step you have to take proactively, so there is some monitoring happening here.
That also means there is 20% of you that I'm talking to today that actually have zero visibility to anything happening on your system, whether it is an authority failure, a system value change, or the creation of a new security officer profile, and we would love to see that down at 0%. We want everybody to have at least basic visibility into that log environment. Now, even if you're not checking it in real time and able to make adjustments as you go, having the ability to refer back to it under a catastrophe situation is typically a regulatory mandate, and common sense, that says we're not going to be able to make corrections if we don't know what caused the problem in the first place.
If you're one of the 20%, then we have to question, would you know if there was a brute force attack? Now, even if you are auditing, and this particular customer was because I did this scan myself, there can be things that kind of fly under the radar. Now this metric right here, this 99 million, is the number of failed connection attempts against one individual profile. So this was a script or a program that had lost control it was trying to sign in to the server, and while the number is certainly quite startling, the concern that I had was not so much the large number, it was the fact that they had in fact logged 99 million entries into their audit journal, and they hadn’t seen them. They weren't looking at it. In this instance, this is a good example where there was a red flag. It was a pretty significant red flag, and it was waving hard but, because they didn't have the necessary procedures and processes in place, they had completely missed it, and that’s startling.
Half of the servers that we looked at did have a profile that had experienced at least one hundred failed connection attempts, which when we get to one hundred, even arguably when we get to 30 40 or 50, it's unlikely that that's a person sitting at a keyboard. It's much more likely to be either an application that has broken somehow or a brute force script of some kind, both of which are scenarios that you want to get involved with as quickly as possible.
You can't stop this from happening, it can spiral quickly, but you can react to it and that's the most important thing.
Now this is what those entries look like. I'm picking on one particular type of entry just because of the 99 million because I think it makes people sit up and take note. An invalid password entry or an invalid user ID is something that the system can easily log. You need to be looking for it, although the pain point tends to come when you get into that log looking for those types of things because we discover that there is just an ungodly amount of raw data in that log, and if we're trying to process that in any type of manual fashion, then the reality is we're going to be frustrated at the least, going completely crazy more likely, and the reality is we're either going to miss things and or we're going to avoid this task like the plague and really just lose that visibility. We recognize that the system can be pretty noisy. Important events often get lost in that proverbial needle in the haystack and become very hard to find. Although, one could argue 99 million entries there is probably a pretty big needle, but for most of us where we're looking at any individual events or two or three occurrences of an event that becomes very difficult to find.
The key here really is the implementation of automation can be very beneficial and makes it a little bit more manageable. There's lots of things you can do here, whether it is to extract data to a file and then run a query or use a commercial tool to report over the entries. Many of you I know are now using SIEM solutions that are processing logs from all of your platforms in the data center and automating that process to enable IBM i to work there can be very beneficial, as well.
The Importance of Perimeter Access Control
Sandi, I don't know the last time you saw a dumb terminal in a twinax cord, but for me, it's been awhile. Back in those days, security was easy, so was network troubleshooting. I remember unscrewing the twinax silver cap, straightening the two pins and plugging it back together and problem solved. It's not a simple world like that anymore. In many ways that's a good thing, but when it comes to security, we also have to acknowledge that there was a new set of challenges that come with it.
For many of us, we still leverage the same controls that we did even when we were on dumb terminals. We’re using menus and application security predominantly because they already exist. They were there back in AS/400 days, and we've migrated those same applications forward, and they were created even back then because they were easy to build, and they were very effective.
There are some assumptions that come especially with menu security that we need to be cognizant of. One is that access comes only through the application that we have deployed. We don't anticipate users stepping outside of that. We don't expect or can't grant users direct command line permission, so we expect that a user can only use the menuing function. And, we can't give them access to powerful database tools like DBU and in even languages like SQL because all of a sudden we are now able to circumvent the menu.
These three assumptions are definitely living on shaky ground at this point, perhaps even worse than that, but we know that they still have a value and a part to play. The messages we know don't have one hundred percent effectiveness.
When we have traditional security controls like this, we tend to see a lot of very common mistakes. In fact, just this morning, I was talking to a customer that indicated that their application, a commercial application, was built on the premise of the object owner profile being a group profile for all of their users, which unfortunately means that people have complete access to all of the objects in or outside the application controls. Users do have command line privileges, we’ll talk about that, and we know that the *PUBLIC has been given a lot of access to the database, as well, and we'll talk about those things.
The bottom line is data is accessible. That's a good thing if you are a legitimate user and consumer of that data because you're making good business decisions, you are driving the direction of your organization, and your responding to the user needs. However, it also has a pain point that comes with it, and we have to take this responsibility of sharing data very seriously.
What TCP/IP Services Are Running?
There's a number of different interfaces that allow data or commands to be run or extracted from the system. Here's a couple of examples. The graphs here are telling us that in the case of FTP, that service is typically automatically up and running on virtually every system. This is a common interface. It's known by everybody, good and bad, when they try to access a system and FTP services tend to be available.
The REXEC side is for remote command execution. It's an interface that allows us to issue server commands against the IBM i in many instances without any indication of whether the user has command line permission, as controlled by the limit capability setting. The good news is most of you don't have this up and running, but almost 20% of you do, and if you do start it, we have to acknowledge that this does provide a channel into the system to run commands.
I don't know about you Sandi, but I hear from customers that they don't use these services or they shut them down. That's desirable in some instances. What we're doing when we shut everything down is trying to get the system back to a traditional dumb terminal type environment, and it's unfortunate because those services do have a lot of intrinsic value.
The other thing worth noting is not all the services can be shut down. You can turn off FTP. You can tell it not to auto start after an IPL, but database services are required for the system to function, and they're not something that you have control over.
Shutting down things that are not used? Fantastic. The expectation of shutting everything down? Not realistic.
The Greatest Impact on IBM i Security: Exit Programs
Now back in the mid-90s when IBM added all of these different access methods, they also opened up a new function, a new capability, by allowing us to leverage what we call exit points that will invoke an exit program anytime inappropriate transaction has come in. Think of this as a user-defined function, where we're given a capability that as programmers we can write extensions to the operating system, and it's a fantastic capability. The actual exit program itself is going to function based on what the programmer puts within it for code. That could technically be an unauthorized act.
We have to be careful, but when we are talking of these in a legitimate form, we have an expectation that an exit program is going to do two things for us. The first, arguably most important aspect, is to give visibility to these connections. IBM i does a very poor job of auditing a PC connection. If a user has FTP access, if they have the ability to access a particular file in the database, they can download that file to the PC, and there's no log of that having taken place, which is an immediate audit finding. Based on that audit information, we may want to extend it to be able to provide some control, as well, and an exit program is able to do that. It helps augment when a system has poor object level security, and it also provides flexibility when there is object level security.
What do I mean by inflexibility? Well, in IBM i there is no expectation or no difference between the different access methods. If you open the door to the green screen, then you've also opened the door to FTP and ODBC. If FTP is not something you want to have happen and you close the door, you've also inadvertently closed the door to the other valid interfaces. So, it's an all-or-nothing type scenario, and that can be a frustration when people want to have different levels of access based on the different doors that are being used.
An extremely beneficial part of the exit program is that it has the ability to return a pass-fail indicator back to the operating system, back to that exit point. And at that point, the operating system will use that pass/fail indicator to determine whether the operating system should be told or instructed to go ahead and try to fulfill the request or to tell the user that their request is being denied. Incredibly powerful, largely because this evaluation happens in advance of any OS security evaluation. So if you are one of those shops that has a lot of privileged accounts, and you’re daunted at the task of reducing that number down, then you can use an exit program to decide that a user shouldn't be pulling data from the system without actually doing a cleanup on the all object.
So as important as this is and arguably to me, Sandi I don't know if you agree, but this is arguably one of the most important and relevant controls within IBM i security. The question is how many of you are using it?
Not enough is right. There are about 30 different doors in and out of the system, granted not all of them are as important as others. There's about a dozen that allow data to flow or that allow commands to be run. What we tend to see is when we set the bar very low, just one or two exit programs perhaps, we see about one-third of you that pass that test. That means two-thirds of you have no exit programs whatsoever. When we take the bar up to the upper level, which we would prefer to be at, that says we're monitoring all of these doors instead of being extremely selective, then we see only 8% of systems passing that test.
This is one of the easiest fixes that can be implemented. It's been around for about 25 years, so it's very proven at this point, and it can offset the risk from many other configuration settings, so it gives you a lot of bang for the buck.
When I talked about this at the Security Tour events, I know that many of you indicate you have not heard of this capability or you have concerns over things like ODBC, but you weren't really sure how that was influenced by the controls that you have implemented, and hopefully this clears that up a little bit.
This is the make-or-break to your system. If there is one thing you fix, this would be what it should be in my opinion.
Now when we see one or more exit programs again, what we're talking about is usually one or two that's usually indicative of somebody who's written their own exit programs, and the functionality tends to be very basic there.
You can write your own exit programs, but we tend to see very low amounts of coverage across the different doors, and we see the functionality being very limited. You also want to be careful with performance implications over things like ODBC, which can be very high volume traffic areas. The comprehensive coverage is usually indicative of a commercial application that is doing that function on your behalf. Without it, or if it's not well configured, you may find yourself in the unenviable situation where your data is leaving the system, but this is not just about data. Data doesn't just flow downhill. It can also flow uphill, and the ability for a user to run commands through these interfaces brings this into a whole new ballgame about availability of the server and applications to the users. It's not just about your data popping up on the dark web. So look at this as total business resilience, not just about data resistance.
Tips for IBM i Special Authorities
So, how about a powerful user. Sandi, I know you and I have talked about this, we've seen references to powerful users and regulations, but who really knows what a powerful user is.
I do too, so I'm going to share that with everybody because, and arguably this is my interpretation of a powerful user because there is nothing cast in stone anywhere that says regardless of your technology that this is in essence how you measure it. So, because of that challenge and regulation saying you can have a certain number of powerful users or powerful users should be controlled this way or that way we have to start in many instances by giving a definition of what that entails, so that we know which users belong to this group.
For me, I have three primary considerations and it's anybody who has one or more of these:
- It could be somebody who has one or more of the eight special authorities.
- It could be somebody who has private or explicit permission to the database or the application.
- It could be anybody working on a server that has very permissive Public Authority.
We know from past security studies that that tends to be most of them. Now, I'll talk about what Public is as a clarifier in just one second, but what we tend to want to see or we expect to see as part of a powerful user is anybody with those three attributes, they also have either a way to execute commands on the system or they have a tool that gives them access to the database. Either one of those elevates these permissions up to the point where they could be deemed an extremely powerful user.
Now, the Special Authorities side of this is as I mentioned a capability that allows a user to have one or more of eight special permissions. The *ALLOBJ is the big guy. This is unrestricted, unfettered access to every object on the system. Not only the objects they create but all the objects created by everybody else, as well. That could be the application objects. It could be the objects that comprise the operating system. In fact, with *ALLOBJ, you could basically delete everything on your box. This is the one you want to control the most tightly.
Security Administrators (*SECADM) can create and manage the user profiles, again a very powerful privilege often seen combined with *ALLOBJ.
*IOSYSCFG allows a user to change the configuration of the communications. If you've shut down those TCP interfaces because you're not using them, an *IOSYSCFG user can fire them back up again. So again, not something outside of the administrative community should have this type of permission.
If you're an auditor on the box, the audit permission (*AUDIT) allows you to start and stop the audit flight recorder. It allows you to find the types of events that are being recorded and to manage the logs themselves.
Spool Control (*SPLCTL) is the All Object for spool files. Now a spool file is basically printed output that has not yet been kicked out of the printer. The importance of this particular permission really is influenced by what your spooling or printing. If there's nothing out there that would be deemed private or proprietary, it's not business confidential, then *SPLCTL is perhaps less influential than if you're printing your accounting information or even payroll checks. Then it becomes extremely important.
*SERVICE allows a user to manage the hardware configuration of the machine. Up until version 5.1 of the operating system, service permission would get you all the way into facilities we refer to as SST or DST that stands for System Service Tools and Dedicated Service Tools. It's kind of like the BIOS of the IBM i system. Starting in V5R1, IBM said there's a little bit of a risk there, understatement, and so they now require you to Login to the SST DSD environments and *SERVICE just gets you to the front door. It's not quite as powerful in authority as it used to be, but there's still a lot of things that can be done that you probably want to protect.
Job Control (*JOBCTL). This one is kind of a headache, and the reason is that there are some benign tasks that require *JOBCTL. If you have people starting their own print writers (*STRPRTWTR), for example, you're probably familiar with them having this type of permission, but job control is also very powerful. Having a user with this on their own account allows them potentially to end sub systems. You bring the system down into a restricted state, which is not a task you want occurring in the middle of the day certainly not by a non-administrative account.
And last but by no means least we have Save System Special Authority (*SAVSYS). This allows a user to create a media save of any object on the system. Even if you're not normally authorized to that object. Important if you're a backup person, if you're an operator, but it also allows somebody to copy an object. When you save something, you're creating a duplicate. If you can get to the copy and perhaps modify it, you may be able to back it up and relay it over the top of the original. You can also delete data with *SAVSYS by indicating that the storage taken by the object should be freed after the save. So this is a very powerful privilege. It tends to be overlooked more often than not. The good news is, the silver lining here, is that this is one of the easiest to mitigate because you can tell pretty easily who's running saves.
The question I always ask is how many administrators do you actually have? Think of this in terms of what you are paying people to do versus the privileges that they have been given. In many instances, those numbers are pretty low.
What we see in the Security Study is that these privileges are being given away far more in excess than the actual job role exists.
For example, All Object Special Authority (*ALLOBJ) on average, there's 159 of them even in the 2020 study that are carrying this privilege. Security administrator (*SECADM), IOSYS Config (*IOSYSCFG), Audit (*AUDIT), and Service (*SERVICE) all exceeded by fairly large multiples here, and then when we get to Spool Control (*SPLCTL), Job Control (*JOBCTL) and *SAVSYS, we're really pushing the roof up. The green mark is an indication of an ideal scenario. We know it's tough to get much below this level because these numbers do include applications service accounts, IBM profiles like QSECOFR, and so we have an expectation there would be some of these but arguably not in the numbers that were actually seeing.
Tracking Insider Attacks on IBM i
With IBM i, we have to consider insider threat as one of our biggest exposures. Most IBM i systems are within the perimeter firewall. We have the perimeter protection in place, but we have to acknowledge that there are people that are given access to the internal network, of course, and there can be times where people penetrate that perimeter whether it's by introduction of some type of malware, through a phishing attack, or somebody has somehow penetrated that firewall, so we have to consider insider threat as definitely an important consideration.
The median detection period for insider attacks can be up to 18 months. Now, when we think of that it's not really a relatable number until we look at perhaps the seasons that we see and if you live in an area of the country that enjoys all four seasons solidly, you'll know how long it can take to cycle through a year-and-a-half of these, and that's an awfully long time for somebody to be doing things on the system that are perhaps unauthorized or undesired.
The median loss for an insider breach falls around four million U.S. Dollars and half of the victims don't ever recover their losses, so this is not a cheap thing. The median loss of course means that there are some significantly larger losses, as well as smaller losses, as well, but millions of dollars are going out the door just because somebody inside did something they shouldn't be doing. That's not to say it's all malicious. In many instances, they press the enter key and it was a complete accident, and I've seen instances where objects have been deleted or moved inadvertently. I had one energy company that had a consultant who dragged a file in the integrated file system from one folder to another without realizing that their entire ERP system was based off of that folder, and it created all kinds of havoc, and he was horrified that he did it, but it was still just as devastating to the organization to be down.
IBM i Password Recommendations
Passwords, Sandi. Topic of conversation in any security shop, is how do we implement good passwords? This is arguably the most hated thing when it comes to the end users, and so we have to find a way to make this effective.
One of the most important aspects of passwords is the length of the password, and what we see is most of you are falling in the six to eight range. We do see some people have gone all the way up to 20. That was the longest password minimum length that I saw this year, but look below there, 12 systems with the password minimum length of one. Startling. Password of the letter A is not going to keep somebody out of your system.
Yes. Because most of you are in the six to eight range, that's a good thing. This is something that has been discussed in many environments trying to figure out what is the best way to implement passwords. Over the last 18 months or so, this has definitely come out in the open and there's been a lot of discussion about it. Traditionally, a 6-character password was considered best practices. PCI, first of all, took it up to 7, and in 2018 NIST said we want you to be at least at 8, and federal government systems should be able to handle passwords all the way up to 14 characters.
Now, I don't about you, Sandi, but when I do a scan and I tell a customer that they should be using you know, 14 to 16 character passwords, there is a little bit of hesitation. Let's just put it that way. The hesitation is not that they don't see the benefit in it, they just know that they'll never be able to sell it to their user community. One of the things that we advise on is the fact that when you go to a longer password, we're really talking more in terms of past phrases rather than traditional passwords, and that can be very beneficial and actually from a user perspective is a lot easier to remember, and they were quite happy when they see it.
The expiration period is another thing that tends to be up for grabs. The most common is that you don't require a password change. NIST nowadays says that's okay. I don't personally agree with that. My issue with it is the fact that we know users will recycle passwords. They use them for multiple purposes, and there is no way for you to know that their password has been used in a site that was compromised, and therefore, even your strong password is now part of a password diction that can be used in a brute force attack, so at least an occasional password change. To me, as much as I hate doing it like the next person, is definitely advised and recommended in my opinion.
You guys are pretty split on whether you include a digit in your passwords. This tends to be commonly used for character distribution using a 1 instead of an I or a 3 instead of a B, but criminals know this, so their password attacks have become more sophisticated that they will try different iterations of passwords doing characters substitutions with digits. So, the effectiveness of this is something that could be argued both ways.
Now here's an example of a password that might be deemed strong. We're using upper case, lower case, we've got some digit substitutions here. This password is arguably a little tough to remember. If I haven't entered it for a week or so, and I come back, I have to remember did I use an uppercase T or lowercase t? Especially where I use this password in different places with different criteria.
I have the challenge here of a password that is a little bit tough to remember, it's difficult to guess, granted, but it's arguably not very difficult to break. This one is seven characters, so it's not bad, it could be worse, but when we go through and we plow through this staggeringly long and comprehensive dictionary of known passwords, there's a very good chance that we could break this pretty quickly.
If we go to arguably a simpler password, but make it longer, there are more characters and combinations of this by multiple exponential amounts. In this case, a password that I would argue is quite memorable is, again difficult to guess, but now very difficult to crack. Again, if you're using the password in multiple places, a password dictionary could contain the whole thing, so that's why we want to have some expiration here.
As a user, once I realized that a passphrase doesn't mean more, goofy characters and just more of them, this is far more memorable, and I for one am all over this.
What about the attempts allowed for signing in? Most of you are allowing somewhere between three and five attempts, maybe six attempts, after which you take an action. That's okay. Most regulations say three to five. We still see a number of systems that are being pretty generous, seven, eight, nine, ten, but the one I'm most concerned about is on the far right. No Max. There is no restriction on the number of attempts allowed, and there is just no justification in my opinion for that setting for two reasons.
Number one is that will not stop a brute force attack. Eventually that ninety-nine millionth sign on attempt would succeed. If you have an environment that has let’s called them less sophisticated users, and you know that you're going to be beating your head against the help desk wall if you set your limit at three knowing that you're going to take thousands of calls every day, then I'm okay with you upping this number to a bigger number.
If a user doesn't know their password in 10 attempts, let's face it, they don't know their password, and they need to be calling, anyway. If it's just a concern that they’re fat fingering it or they maybe have the non-recommended but common shared account and maybe somebody else changed it, if they get to eight, nine, or ten attempts, they're not going to discover it.
Having a higher limit but having a limit is the most important thing in my opinion.
If the user exceeds whatever threshold you set, then we have to perform an action to respond to it. Ideally, you’re disabling both the device and the profile, the device is also often referred to as the workstation. We want to at least make sure that the profile is disabled, in more than half of the cases that was true, disabling both is another 36 percent, so we're in pretty good shape here, but there’s definitely a concern with the nine percent of systems that are only disabling the workstation device. The concern is that if the user is not using an interface that uses a workstation device, it's typically anything outside of the green screen, then the effect on the user is not noticeable. So, a script under SQL will continue to try to log in because disabling the workstation has no discernible impact. That's a huge problem.
I came across this a couple of days ago with a gaming organization and recommended that they change it that day. While many changes are recommended to go through testing and everything else, this was significant enough that I suggested that they change it immediately.
If, like most, you are not a big fan of passwords, you don't believe they really add a lot of value, then consider using techniques like multi-factor authentication or even single sign-on to eliminate passwords.
Multi-Factor Authentication (MFA) is a technology that is available to you with IBM i. There is some OS capability though, although you have to write a bunch of code yourself. There are third-party tools, such as ours that enable you to do this very easily, but the concept is something that most of us use and don't realize it. You have probably seen an uptick in the websites that now send a code to your phone that you have to enter in, and there's good reason for that. But even something as simple as an ATM or cash machine that requires you to be in possession of your ATM card in conjunction with the PIN number that's in your head, and if you have one without the other, you're not going to succeed. This is just a great way of at least eliminating the exposure that something as simple or as known as a password is the total restriction that we implement on users.
Unfortunately, a lot of us are not taking passwords overly seriously. We saw an average of 120 profiles on each and every system where the password mirrored the name of the profile. So, Sandi, my user ID is RTatam and my password is RTatam, not good.
72 of those were enabled, so that's even worse. Even if you are only allowing a user three attempts to sign in, let's face it, one of those attempts is probably going to be the default. So, there's a good chance that that user is going to be connected.
One system had over 2,000 profiles with the default password. This is something you can report on. The Analyze Default Passwords (ANZDFTPWD) Report will tell you who these profiles are. This is something that you should be reporting on, at least on a weekly basis, ideally depending on how often you provision people maybe even a daily basis, but this should never be done. It's an administrative imposed issue and users can't pick default password, so we can't blame them for once, and it's easy to fix. This is low-hanging fruit, that you can remediate today.
3 Quick Wins for IBM i Profile Security
Dormant accounts. Things that are on just about every system. We see a lot of profiles out there that have not been used in at least the last 30 days. We saw an average of 400 of them and almost one-half of those in an enabled state, so make sure that you're taking care of your profiles at end of life. We want to at least disable them, we ideally should delete them, but if we're not going to do that, then we want to strip away some of the privileges they have. Having a dormant enabled profile with *ALLOBJ is much more serious than a profile that is just simply an end user account that has no public, private, or special authorities associated with them.
Command Line Restrictions are another quick win for you guys. The limit capability setting is a really good way of restricting what a user can do admittedly, predominantly on the green screen. We need exit programs to control this on the PC side of the house, but this is still a great control to use. Unfortunately, 255 profiles, again as an average, can run commands right here on the green screen. The rest of them, if you don't have exit programs, can run them from a DOS prompt. This is an area where you can see very rapid improvement of your security, and we'd love to see a distinct improvement here by the time we run this next year.
Who cares about this? Well, a lot of people do. PCI is a regulation that a lot of you are dealing with. Even if you don't handle credit cards, this is a good regulation actually to follow anyway, because they actually have common sense requirements.
One of them is to establish Access Control based on a need to know. In other words, set your security to a “deny all” and then open the door gradually to the people that have a business need to do so.
They even give us a warning that some access control systems are set in the opposite approach, which is allow everything until you explicitly restrict it. That's not a good approach. It's completely backwards to what it should be, but the startling thing here is IBM i is pre-configured in this state.
If you create stuff on IBM i and you don't change the configuration, it says do what you want, come on in, and that's not what we want to be doing.
I've mentioned *PUBLIC a few times. This is really a reference to any user that doesn't have all object and that doesn't have an explicit permission based on their profile. What we see is the authority to the libraries on the box is wide open. Only 10% of the libraries are in the desired *EXCLUDE category. 2% are arguably a margin of error, an authorization list (*AUTL) first, then a decision somewhere else, and we don't know if it's open there or closed there, so I don't know what's happening, but it's a relatively small percentage, not hugely influential.
What's more important is that 88% of the libraries say you have a user ID, then come on in.
When we look at the objects inside those libraries and how they're created, on the left side, we see 12% explicitly being even *CHANGE permission, but the vast majority of libraries defer the decision on how much public permission should be given to a system value. So, we chase that trail and we went to the system value and said, how are you set up? Another 78% of those system values said set it to *CHANGE. By far the most common setting here at an object level is *CHANGE permission, which is enough to see data, modify data, and even delete data.
Malware, Virus Protection, and Security
Now Sandi, I know malware is a topic that is near and dear to your heart. You have spent many years working with customers, and when it comes to malware and different viruses that can impact the system, talk to me just very briefly about what was going on with this guy right here.
Ooh, ouch this was a bad situation. You know, this is a combination of all those things we have already talked about with users with too much authority and too much access to the system. Map drive with All Object Authority (*ALLOBJ), PC got hit by ransomware, followed all the network drives including that map drive, and ended up encrypting half a million files on this customer’s system. And when we help them scan, we found that two hundred and forty-eight thousand copies of that malware had been dropped out in their directory. Significant downtime, significant impact, and the recovery was really painful for them. It’s a very unfortunate situation that I hope to never see again.
Well and sadly, we do see it. Maybe not to this extreme, this is kind of the poster child I think internally when we talk about viruses on IBM i, but the bottom-line folks is that you are not immune. There is definitely risk here. It's a different risk or it manifests itself slightly differently than perhaps on a Windows server but don't be fooled into thinking you cannot get a virus on IBM i. We can host, we can also be impacted. IBM acknowledges that in many instances most obvious in the fact that they included some antivirus functionality in the operating system. What we saw in this study was that 14% of the servers that we looked at were indeed scanning the system prior to a File Open.
Now the important thing to know here is the OS has the infrastructure, it does not have a built-in scan engine, so that requires the addition of a third-party commercial scan engine. Fortunately, HelpSystems has that engine through a relationship we maintain with McAfee, but this is a growing control. People are waking up to the fact that IBM i is not immune, and I'm very encouraged by this statistic, and I hope to see this even bigger next year.
What IBM i Security Issues Are Businesses Struggling With?
All right. Let's shoot out another poll here. I'm going to pull up a question that I wanted to ask everybody. And that is, what IBM i security issues are you dealing with? Is it an audit reporting challenge? Is it oversight of the PC connections that are out there? Are you dealing with accounts that have all object or other privileges that you're not sure how to reduce? Are you trying to stay in touch with security mandates, so you need to be compliant with something like GDPR, PCI or SOX? And are you also struggling with management justification for doing security work? We know that management is at least taking a closer look at security. Maybe the IBM i platform doesn't get as much love and attention as perhaps some other operating systems that shall remain nameless, but we're interested in whether there's a justification challenge there, too.
Looks like management justification, Sandi, is about 42%, jumping out there ahead of the PC connections. Not entirely surprised by that. I'm encouraged by that because I'm glad that you guys are seeing that your menu security and your command line restrictions, while they have their place, they're not fully effective for you today. So, that's good feedback, thank you. Definitely across the board concerns. Every one of these is over 30%. All but one of them are over 40% of you saying that you have challenges here. So thank you. I'm going to go ahead and close that poll out, and we'll go ahead and wrap up here.
Key Takeaways from the 2020 State of IBM i Security Study
I want to leave you with some key takeaways here. I don't about you Sandi, but I've been using take out a lot recently. Hopefully we can get back to sitting in a restaurant pretty soon, but this is an important aspect from the security side, as well.
My first takeaway is that we want to make sure that we are policing the perimeter using Exit Programs. I cannot emphasize enough how important that is, even if you have good security configuration this still has a part to play in auditing those connections. If, like the vast majority that we've seen through the study, you don't have the best configuration, then this can be the make-or-break for you. And these are the high-risk interfaces that are least guarded, so let's get some resolution there.
Restrict access to the database. Now, that's a matter of restricting Public Access, it could also be Encryption, as well, making data so even if it is accessed, that it's non-readable. That has a lot of benefit under regulatory mandates, but just common sense tells us, if somebody breaches my data and they can't read it, then I obviously have saved the day. Again, I don't want to over emphasize the fact that it's not just about data breaches. It's also about system access, system interruption, and business process down time.
Limit the IFS. Sandi, you gave us a great example there of somebody who unfortunately had an open IFS and that is more often than not the case. Some of you are now starting to scan that, but we would love to see that grow further, take advantage of the integrated file system (IFS) and its capabilities but do it safely.
Check your system values. Too many of us have the default settings that IBM set because we don't know what to change them to, or we changed them in 1992, and we have propagated them from server to server every iteration of the box since. Set them but don't forget to go back and look at them once in a while to see if they've changed.
Have good password policy settings. If a user can access the system, we know there's a lot they can do, so make that hurdle a decent one. If you can, implement multi-factor authentication or at least consider single sign-on to eliminate passwords, but make sure that we have secure connections, we're using SSL TLS for our communications, for our sessions, so that people can't just sniff the communication channel and see those user IDs and passwords on the wire.
As I mentioned, there is a growing understanding that the security is important. We do see improvements. We're anticipating more this year. We've got a significant transition right now to a work from home community, and people are concerned that that has introduced a lot of new security risk. Honestly, a lot of the risk was already there. It's just shining a spotlight on it, and we have a lot more users, so it's harder to manage but this good security requirement is definitely something that that we see coming to the forefront.
Robin’s Call to Action: 5 Things to Improve IBM i System Security Today
Now Sandi, I have a call to action. I have some homework for people to do. I've got five things that people want to consider here.
- Do some kind of scan of their system. It can be a free Security Scan or it could be a detailed Risk Assessment, and we can help with both of those but look at your system and understand where you're at.
- Fix some of the low-hanging fruit items. You have default passwords, you have dormant accounts, that's easy to fix and should be resolved quickly.
- Check out other profile settings. Limit attributes or limit capability attributes, special authorities, password rules, all of these things should be documented, looked at, and determined as to their reasonableness.
- If you have exit programs or even if you don't have exit programs then do some intrusion testing against your FTP and ODBC type interfaces. How far can people get? What can they do? Can they run commands? Can they download that production file? And if they do so, do you get a notification? Oftentimes we see exit programs in play, and they’re not well configured or they’re not sending notifications and we're reliant on them, but we don't realize they’re not set up right.
- Number five, in many instances these solutions have already been written and deployed and tested. Don't reinvent the wheel, take advantage of what's out there. You'll get to market a whole lot faster if you take something off the shelf, especially when it comes to things like exit programs and audit reporting, the third-party solutions that are out there can be extremely beneficial in giving you ongoing visibility.
I mentioned we can do a free scan. This is a process that Sandi and I live and die by. There's no cost for the use of the software, and there's no cost for us to review that finding with you. We will go through it line by line and explain exactly what it means. There's no cost for our time involved there either. There's no intrusion into your system from that standpoint. We don't look at any application data, we are not changing any settings, we're simply scraping off some of the statistics and the metrics around your security configuration. This is just a great kind of health check for where you're at.
I have one final poll for you guys if you are interested in this then let us know or if you would like more information, which I totally understand you may not be willing to commit now, if you would like to take advantage, but you'd like to know more about how it works and what it does then let us know that, as well, and we can follow up with you. If you decide later on that you would like to do it, you can always go out to the website and sign up for one. The benefit of saying you'd like to do it now or that you'd like more info is you don't have to fill out a form. We will just take care of it.
Sandi, I think that is kind of the lion's share of what I want to share. I know we're just a couple of minutes over here, and I want to be respectful of everybody's time. Knowing that this was the first time we've presented this, I wanted to make sure we packed as much into this as I possibly could, and I think that's important. So it's good that we've had that time to converse.
The takeaways from here are that we know that security is something that a lot of us in the IBM i community are not overly comfortable or familiar with. Unfortunately, the audit professionals that we rely on for this are not that knowledgeable about our platform, so we have this disconnect. Even though we have tons of valuable data, it’s not well secured. Our users are too powerful and it's obviously accessed easily via a PC interface as we've seen in many instances with no oversight, so we want to address some of these things.
The Security Study is packed with 17 years worth of insight at this point. It's available for download now, but just to save you that trouble we've thrown it into the handouts section of the webinar, so if you want to grab that and take it with you, feel free. If you don't catch it or if you share this recording and it wasn't available in the recording, then just head out to our website and you can grab it from there. And this is literally off the presses today, so the image for the cover here, Sandi, you may notice is not the correct cover because it only was uploaded today. We wanted to get this information into everybody's hands as quickly as possible, and so we're really really excited about how current and fresh this content is for everybody.
Awesome! It was good information, and I appreciate that you took the time to put this all together and share it with everybody.
I do see one question here. Somebody's asking about how this relates to prior years’ studies. The answer is we don't review the same servers year in and year out, so we have to be careful when we look at trends. There's definitely some areas where we're seeing enough improvement for example with the security level and the people that are scanning for viruses that I think aligns with a lot of market initiatives, so I'm encouraged by that. There's other areas where we don't see a lot of improvement. The public permission is something that doesn't seem to change much each year, and I think that stems back to a lot of us are intimidated by that function. My suggestion is look to compensating controls. There are things that can be done that are not as time consuming or require as much planning, so that while I encourage you to have that on your long-term radar, we look to things that can be done in the short term. So, the study will give you the indication of some of those things. In the future, I'm hopeful that we continue to see these trends coming up but only you guys have control over that. It's your decisions as to the changes that you make or the questions that you ask that will tell us in 2021 whether we're having the same conversation or whether we have made some improvements.
All right, folks. I appreciate the time. Thanks for sticking with us. We are always available. If you have any security questions, we have lots of solutions and services that we can bring to bear and we would love that opportunity. But even if you just have a simple question and you want to ask, we are very well connected, we have a lot of great people on staff that can answer those questions, and we really enjoy having that interaction. So please take advantage of that resource and keep Sandi and I busy. That’s what keeps us out of trouble, right Sandi? And we need that.
There you go. All right, if we missed any questions, we'll be sure to get back with you offline.
Absolutely. Keep safe everybody and stay healthy. We appreciate you being with us and look forward to talking to you again soon.
Find the security vulnerabilities on your IBM i and get expert recommendations for fixing them with a free IBM i Security Scan.