Is Cron Scheduling Holding You Back? Find Out With These Resources

Thank you for joining me, everyone. Welcome to today's webinar on “When Cron Doesn't Cut It.” What happens with cron, it seems, is people outgrow it or their systems outgrow it, and that's a good thing because that means you're growing, but sometimes cron just can't keep up. My name is Pat Cameron. I'll be your presenter for today, and I've been with HelpSystems working on 17 years now. So I've worked with...I was an operations manager before that, so I feel your pain out in the real world, and I've been working with a number of different products that we have for scheduling, automation, and monitoring. And today we're going to be talking about outgrowing cron like I said.

Cron is a great little scheduler for any of your UNIX and Linux servers, but there are some pros and cons to using it for scheduling. We're going to be talking about some of those. We'll talk a little bit about cron. I'll talk a little bit about scheduling with Skybot Scheduler, which you can use to replace cron, and it's got some additional functionality that's not available in cron. I do have a live demo. So I'll show you how you can very easily import your jobs from cron into Skybot. We want to make it quick and easy for you to move from your current cron scheduling to Skybot.

A quick show of hands before I get started: how many of you are using cron out there? There should be a little hand down below your name, and if you want to just click on that and raise your hand, I can see if there are a few of us out there that are using cron. All right, I've got a few people out there that are using cron right now. Great. Well, let's see if we can talk about a better way.

So like I said, cron is a great scheduler. It’s free, for one thing. It's hard to compete against free sometimes. And if you've got one server or two servers, I think it works quite well. If your schedules are simple and time-based. What I seem to be running into with a lot of our customers, is that they have event-driven schedules, and they want to trigger tasks based on some events outside of the script that they've got scheduled within cron and that's where they run into problems. But again, if it's a time-based schedule and you need something to run every day at a certain time or a couple days, a month at a certain time, cron works just great.

The problem comes when you have some dependencies, and then you what you've got to do, is you've got to edit your scripts to take care of those resources and those other dependencies. And so what that does is make your scripting more complicated, puts a bunch of stuff in there that isn't really what your production jobs need to be doing, but you have to kind of build your scheduler into a lot of your tasks. And it's great if you've got one system. I would certainly recommend using it if you're just managing one VM or one system these days, for any type of scheduling. I've talked to a lot of customers that are using cron to schedule their jobs on Linux and various flavors of UNIX, and like I said, it's free, so it seems to get the job done if you've got simple schedules.

But what I have heard from customers, is a couple of things. One, they found it kind of difficult to edit that crontab file because of the way the editor is in UNIX. They weren't very happy with that, as far as knowing what all those symbols mean as well within the cron entry. Another customer used their developers to write scripts to manage those other resources and other job dependencies that their processes needed. So what they were using is that expensive developer time to manage their day-to-day tasks, and that just was not a good use of that time or money. And also this makes the scripting a lot more complicated than needs to be. One customer commented that, "I'm not a programmer," and just didn't quite understand how our cron is working.

One of our other customers was looking for a way to audit changes that were occurring in their crontab files. There really is no audit trail. If somebody changes a file and saves it, that's it. The other problem with it is that you will have multiple crontab files on your systems. They might be running under specific users or they can have a daily, weekly, monthly type crontabs, and what happens is you kind of lose control. You don't know who's running these jobs and where they're running and the time, and you might end up with tasks that run concurrently that really should not, and that could cause you problems in the operations area as well as whoever is managing those systems for you.

Often there are different types of events, like file events—that sure seems to be common these days. There's so much file movement that's going on between systems, between your clients and your servers or your vendors, or the government. I was on a webinar earlier this morning and people were pushing reports out to the government using FTP, and they had to wait until files were available to do that. And so what they would have to do is kind of build in some delays into the schedule, instead of being able to run those dependently based on that event of that file changing or the file getting created. So, some good things about cron: if you're a small shop and you have a simple schedule…but it's very easy for you to outgrow it and with the complexities of the way the world is these days and the way operations have to operate that cron can be quickly outgrown.

So let's take a look at a couple of ways this Skybot Scheduler can help. Actually, before we do that, I've got a couple of polling questions, like to get a little bit of information about you. So if you could answer this, that would be great. Are you using an enterprise scheduler already or multiple schedulers? A lot of times people will have different applications in addition to cron. They might be using cron and then maybe some applications that include a scheduler as well. And so sometimes they're using that for their scheduling instead of having it all in one place.

So, interesting to know where you guys are all at. Looks like we got a few multi-platform, probably have some Windows using the task scheduler and cron. We can also import that Windows task scheduler for you as well. So like I said, I want to make things easy. All right, fabulous. Thank you very much. I can share those. Hopefully you can see what we've got here. Thirty-six percent of you are using an enterprise scheduler, 55 percent have a multi-platform base schedulers, so maybe you have some work to do to make things a little bit easier in the operations area.

So talking about Skybot. So what Skybot does is it replaces cron, and it would replace any other schedulers that you have out there. Windows task scheduler as well. It's easy to use, there's a browser interface, so there's no type of a client that it need to install. I think the biggest advantage that we give to our customers is this ability to have a central console to go to one place, to be able to set up jobs to run, monitor those jobs while they're running, set up those dependencies across different servers, etc. Everything is done with drop-down lists of options, so we want to make it easy. You don't have to be a programmer to schedule Skybot jobs certainly, and you don't have to be a developer. So it's nice to be able to pass that function to a production control area, sometimes to the help desk or the operations area, so there are other staff that are able to maintain those schedules for you. Make it easy to do dependency scheduling. You can have role-based security set up in Skybot, you can have different people that have different types of access to the schedule. Some groups may be only able to view the schedule whereas others can execute jobs, or some can actually make changes, maybe only to a certain subset of jobs as well. So you can get very specific in who has what type of access to the schedule.

As you can see here, we do have some integration with some of the big applications out there in the world: SAP, Informatca…probably two of our most popular ones. Both of those products come with a scheduler, but it's not as robust as Skybot is, and also Skybot will allow you to schedule an SAP NetWeaver job and as soon as that completes, trigger a SQL Server agent job. So you can schedule jobs from all different applications from one place. And we do have an import function for cron and we'll be taking a look at that shortly. So the central scheduler allows you to create those dependencies and then view them from one place. You can create job flow diagrams, so that you can document what that reactivity is, what those dependencies are.

I know when I was in operations, that's the hardest thing to figure out. Okay, when this job runs, what's going to run next? And so if I've got a nice flow chart that's going to show me what those dependencies are and to make sure that if this job fails, this one won't start. That's the way that you want your schedule to be set up so that you don't have any errors, you don't have to go back and do reruns or restores because something ran out of order or ran before the resources that it needed were really ready. One of the other advantages that we give to our customers is that ability to monitor that schedule so you don't have to. Here at HelpSystems, we always talk about managing your systems by their exceptions. If everything is running fine and on time, don't bug me, and I don't want to have to go login and take a look at it. But, if a job fails or if a job is late, then I need to know about it right away. I can notify the help desk. We've got a couple of ways that we can notify for those types of exceptions. We can send an SNMP trap to your help desk ticketing software, open up a ticket automatically…we can send email to individuals or to groups and text messages as well.

And there are a number of different types of the events that we can notify you of. I'm a firm believer in don't send me an email every time a job completes successfully. I don't really care about that, but please let me know if something is out of sync, or something is aired, or there is some type of an exception. And then with the security options that we have, like I said, you can set up different groups of people that have different types of access to the schedule, and will interface with active directory. Anything that we could do to cut down on the ongoing maintenance on the product, we tried to do in this, and this is one of them. So you don't have to set up a bunch of separate users for Skybot, you can create some groups over on your AD server and then map those groups over to roles in Skybot and manage your users that way instead of having to, like I said, set up separate users just for our product.

So I'm going to go online and show you a few things about Skybot. We've got some dashboards, we can take a look at that. What I'll do is I'll run through the import and show you how we can import those crontab files and create Skybot jobs very quickly, and then maybe add some notification to them, add some dependencies. So once you get them onto your Skybot server, it's a simple matter of setting up any kinds of additional functionality that you need surrounding those jobs. So let's go online.

So now before I continue on, if a question comes up and you want to send me a question, if you put your cursor up at the top of the screen, that drop-down box will come from WebEx, and you can just click on the chat and then it'll bring that chat window up on your screen. So, this is the interface for Skybot, and what I have here is a flow chart of jobs that are dependent on one another. So, everything is managed through these drop-down menu options. And this is a quick and easy way to have a job. So this is a job that runs on one of my Linux servers, you can see here it's a JD Edwards job.

And what I've done is, I've just set up a prerequisite. So that demo SQL agent job needs to complete before this JD Edwards job will run. Actually, I've got a couple of prerequisites, or I've got a scheduled job called "checkpoint one" that also would need to complete. But I've got these grouped together to save either of these events that occur, run that JD Edwards example. So those jobs are over on a Windows server. Actually, this is a SQL Server job, and it can trigger a job over here on my Linux server very simply and very easily.

And if you keep all of that outside of your scripting, it makes it a lot easier to change things up. So I'm going to go click on...this is the import center that we have with Skybot. And these are a bunch of crontab files that I've imported into Skybot and I'll do one for you in just a second. We also can import other file types: Windows Task Scheduler, other schedulers that you might have. Typically you can export those schedules into some type of a text file, and then we can take it and do this import. So I'm going to bring up one of my UNIX servers.

So just to show you, so when you install Skybot on a host system, the main server, and then you install an agent on any of the Linux and UNIX or Windows servers that you want to run tasks on, when you install that agent, we install this shell script that you can run for importing your crontab files. So to run that, I just run that, hopefully I spell it right, and then the parameter is the name of the crontab file that I want to import. So I need to know where it is and I need to know the name of that file. If I go ahead and hit enter, what Skybot does, is it goes and it reads through that crontab, it leaves it right where it is, we don't make any changes to that file. It takes a copy of it and it puts it over on the Skybot server, so we can import it. If you've got a big crontab and you've got lots of entries in there, lots of jobs that are running, you don't have to switch overnight, certainly. I'll show you when we do the import, we will import those jobs on hold, and so you can decide or put a project together of when you want to move those jobs from cron to Skybot.

So now if I go back here and refresh the screen, I should have a new crontab. As you can see, I use the same one, but here's the one that I just imported just now. So I can right click on it, import to Skybot, Skybot goes out and reads that file, and it looks at all the different entries in that crontab file. And now it gives me an option, do you want to import all of them? Or I can just import a couple of them. Again, however your plan is for moving those jobs, you can be very flexible in it.

I'm going to put these on hold, when I bring them in, so they're going to keep on running over in cron until I either edit them out of the crontab file, put them on hold, delete the file, whatever. And then I'm going to put a prefix on here. So what Skybot's going to do is create different jobs, one for each of these entries, and I'm just going to put this prefix on there so I know that these are the jobs. So you can put anything that you want on there. We've got a cron tag that we put on, you can add other tags if you want to. These run under the root, so I'm just going to set this environment up, so when it imports those jobs, what user do you want to run these under? And that's it. I don't have any variables that are here, I'm going to go ahead and import. "Are you sure?" Yes, I am.

And so Skybot goes out and reads that file, and will create eight Skybot jobs for me. So if I want to see those, now we have a list of the Skybot jobs that were created. They're just named one through eight with my webinar, cron, on the job name. As you can see here, they're all on hold, and that means they won't run until I take them off of hold or move them or do something. And we also have the schedule. So we import the schedule, and we'll tell you the next time that this job is scheduled to run. If I go ahead and open this guy up, I'll just show you. So this is how you set up a Skybot job. You'll always get a template that you can fill in the different options for the scheduled job, name, the agent that it's running on, but again, our import does that for you automatically.

Now, you can change this job name to something that makes sense or something that maybe some of your business analysts need to view some of the jobs that are running in your schedule. You can make this a job name that they will understand, and also change the description. We just take that entry and put it in here and we put it in a number of places, so that you know what you started with. This is the agent that it's running on. We might have some multiple queues, we can have single threaded queues etc., kind of manage jobs that way. I'll just leave these at the defaults. Calendars that are used for holidays, I will always default to standard, but you can change it to a holiday calendar or a fiscal calendar if you need to.

And then for the schedule, when we import a cron job, we've got a schedule type called "cron exppression" and we'll just use that same syntax that's used in the crontab file. You can leave it that way or you can change it to maybe a day of the week that runs at 11. I would get the same result, this job would run exactly the same way as it does whether it uses this day-of-the-week type schedule or the cron expression. Either one is always going to run at 11. So those of you that are comfortable with this kind of format, you certainly are welcome to use that.

I can build in some exceptions. Maybe I don't want this to run on holidays, so I could say, "Run on working days only," can add that to the schedule. Here it's filled in the agent environment with a user etc. Notice that you don't have to know the password for whatever user these jobs are running under, that's why it's nice to be able to pass this to another group, system administrators don't have to be setting up jobs. And then the commands it's running.

So here we've got the command line. Again, this is just a comment and it's the entry from the crontab file, and then this is the actual command it's running. I can remove that comment if I want to, and I can add some other tasks to this same job. So maybe when that first script was done running or that first command, I would want to do a file transfer. I can very easily do that as well. You can define your FTP servers to Skybot, the remote and local file name, etc.

Again, you'll always get some type of a template to fill in and fill that out, and you can add that. So you can have multiple commands, multiple scripts in one Skybot job. We capture the log, so anything that's written to standard out and standard error will create a job log and keep that with our history. So I'm kind of contradicting myself here with the cron expression. You determine how much history you want to keep and we'll keep those history records. Every night Skybot does some kind of a cleanup and removes any of the old records based on whatever retention time that you've given it.

A couple of monitoring type options or notifications. So here's where you would set up if I need to be notified of an overrun, underrun, or late start. So if this job tends to get in a loop, I can say, "If it runs longer than 30 minutes, that's a problem, let me know," or if you've got an SLA and you need to have it finished by a certain time, you can put in that time. These monitors were put in specifically for SLAs. And then the other notification that we have is if it fails, or any completion code actually under status notification. We submit it to a queue. It may be skipped based on some condition that you've selected. Here the job is active and running. If it completes successfully, I can send an email maybe to accounting and let them know that their report is finished, email the report to them. But if it fails, I can send notification to the help desk.

So depending on the status that a job gets, you can have different types of notification. I can set up a dependency here as well, so I can use the same job and I can put a prerequisite on it, and that prerequisite might be...I've got some file monitors that I have out there that are called "agent event monitors,” and again I would just pick from a list, get a backup file, make sure that backup file exists, and then it's okay to run that cron job. You can build these prerequisites—as many as they are—build that flow chart like I've got, depending on what your prerequisites are, for lots of flexibility and when a job runs and also notification if there are any problems. So this is a great view. If you want to see what those dependencies are, you can build this job flow diagram. Skybot will kind of help you build it. It knows what those dependencies are, so you just select a place to start and it will automatically build all those dependencies in.

So like I said, I've got these on different servers and they're triggering one after another based on the completion of the previous one. So this JD example will not run unless this job is successful. If it fails, it's going to stop and send out that notification. A quick look at our dashboard, this kind of gives you an idea of what's happening, how busy is your system, how many jobs have been submitted over the past 16 hours, these are your completions from the previous week, 7 days. And if I want to see what failed yesterday, I've got all my terminated jobs, I can just click on one of these charts and it's going to take me into the detail for all of those failed jobs.

So here is a place where you've got that central view. I've got the name of the jobs that are running, the agent that they're running on, all my different servers, and then the status. So these are all my failures, all my bad boys. And then over here I've got the start time, the end time, the duration etc., so these are the types of history records that we keep. Also if a user runs a job or if it is reactive or if it's scheduled for a certain time, what triggered that job to run?

And then always from here, right clicking will give me the option to download the log and see what the problem is. Maybe I've got to go edit that job and make a change, the command line was wrong, and then I can restart it from here. So this is a great screen for the operations people to see what's happening. The other one is our schedule activity monitor, and this refreshes automatically, shows you the jobs that are forecast for the next 24 hours. What's going to run later today? What's going to run tomorrow morning?

And these jobs were missed. They were in the forecast and they didn't run, no doubt because we're doing a bunch of testing and the prerequisites weren't met for those jobs. These jobs are active, and this graph shows me when they started, and then based on history, how long they normally run. So this is another great view of my whole enterprise schedule. I was going to show you the audit history before I leave.

So changes to jobs. So in my audit history, as you can see, I created a bunch of jobs just now. So it's going to give me the date and the time when I did that import. Skybot tracked all of those changes and if I look at the detail, it will tell me the name of the object that was changed, the type of object, who did it. It was me, and this was the date and the time. And then these were the changes that were made. It will always show the original value and the new value. So when we did that import and created a bunch of new jobs, here's the new jobs, and then it goes through and updates the commands etc., and it will show me all of the different fields and all of the values for those fields. So that's cron and Skybot.

We have a Geek Guide that we worked on with a Linux journal that we'll be sending a link to you after today's webinar sometime, so that might be helpful in determining: when is it a good time to move away from cron, have I reached the limit? And so you can download that along with information about this webinar. So if anyone has any questions, I'd be happy to take them. I will hang out here for a couple minutes. It looks like we're right at 29 after the hour, so I've got a couple minutes for questions. If not, that's all I had to say today. So thank you for joining me, I appreciate you taking the time to take a look at Skybot and see if it can make your life a little bit easier as far as scheduling is concerned. So thank you very much and have a great day.

No transcript available at this time.