Alternate Reactive Paths: How Smart Is Your Job Schedule?

Welcome everyone to today's webinar on alternate path processing. My name is Pat Cameron and I'm going to be your host today for our webinar. We are going to be talking about how you can build an alternate path into your job schedule. Couple of housekeeping things: over on the right hand side, if you have any questions during today's session you're welcome to send me a chat message. I will try to get all of your questions answered as soon as I can. The other thing I'm going to do is I've got a polling question if you could bear with me just a minute and I'm going to go ahead and open that up. We just've got a couple of questions about your environment, what kind of job schedule are you using now. I see we have a lot of Skybot customers with us today: welcome to all of you! I see a lot of familiar names so that's always great to see. So if you could take a minute when you have a chance and go ahead and answer those questions and then when I get a chance, I'll go ahead and share those answers with you.

Alternate Path Processing

So today we are going to be talking about alternate path processing, what is it and why is it important. We are going to take a look: I've got an example of a business process that runs every day but it's got some alternate steps on certain days of the week or certain days of the month or what if there is an error on one of the steps, what can you do. I'm going to take a look at how we can build those exceptions into your Skybot scheduler so that you don't have to monitor those jobs every day. We can build those steps in up front and kind of set it and forget it. For those of you that are Skybot customers, we've added some additional options to jobs suite recently that you might not be aware of so I'm going to go and show you some of those as well as some of the other options that we've got for scheduling that might not be used as often as they could be. So hopefully after today we can take your scheduling up to the next level in automation.

Can You Handle Exceptions Automatically?

So this is the business process I'm going to work with today and, as you can see, it's probably kind of typical. Actually it's got a CRM chunk up here that's got multiple steps in it. It's got some orders and payment steps again, additional steps, they probably run together. Down here we've got ERP chunk of information and all of these feed into the data warehouse. So this is where all of our inputs go and then from the data warehouse we output into different applications, SharePoint, Excel, etc., and then also run some management reports. So this is kind of a typical business process that probably runs every day, maybe multiple times of the day, and if you can see, we've got smaller pieces of a larger flow. Most of these pieces depend on others and, again, they probably feed into that data warehouse piece. So this is pretty standard: multiple steps, multiple applications, they all kind of depend on one another and they all feed into a specific process. But what happens if there are changes to this schedule? Because these scripts are related in some ways and they stand alone in others, there might be some differences in how they need to be scheduled: an alternate path. Maybe that on certain days of the month or certain days of the week and how can these exceptions be handled automatically and easily within the schedule?

Errors in the Job Stream

So the first thing we can see is we might have some errors; and what can we do if we see an error in that jobs stream? Is it okay to skip over that step or is it critical that that step gets run and reprocessed? One thing that is critical is for whoever is running the schedule or creating the schedule that they know what those requirements are for your jobs. Are these dependencies absolute or can they be passed over periodically? To schedule change during the week, maybe you've got some jobs that run only certain days of the week; you've still got a dependency here in your ETL process and your data warehouse but if these don't run every day, how are you going to handle that? Month-end, good old month-end, always seems to be a problem for just about everybody that I've ever worked with. So month-end or weekly jobs can change the prerequisites and once these variables are known, then it's possible to set up your schedule for all those tasks to run.

I like to make sure that the schedule is transparent so that anybody that's monitoring it or running it is able to see exactly what's going to run today and tomorrow, next week and next month. You want to make sure that the operations people or the help desk are able to see what's running now and what's scheduled in the short term and what's scheduled for the long term, next week or next month. All of these types of views are important for anyone that manages the schedule and is responsible for running it successfully.

Skybot Scheduler and AutoMate

Let's take a look at a few things that Skybot can do to make sure that all of these jobs run correctly. We're going to take a look at Skybot Scheduler and how we can run tasks within the Skybot product. For those of you that are new to Skybot, and we do have some people that are not familiar with Skybot with us today, it's an enterprise scheduler and what it will do is trigger tasks across multiple applications, multiple servers, and multiple platforms. We do have several interfaces into some popular applications, but we like to keep our products pretty generic so that you can run any type of a batch, executable, web service request, or any kind of a script—pretty much anything that will run from a command line you can run in Skybot.

I am also going to do like a sneak preview today of another product that we have called the AutoMate. AutoMate is a fairly new product and we've integrated it with Skybot. What AutoMate can do is it can create scripts for you so that you don't have to. It can do things like monitor an inbox on your email and then perform some application, perform some function, process something based on that email. Create a spreadsheet in Excel; log into a web browser and download some files. What I'm going to do with it today is I'm going to run a little SharePoint process that's part of our business process and to show you how you can do that with AutoMate. We've had a lot of requests for those types of tasks and we're pretty excited about that product as well.

Complex Job Streams

So what I've done here is that I've taken that process that we've got that runs all the business processes and I've broken it into some smaller chunks. So within that process we've got an order and payment processing. So we get new orders that come in, we monitor for those files and then we trigger the processing for orders or the processing for payments, either one. Those are prerequisites to an accounting job that runs. So these guys might run multiple times a day, hopefully they're running hundreds and thousands of times a day if you're getting orders and payments, but our accounting jobs just runs at 18:00 every night. So it kind of summarizes that information and then feeds into our ETL load for the data warehouse, so that's one little chunk that we have.

CRM processing. So over on the CRM side of this job stream we've got different modules that run concurrently with that order payment processing. They might run at certain times of the day, our various models run throughout the day at different times but notice here that this channel module, it's only scheduled for a couple of days a month. It runs of the 1st and on the 15^th, but it still needs to be a prerequisite for this ETL load. So how can I set this job up so that even though it's only running a couple of days a month I don't want the fact that it's not running to stop this ETL load from running. So we'll take a look at how we can do that; it's kind of an interesting alternate path.

Also down here notice that this is analytics job has a dependent job if it fails and so what we can do is we can run some kind of a recovery and then restart this analytics job. We'll force that to run again. So it's got a little loop down here that will do the recovery and then run the analytics job. We're using our web service commands to run this CRM analytics job again. I found a lot of uses for that web service command. I'll show you a couple of those today.

Then we've got our ERP processing. So over on the right hand side of our flow chart, we've got the accounting. Again, this is based on our order processing that's up above here. It has a different prerequisite at month-end: how are we going to handle that? We've got some month-end accounting that we need to run. Also down here you can see the ERP planning. Some days it runs and some days it's skipped, so we want to be able to make sure, again, that the ETL loads runs every day, no matter if this planning job runs or doesn't. So how can we set that up within Skybot?

Then what we've got at the end of this process, we're triggering a reporting suite of jobs and then this is where we're also uploading some files to SharePoint, I'll show you that when we go online. So when we put that all together, we've taken all three of those chunks and put them into one business process. We've got our payments and orders, we've got our CRM chunk of jobs, and then we've got our ERP and ETL jobs, and again, these are all kind of smaller pieces that belong to one big process.

Job Suites in Skybot Scheduler

Let's go ahead and go online and see how Skybot can handle these. Hopefully you can see my job suite. I'll give it a second to catch up. I'm going to bring up the chat window. So now I've hidden the chat window. I told you can ask me questions, now you can't, but if you put your cursor up at the top of the screen, WebEx will drop down a box and you just click on that chat icon and that will bring that up in your window. Again, if you have any questions, you can send them to me.

So just a quick view of what we've done with job suite for those of you that use them, you can see that I've got a couple of other different fields here. So in the past, when you've created a suite of jobs, so these jobs are all triggered at a certain time each day and they may have some prerequisites as well. In the past you could only react to a completed status on a previous job. What we've done is we've given you the ability to change that dependency.

So once you get your suite set up and you've got all the lines that are indicating those dependencies, right click on anyone of them and they will give you this dependency option and now you can change that. So you can say if this dependent job or this prerequisite job, I don't really care if it's completed successfully or if it fails, I want the next job to run. So I can react to any of those statuses and so you can change any of the jobs in your suite to react appropriately. The other thing you might notice here, see this little warning signs that's on this end, a warning sign here we've got a number of those.

Another addition to our job suite is this member option. So again, once you get your job suite set up, you can determine if this one fails, so if my CRM one job fails, do I want to fail the entire suite or not? And you can select that. Then also the required prerequisites: for down here, when you've got jobs that have prerequisites, do you need all the prerequisites to run? Do you need any of those prerequisites to run? Then that member might run multiple times if it's got multiple prerequisites. It might run every time a prerequisite runs or just the first time one of those prereqs.  So we've given you a little bit more flexibility here in creating a job suite and how those jobs run.

Special Instances

The other thing that you might notice here are the big red Xs. So I've got this job suite scheduled to run every day but I may not want all of the members in this job suite to run every time, so you can create what's called a special instance. I've got a couple of places today where I'll show you special instances. Special instances are like a little calendar or a little schedule, and I've got one that flags the 1st and the 15th of every month and I've got a drop down list of the available special instances for this job suite. I can add that to any of the members in this job and then I can say, skip all the of the run times that match this special instance, which is the 1st and the 15th, or I can say skip all the run times that are not matched by that special instance. Again, depending on the days of the month or days of the week, you might have some type of schedule exception. It's nice to be able to build that into Skybot without having to create a whole separate job stream to do that. So those are the three new things that we have in job suites and again, that's going to allow you to be more flexible depending on your requirements.

Breaking Down the Job Flow

So let's take a look at our job flow and now if I go way over here at the end. So here's my whole job flow and what I've done to it is I've broken this down into individual pieces. It's a little bit easier to look at if you don't have all of those lines and boxes. So I'm going to go over to the order payment chunk. So this is my job stream, we wait for these payments to come in. The payment processing jobs are dependent. If I look at the prerequisites for these guys, they're dependent on that payment occurring. As soon as that file comes in for a payment or a file for an order, these guys are going to run.

So when they run, like I said, many many times a day, hopefully lots of times a day because you are getting lots of orders and lots of payments, but my ERP, my accounting job, is scheduled at 18:00 if I've had some order or payments. If I haven’t had any, if I look at the prerequisites for this guy, if neither of these jobs run, then this accounting job won't run. So I've got these linked together with order statements, order status or payment status, and we're going to talk about month-end as well, okay? Which isn't showing up on here. So that's the first one, that's fairly straight forward. These guys run all day, this job just runs at 18:00, and then my ETL load, again, is dependent on that accounting job.

Let's take a look at CRM piece. So here I've got my CRM modules that are going to be running throughout the day, these are probably scheduled. They're going to run concurrently with the payment processing and they might run throughout the day but here's the guy that only runs on the 15th of the month or the 1st. So it's got a schedule. We take a look at this job and it's scheduled to run on the 1st and the 15th of the month. So in order to set that up, my ETL load is going to run. I do want when this does run on the 1st and the 15th to be a prerequisite for this ETL load. I don't want it to run unless that job has run. So here we can go back to special instances. Actually, if I look at the prerequisite here, I will see I've got all these prerequisites for this ETL load: here are my CRM analytics needs to be completed, the channel need to be completed on the 1st or the 15th of the month. When you set up a prerequisite you get this box which defaults to any time, but I only want it to run on the 1st or the 15th. So those other days that the CRM channel job doesn't run, it doesn't bother my prerequisites.

The way you set up that special instance up is that it's over on my CRM channel job, I create a special instance and tie it to that job and, like I said, just like a little calendar, I can create a day of the week or a specific date list or day of period type and add those days to it. So that's one way you can build that schedule and build those prerequisites even if your prerequisite job doesn't run every day. Let's get back to my job flow diagram.

The other thing I've got here, the CRM analytics job, sometimes it fails and so if it completes successfully, I'm going to go this way and I'm going to run ETL load. If it fails, I'm going to come down this way and I'm going to trigger a recovery job to run. So if you do have some type of a recovery that you know what the error is, you can go ahead and run that. So I'm reacting to that failure and then the other thing I'm doing in this recovery job is that I'm using our web services command to run that analytics job again. So the first command in this job is my recovery, whatever that executable is, and then the second one is our web services.

Manipulating Agents in Skybot

So we ship this with the Skybot product, job service .bat or agent service, so you can manipulate agents as well. The parameters for that—the config file—these are my credentials for running this analytics job. I can put those in a file that's secured. The type of action that I'm going to perform or the type of process is a job. I'm going to run it and then this is the job I'm going to run. So as soon as that recovery step is finished, then I'm going to go ahead and I'm going to trigger this job to run again. You don't have to have a separate job for that. So that's a great use of our web service commands.

Adjusting for Month-End

Let's take a look then over at the ERP side. So now we've got our accounting that runs every day and here our order payment processing jobs are up here, but on month-end, I want us to run at a different time. Most of the times this job runs at 18:00 every night, but for month-end I want to run it earlier because of the month-end processing that it needs to do. So I've got a job that will do a couple of things: the first thing that it will do and this is the job, so this is my dependency that runs at 16:00 and the commands that it uses. The first one is it clears those prerequisite statuses. So if there are payments or ordered jobs that have triggered throughout the day, what I want to do is I want to run this accounting job, but I want clear those prereqs because, when 18:00 rolls around, I don't want it to run again. I just want it to run at 16:00. So I'm going to clear those prereqs and then I'm going to use that same job service command to go ahead and run this ERP accounting job. So that's going to kind of change up that schedule for month-end without having to have a whole separate slew of jobs actually to run, or separate schedule for this job.

Now I do have some dependencies, my ERP planning job. This job runs most days but I think I've got an exception in here if we take a look at the schedule. I don't run that ERP planning job on the last day of the month because of month-end and so what I've done is I've created a list of omit dates and tied that to this job. Now this job will just not run. It will be skipped and I've still got my prerequisite set correctly for this ETL load because it will run when this is skipped or when it is complete. So if we take a look at these prerequisites, this is the one that's a little crazy. Here are all my CRM jobs, all those need to complete, or this ERP accounting status job, and the ERP planning, or if the ERP planning is skipped. So you can group all those prerequisites together for whatever your job stream requires.

Uploading Files to SharePoint with AutoMate

The other thing I was going to do, I'm just going to show you the SharePoint quickly. So when the ETL finishes, actually I will go to the big guy here. So if I go to this job stream, now I've got all of those pieces in here: my order processing, my CRM over here, my ERP, and here's my ETL load that's getting fed in from all of these. My output is going to run another sweep that's running a bunch of reports and it's going to upload some files to SharePoint.

So in order to do that we've integrated with our AutoMate product, and if we look at the command, we've got a new option that you can see here, an AutoMate task or an AutoMateBPA. So it depends on the version of AutoMate that you're running, along with all of our other integrations, and I just pull a task from the AutoMate server and run that and this is my SharePoint task. What that looks like in AutoMate is this. So this is the AutoMate user interface and over here on the left side you can see I've got all kinds of different SharePoint actions that I can run. This creates a session so it logs into SharePoint with the correct credentials and then it uploads the file, this is my test file that I'm uploading into SharePoint and then it just ends that SharePoint session.

Conclusion

So if I run this from here, it will show you the steps as well. So I can run it from AutoMate or I can run it as a Skybot job. So it can be part of my business process along with all of the other processes that I need to run. That's kind of a quick view. I've got a bunch of questions that have rolled in so hold on just a second and I will get to those. Hopefully you can see that if your job streams have alternate paths you can still schedule them in Skybot, whether it's a weekly change, an error, a monthly change. Between Skybot and AutoMate, you should be able to have all of those jobs run smoothly and as they should, no matter what day of the week it is or what day of the month it is.