Batch Processing Systems: How They Started and Evolved

BATCH PROCESSING: THE EARLY DAYS

The earliest batch processing systems were developed in the days of mainframes and punch cards. Programs were entered on the punch cards, and system operators would have a stack, or batch, of the cards to feed into the computer. Modern batch processing software is quite a bit more sophisticated, but still involves managing the execution of multiple processes.

Batch Processing Systems Have Evolved

Batch job scheduling has changed in the past 20 years, but it isn’t dead. Data Entry Clerks are no longer needed and many transactions are processed in real time. The flow chart above has two branches with different timing needs.  The right branch should process immediately, whereas the left branch has time sensitivities, including daily and monthly reporting requirements. Both are accomplished using a modern workload automation (WLA) tool and event-driven scheduling. This batch job system optimizes efficiency because processes run as soon as resources become available.

Do You Need Batch Processing Software?

Look for the gaps and delays in your job flows. This is where good job scheduling software can streamline your business processes. Some examples include:

  • Manual processes: How do you ensure that they will get submitted on time or in the correct order?
  • Waiting for a previous process to complete: How do you know when it will complete?
  • Waiting for files to change or new files to arrive: How often is your script looping? Are you manually checking for new files?
  • Jobs set up with retries: How much overhead is that putting on the server?
  • Dependencies across servers: Do you know if the server is available?

Embrace Modern Batch Processing Systems

These days batch schedulers have a few features that are essential to the modern enterprise. To save time and resources, make sure your batch processing application supports dependencies, monitors, and notifications.

Dependencies:

What triggers the next step in your job flow? It could be a file arriving from a client or an online transaction. As soon as that file is created, the next step needs to run without delay.

The completion of another process or job on a local or remote server is a common prerequisite for jobs. Some run sequentially and some branch according to the requirements of the process.

Services or processes starting or ending may trigger some type of error recovery process or the next step.

Because each business division will look for the best solution for its needs, you may have different applications running on many different operating systems or hardware. Your workload automation tool needs to be compatible with all of those different operating systems and make sure dependencies across those different platforms are supported.

Monitors:

Monitors look for the events that occur on your servers and trigger the next step in the process. Monitors might detect new files that are created by another process or pulled from an FTP site. Once these files are created, the next step should be triggered immediately.

Besides monitoring for prerequisites, your batch scheduler needs monitors for delays and errors. If a process runs late because it’s waiting for a file or another resource, and it goes unnoticed, you could miss an SLA. SLAs are critical to the customer service you provide to your clients. If you miss an SLA, you may end up missing the client as well.

Your workload automation tool must include the following types of monitoring capabilities:

  • Files and directories
  • Processes
  • Underruns, overruns, and late starts
  • Errors
  • System availability
  • Notification

Notifications:

If a problem does arise or a job stream is delayed, your workload automation tool must have notification options so a small problem doesn’t evolve into a large one.

In addition to notification options for job overruns, underruns, and late starts, your batch scheduling system should include notification for the following events:

  • File events: Do your sales reps want to know about a new order?
  • Job failure: You should also receive job logs for troubleshooting.
  • Server down: System Admins need to know before users.
  • Process or service down: Your WLA tool should try to restart as well as notify you of the problem.

Management by Exception is Critical to Survival

All of the above options—supporting dependencies, monitoring systems, events, and job streams and notifying the correct staff of these problems—allow you to manage your systems by exceptions.

You must have confidence in your workload automation tool that it will run jobs when they are ready, and notify the correct people if there are problems. You shouldn’t have to log in to your system to make sure things are running smoothly. Enjoy your weekends and nights instead.

Get Started

See how a batch processing system can automate your jobs with a demonstration of JAMS.

REQUEST A DEMO