Workflow Gateway Overview

Signiant Workflow Gateway is designed as a two-stage DropBox application. At the first stage, Workflow Gateway collects content placed in a monitored DropBox folder. When file readiness checks determine that all referenced content is available, Workflow Gateway then kicks off a secondary job template, such as MediaDistributor, for the defined content.

Workflow Gateway gives customers the ability to control how many jobs are launched for a specified source path, which can benefit customers using job-group resource controls.

Each Workflow Gateway job can include a set of files or folders to limit the number of files processed on each scheduled run. The content can also be specified using configuration files placed in the DropBox root folder. Depending on the chosen workflow launch mode, the secondary job is run for each top-level file, top-level folder, individual file, or job definition file. Multiple workflow jobs are run in parallel, ensuring that processing is not delayed by any incomplete jobs.

Each DropBox folder associated with a Workflow Gateway Monitor has a structure of folders created the first time the monitor workflow is run.

workflow gateway flowchart

Note: Queuing job groups ensures optimal Manager performance when using Workflow Gateway.

Initiating workflows directly allows you to run jobs without embedding passwords in scripts or configuration files. The default user and password to launch the workflow jobs is configured as part of the Workflow Gateway Monitor job. This applies unless a username and password other than the default configured for the Gateway Monitor is required for a given job.

Workflow Gateway is distributed as an application zip file. Workflow Gateway and the WorkflowGatewayMonitorAndLaunch component are licensed under the Workflow Developer Toolkit license key.

Workflow Gateway DropBox Monitor Workflow

The Workflow Gateway DropBox Monitor workflow is contained in the Workflow_Gateway_DropBox_Monitor job template.

The workflow consists of three components:

Workflow Gateway Component Diagram

WorkflowGatewayDropBoxMonitor start component: Provides the prompt interface for defining the DropBox monitoring jobs.

WorkflowGatewayMonitorAndLaunch command component: Processes all DropBox folder content. For details, see Launch Mode Behavior.

FormulateLongRunningJobsEmail​: Determines if any launched jobs have crossed the long running job threshold and assembles an email notification listing the jobs in a long running state.

LongRunningJobsNotification​: Delivers the long running job notification email to interested parties.

MonitorReport job report component: Generates and sends a job report if the WorkflowGatewayMonitorAndLaunch component has reported an error. This report is generated when the component fails to run, or when jobs cannot not be created or started. For successful jobs, job-specific notification is the responsibility of the workflow run in the given job.

Launch Mode Behavior

Each time the Workflow Gateway Monitor is run, the WorkflowGatewayMonitorAndLaunch component performs operations.

In all launch modes, when all source content is ready, processed jobs are moved from the DropBox folder to the job-specific work folder unless you choose to have processed jobs remain in the DropBox folder indefinitely or only until successful completion. To set the DropBox folder content preservation mode, see DropBox Monitor Options.

Note: The launch mode behavior described in this section applies when Preserve DropBox Folder Content is set to Never - Move content to work folders on the Advanced tab.

When workflow launch mode is per top-level file or individual file (or per top-level file and top-level subfolder)

  • Checks file readiness of each file.
  • If the file is ready, creates the job-specific working folder under the _Active subfolder and moves the file to the job-specific subfolder.
  • If the file is not ready, leaves it in place.

When workflow launch mode is per top-level subfolder (or per top-level file and top-level subfolder)

For each subfolder in the DropBox:

  • Determines the list of files in the folder.
  • Checks file readiness of all files in the folder.
  • If all source content is ready, creates the job-specific working folder under the _Active subfolder and moves the entire folder to this job-specific subfolder.
  • If all source content is not ready, leaves the folder and its contents in place.

Note: Workflow launch mode ignores system-specific folders.

When workflow launch mode is per job definition file

  • Scans for job definition files. These have the .siggateway.json ​suffix.
  • For each job definition file:

    • Determines the list of referenced source content in the SourceContent bundles.
    • Checks file readiness of all files in SourceContent bundles.
    • If all source content is ready, creates the job-specific working folder under the _Active subfolder and moves all relative-referenced files to the job specific subfolder.
    • If all source content is not ready, leaves the job definition file in place but marks it as potentially stale.

    Note: Job definition files that are stale for five days or more are moved to the _Stale folder, along with any of their existing relative-referenced files. Absolute-referenced files are not moved.

For all job-specific folders under the _Active folder

  • If no job exists for the given folder, creates and runs one.
  • If a job exists for the given folder but has ended in error, moves the folder to _Retry.
  • If a job exists for the given folder and has ended successfully, moves the folder to _Success.

For all job-specific folders under the _Retry folder whose retry times have been reached

  • If no job exists for the given folder, creates and runs one. It may have been accidentally deleted.
  • If a job exists for the given folder but has ended in error, increments the retry counter. If the last retry has not been reached, re-runs the job. If all retries are exhausted, moves the folder to _Errored.
  • If a job exists for the given folder and has ended successfully, moves the folder to _Success.

For all content in the _Success, _Retry and _Errored folders

DropBox Subfolders

This structure of subfolders is located either in the DropBox folder or in another specified Job Status Folder Location.

_Active: Contains one subfolder for each job that has yet to be run or has only been run once and has not completed processing. If the content was defined in a job definition file, that file will have been moved from the DropBox root folder to the job name subfolder also. The job name subfolder name will match the job name exactly.

_Retry: Contains one subfolder for each job that failed its initial run but has not exceeded the maximum number of retries. Job subfolders and their contents get moved from _Active to _Retry after the first failed job run. The job name subfolder name will match the job name exactly.

_Errored: Contains one subfolder for each job that failed after the maximum number of retries. Job subfolders and their contents are moved from _Retry to _Errored after the final failed job run. The job name subfolder name will match the job name exactly.

_Success: Contains one subfolder for each job that has completed successfully. Job subfolders and their contents get moved from either _Active or _Retry to _Success after a successful job run. The job name subfolder name will match the job name exactly.

_Stage: Contains a staging area where files can be placed in preparation for processing. All content in this folder is ignored by the Monitor job. If a large set of files is going to take significant time to write to the DropBox folder, write them to the _Stage folder and then move them all to the DropBox root folder at once, with the job definition file if applicable.

_System: Contains control files used by the Monitor to track the state of jobs. It may also contain the optional defaults.siggateway.json ​file .

_Stale: Contains the JSON job definition files that have not had their source content satisfied (all files/subfolders present and ready) within a configurable period.

Job Definition Files

This section outlines the job definition file used when the monitor workflow has been configured to use Job Definition File mode as well as the global job defaults JSON file, which is applicable to the per-file and per-subfolder workflow modes.

Job Definition File Mode

In Job Definition File mode, Workflow Gateway is driven via files with the naming convention ​<job_name>.siggateway.json​. These files contain all of the information required to define a job, although some of the optional fields can be specified as prompts to the Workflow Gateway Monitor job, e.g. User, Password, Job Group.

JSON Job Definition File Structure

{
        "SigniantWorkflowGatewayJSON" :
        {
                "Job" :
                {
                        "JobName" : "",
                        "JobTemplateLibrary" : "",
                        "JobTemplate" : "",
                        "JobUser" : "",
                        "JobPassword" : "",
                        "JobGroup" : "",
                        "Inputs" :
                        {
                                "Input" :
                                {
                                        "InputName1" : "",
                                        "InputName2" : "",
                                           ...
                                        "InputNameX" : ""
                                }
                        },
                        "SourceContent1" :
                        {
                            "Path" :
                            [
                               "/path1",
                               "/path2",
                               ...
                               "/pathX"
                            ]
                        }
                }
        }
}
ElementRequired/OptionalDescription
JobNameOptionalDefines the base for the job name. If no JobName element is specified, the job name is either derived from the <job_name>.siggateway.json​ file, in job definition file mode, or from the single file or subfolder name, in per-file/per-folder modes. A unique numbered suffix is added, as is any job name prefix defined in the Gateway Monitor.
JobTemplateLibraryRequiredThe job template library containing the workflow to launch.
JobTemplateRequiredThe name of the start component of the workflow to launch.
JobUserOptionalThe userid under which to create the job. If JobUser is not specified, the value set in the Gateway Monitor will be used.
JobPasswordOptionalThe password used to create the job. If JobPassword is not specified, the value set in the Gateway Monitor will be used.
JobGroupOptionalThe job group to create the job in. If JobGroup is not specified, the value set in the Gateway Monitor will be used.
InputRequiredEach input/prompt to the workflow for which a value is to be set requires an Input element, where the name attribute defines the name as it appears in the Show Web Services Info on the job template canvas. The value to set is the content of the element. Certain variables are supported as content values.
SourceContentOptionalFiles and/or folders on the source system which must be checked for existence and file readiness should be defined using the SourceContentN "bundles". Up to 10 such bundles are supported, numbered SourceContent1 through SourceContent10. Each of these bundles can have one or multiple Path elements. The given job is only scheduled for launch when all content identified in these bundles has been verified to exist and have passed file readiness checks. Each of these bundles can be substituted into the contents of an Input element using variable substitution values. SourceContent variables 1 through 10 are defined in the Supported Job Definition File Variables table.

Supported Job Definition File Variables

The Inputs section of the job definition file supports the use of certain variables as content values. These variables are evaluated by the Workflow Gateway Monitor and substituted with the appropriate values prior to job creation.

Variable NameDescription
%SourceAgent%Evaluates to the hostname of the source Agent, where the Workflow Gateway Monitor is running.
%SourceFolder%Evaluates to the job-specific folder, not the DropBox folder, where the source content referenced in the job definition file has been moved. This is effectively the working folder for each launched job.
%SourceContent1%Evaluates to a SigList format file list corresponding to the SourceContent1 bundle.
%SourceContent2%Evaluates to a SigList format file list corresponding to the SourceContent2 bundle.
%SourceContent3%Evaluates to a SigList format file list corresponding to the SourceContent3 bundle.
%SourceContent4%Evaluates to a SigList format file list corresponding to the SourceContent4 bundle.
%SourceContent5%Evaluates to a SigList format file list corresponding to the SourceContent5 bundle.
%SourceContent6%Evaluates to a SigList format file list corresponding to the SourceContent6 bundle.
%SourceContent7%Evaluates to a SigList format file list corresponding to the SourceContent7 bundle.
%SourceContent8%Evaluates to a SigList format file list corresponding to the SourceContent8 bundle.
%SourceContent9%Evaluates to a SigList format file list corresponding to the SourceContent9 bundle.
%SourceContent10%Evaluates to a SigList format file list corresponding to the SourceContent10 bundle.

Note: Variable names are not case-sensitive.

Relative vs Absolute Paths

The elements within each SourceContent1..10 bundle can be specified as relative paths or absolute paths. These path types are handled differently:

  • Relative paths, those with no leading drive letter or slashes, represent files and folders which have been placed directly in the Workflow Gateway Monitor’s root folder, i.e. the DropBox folder. Prior to the creation of the job, this relative content is moved to the job-specific subfolder, under _Active or _Retry. The job will be able to find the files since they will be in the job’s working folder. If the Preserve Subfolders for Individual Files option is selected, any relative files specified in the JSON manifest will have their relative subfolder structure maintained when passed to the secondary workflow.
  • Absolute paths, which are defined with a leading slash, leading drive letter, or UNC path, represent files and folders which are outside of the Workflow Gateway Monitor’s root folder, i.e. the DropBox folder. Unlike relative files, these files are not copied to the job-specific subfolder. Jobs will have the correct information for locating files referenced with an absolute path.

Sample Job Definition File

{
   "SigniantWorkflowGatewayJSON" :
   {
      "Job" :
      {
         "JobTemplateLibrary" : "Media_Mover_Workflows",
         "JobTemplate" : "MediaDistributor",
         "Inputs" :
         {
              "Input" :
              {
                 "MediaDistributor.Source.SourceAgent" : "%SourceAgent%",
                 "MediaDistributor.Target.SourceDirectory" : "%SourceFolder%",
                 "MediaDistributor.Source.SourceData" : "%SourceContent1%",
                 "MediaDistributor.Source.skipTopLevelFolders" : "No",
                 "MediaDistributor.Target.TargetAgents" : "target.agent.name",
                 "MediaDistributor.Target.TargetDirectory" : "C:\\Incoming",
                 "MediaDistributor.Transport.BandwidthThrottleByTimeOfDay" : "100000",
                 "MediaDistributor.Transport.BandwidthCeiling" : "100000",
                 "MediaDistributor.Schedule.timezone" : "GMT",
                 "MediaDistributor.NotificationAndLogging._sp_log_severity" : "1"
              }
         }
      }
   }
}

Per-File and Per-Folder Modes

In the per-file and per-subfolder modes of Workflow Gateway, there is no unique job definition file associated with each file or folder, as there is with the job definition file mode. The per-file and per-folder modes do still require the existence of the global job defaults file.

Global Job Defaults File

In job definition file mode, the global job defaults file can be used to minimize the amount of information that must be specified in the job definition files. The global job defaults file has exactly the same format as the job definition files and may contain any or all of the same information. Prior to job creation, the content of the global job defaults file is merged with each job definition file to create the job definitions.
Note: In the event that the same attribute appears in both the global job defaults file and a job definition file, the value in the job definition file has precedence.

The global job defaults file is optional when using job definition file mode, but is mandatory when using the per-file and per-folder modes. In per-file and per-folder modes, the global job defaults file is the only source for job definition parameters.

The global job defaults file is placed in the _System subfolder of the DropBox folder and must have the name defaults.siggateway.json.

Sample Global Job Defaults File

In this example, all of the job information, other than the actual SourceContent elements, has been specified.

{
     "SigniantWorkflowGatewayJSON" :
     {
         "Job" :
         {
             "JobTemplateLibrary" : "Media_Mover_Workflows",
             "JobTemplate" : "MediaDistributor",
             "Inputs" :
             {
                 "Input" :
                 {
                     "MediaDistributor.Source.SourceAgent" : "%SourceAgent%",
                     "MediaDistributor.Target.SourceDirectory" : "%SourceFolder%",
                     "MediaDistributor.Source.SourceData" : "%SourceContent1%",
                     "MediaDistributor.Target.TargetAgents" : "localhost.example.com",
                     "MediaDistributor.Target.TargetDirectory" : "C:\\Incoming",
                     "MediaDistributor.Schedule.timezone" : "GMT"
                 }
             }
         }
     }
}

Note: In per-file and per-subfolder modes, the %SourceContent1% variable is used to represent the single file or single folder being processed for the given job. The sample global job defaults file represents a complete configuration for the launch of a MediaDropbox job on a per-file or per-folder basis.

Sample Minimal Job Definition File

Using the example global job defaults file, the job-specific definition file only needs to define the SourceContent elements.

{
    "SigniantWorkflowGatewayJSON" :
    {
        "Job" :
        {
            "SourceContent1" :
            {
                "Path" :
                [
                    "/Folder1",
                    "C:\temp\test1.mov"
                ]
            }
        }
    }
}

Sample Effective Job Definition

In job definition file mode, the contents of the global job defaults file and job definition file are merged prior to job creation, resulting in effective JSON.

{
 "SigniantWorkflowGatewayJSON" :
 {
   "Job" :
   {
     "JobTemplateLibrary" : "Media_Mover_Workflows",
     "JobTemplate" : "MediaDistributor",
     "Inputs" :
     {
       "Input" :
       {
          "MediaDistributor.Source.SourceAgent" : "%SourceAgent%",
          "MediaDistributor.Target.SourceDirectory" : "%SourceFolder%",
          "MediaDistributor.Source.SourceData" : "%SourceContent1%",
          "MediaDistributor.Source.skipTopLevelFolders" : "No",
          "MediaDistributor.Target.TargetAgents" : "localhost.example.com",
          "MediaDistributor.Target.TargetDirectory" : "C:\\Incoming",
          "MediaDistributor.Schedule.timezone" : "GMT"
       }
     },
     "SourceContent1" :
     {
        "Path" :
        [
            "/Folder1",
            "C:\temp\test1.mov"
        ]
     }
   }
 }
}
About SigniantSigniant’s intelligent file movement software helps the world’s top content creators and distributors ensure fast, secure delivery of large files over public and private networks. Built on Signiant’s patented technology, the company’s on-premises software and SaaS solutions move petabytes of high-value data every day between users, applications and systems with proven ease.LEARN MORE