Version
13.5

Environment Optimization

Manager+Agents 13.5 introduces new opt-in features aimed at improving overall performance of your Manager including:

  • New REST interface architecture
  • New scheduler architecture
  • Settings to improve resource controlled job performance
  • Database optimizations

These improvements address issues related to performance based on job performance and overall processing load when using the Manager. These changes strongly improve Manager performance, but must be manually enabled via your configuration files.

Enabling the new REST interface architecture

By default, using the REST API relies on your custom integration to manage loads on the REST interface. Prior to 13.5, jobs would occasionally fail due to lack of available resources in the REST interface.

With the new interface, the REST API queue uses a Token Bucket algorithm to balance requests against a set number of resource tokens.

You can now execute over 100 concurrent REST requests with minimal impact to overall performance by enabling the new feature.

Manager 13.5 Job Execution Performance

To enable the new REST interface:

  1. Create a backup of your signiant.ini file.
  2. Open signiant.ini and add the following lines to the file:

    # Token Bucket Architecture Opt-in
    BUCKET_CAPACITY=500
  3. Save and close your signiant.ini file.
  4. Restart Manager's JBOSS service using a service manager.

Enabling the new Scheduler architecture

Job Evaluator improvements have been made to allow for better performance for large databases with a high amount of job runs.

This change dramatically improves responsiveness to tasks with a high level of concurrency and increases performance from within the Manager application, especially with regard to displaying large amounts of data at a time.

Manager 13.5 Job View Load Time

To enable the new scheduler architecture:

  1. Create a backup of your signiant.ini file.
  2. Open signiant.ini and add the following lines to the file:

    # Java Evaluator Opt-in
    SCHDSVR_EVALUATE_JOBS=no
  3. Save and close your signiant.ini file.
  4. Restart Manager's JBOSS service using a service manager.

Optimizing Interface Performance for Resource Controlled Jobs

As part of its running environment, Signiant Manager uses a run control file to set how the background Java Virtual Machine (JVM) runs according to your /usr/signiant/dds/init/.siginitrc settings file.

If you are running a high number of jobs and encounter an OutOfMemory error, you can increase the memory allocation pools for your JVM and rule server JVM.

To increase your JVM memory allocation:

  • Set JAVA_JBOSS_SERVER_OPTS to -Xms1024m -Xmx4096m
  • Set JAVA_DB_SERVER_OPTS to -Xmx2048m

Note: Do not modify any other settings in the .siginitrc file unless otherwise instructed.

High Concurrency

Manager installations include command line tools that allow you to send override commands to an Agent's configuration using the dds_admin application.

Increasing Resource Control Timeout

When submitting a large number of jobs concurrently, or with a very small interval between them, you can prevent Agent based resource controls causing a timeout error by increasing the timeout value for the resource control service.

Configure Agent timeout values by setting the configuration via the command line:

> dds_admin -one_shot "set RESBRKR_RESPONSE_TIMEOUT 60" customer*Admin agent.example.com

Note: You can also run set RESBRKR_RESPONSE_TIMEOUT 60 from the command line on the Agent.

Increasing Port Range

If you expect to run more than 100 jobs at a time, you must to increase the inbound and outbound port ranges from 100 to 300 to allow your Agents to handle higher capacity.

Configure the UDP port range by setting the configuration via the command line:

  1. Log into the Manager via command line.
  2. Run dds_admin.
  3. At the dds_admin prompt, set the UDP destination port range:

    >set UDP_DEST SIZE=300
  4. Set the UDP origin port range:

    >set UDP_ORIG SIZE=300
  5. Quit dds_admin by entering the q command.

Note: You may also need to work with your firewall administrator to allow access through additional ports as needed (e.g. from 49221-49321 to 49221-49531).

Database Optimization

Signiant Manager uses a PostgreSQL server as part of its operational environment, and is configured for maximum compatibility for system requirements. If your system surpasses the minimum requirements, you can optimize your database for higher capacity.

If your workflow does not require data retention, you can choose to automatically delete jobs from your database to increase overall performance.

Increasing Database Operability

Increasing your system's overall capacity to connect and interact with the database can be useful when working with a large number of jobs under resource control.

To increase the number of concurrent jobs you can run at the same time, you must increase your database's maximum active connection count by adding the following line to your signiant.ini:

DBMAXACTIVECONN=150

Additional database settings can be modified using the /usr/signiant/dds/db/pgsql/conf/postgresql.conf file.

The Connections and Authentication section of the configuration file contains the max_connections setting.

  • For optimal performance, increase your max_connections to 400.

The Lock Management section contains settings to manage a database's ability to preserve database values while in operation.

  • For increased performance, set max_locks_per_transaction to 2000.

The Resource Usage section controls how your database uses system memory to use when performing database operations.

  • Depending on your system's available resources, you can increase the shared_buffers to 1024MB.

Note: Large-scale organizations with sufficient hardware capacity can increase the max_locks_per_transaction setting up to 40960, and shared_buffers up to 8026MB.

Automatic Job Deletion

To preserve your database's performance, you can choose to automatically delete jobs once they have run by adding parameters to your signiant.ini file. You can choose to remove jobs using soft or hard deletion.

A soft deletion removes the job from the Manager, but allows you to query the job history directly from the database before it is removed by a maintenance job.

A hard deletion immediately removes the job record from the database, including its history and record of files included in the job.

To remove a job from the database after it is successfully completed, set SCHDSVR_JOBOPT_AUTODELETE_IF_SUCCESS in your signiant.ini file to your desired level:

  • SCHDSVR_JOBOPT_AUTODELETE_IF_SUCCESS=off - Do not automatically delete jobs.
  • SCHDSVR_JOBOPT_AUTODELETE_IF_SUCCESS=soft - Automatically delete jobs from the Manager after a successful completion, and queue them for removal from the database during the next maintenance job.
  • SCHDSVR_JOBOPT_AUTODELETE_IF_SUCCESS=hard - Immediately remove jobs from the database after a successful completion.

To remove a job from the database regardless of its completion status, set SCHDSVR_JOBOPT_AUTODELETE in your signiant.ini to your desired level:

  • SCHDSVR_JOBOPT_AUTODELETE=off - Do not automatically delete jobs.
  • SCHDSVR_JOBOPT_AUTODELETE=soft - Automatically delete jobs from the Manager after a job run, and queue them for removal from the database during the next maintenance job.
  • SCHDSVR_JOBOPT_AUTODELETE=hard - Immediately remove jobs from the database after a job run.

Post-Upgrade Optimization

During a version upgrade, some field values are preserved, while others are overwritten. Settings that are overwritten must be re-applied to maintain optimization.

All Signiant.ini settings are preserved on upgrade.

All Postgresql.conf values are overwritten, with the exception of those in three fields:

  • listen_addresses
  • log_directory
  • max_locks_per_transaction

All other customization must be re-applied after upgrade.

About SigniantSigniant’s intelligent file movement software helps the world’s top content creators and distributors ensure fast, secure delivery of large files over public and private networks. Built on Signiant’s patented technology, the company’s on-premises software and SaaS solutions move petabytes of high-value data every day between users, applications and systems with proven ease.LEARN MORE