Hvrmaint

Contents

Name

hvrmaint - Housekeeping script for HVR on the hub machine.

Synopsis

hvrmaint [optfile] [-options]

Description

Command hvrmaint is a script for regular housekeeping of the HVR on the hub machine. The script can run on the hub machine and can be scheduled on Unix using crontab or as a Windows scheduled task. 

The hvrmaint can be used for three main purposes:

  1. Maintenance. Schedule hvrmaint nightly (or weekly) with options -stop and -start. These options instruct hvrmaint to restart the HVR Scheduler. Often other options can be used, such as -scan_hvr_out (scan log files for HVR errors) or -archive_dir (move old log files to archive directory). Option -email can be used to send an email with the status summary to operator(s). When used in this way, hvrmaint could be scheduled on Unix using crontab, and on Windows as a Windows Scheduled Task.
  2. Monitoring. Run hvrmaint frequently (e.g. every 15 minutes) with options -ping, -check_logfile_growth and -scan_hvr_out to check if the HVR Scheduler is running and to scan the HVR log files for errors. Running hvrmaint this way does not interrupt the HVR Scheduler. There is option -email_only_when_errors to send emails only if an error has occurred.
  3. Backup. The last way to use hvrmaint is as part of a larger nightly or weekly batch script, which halts all server processes (including the DBMS), does a system backup and then restarts everything again. In this case, hvrmaint would be called at the top of the batch script with option -stop (stop the HVR Scheduler) and would then be called again near the bottom with option -start (restart the HVR Scheduler).


Command hvrmaint cannot process log files containing more than 12 months of data.

The behavior of hvrmaint is controlled by an option file, which can be generated using the HVR Maintenance Tasks dialog available from the context menu of the Scheduler.

  1. Right-click the Scheduler node and select Maintenance Tasks from the context menu. The Maintenance Tasks dialog opens with the list of tasks on the left pane and configuration options on the right pane (see the description of options available for command hvrmaint below).
  2. Click the Add button at the bottom of the left pane to create a new task (option file). Type the name of the task and click OK.
  3. Select the required options, specify parameters for them, where needed and click Save.
  4. Click Run to run the hvrmaint script you created against the hub. You can click View Log to watch the output of the script.
  5. The time options on the bottom pane allow you to schedule the task to run at a specific time, namely at regular intervals, daily or weekly.

    Select Highest Privileges option to run the task with administrative permissions.


Options

This section describes the options available for command hvrmaint.

Parameter

Description

-task_name=task

Task name is used internally by hvrmaint to locate its option file and name its offset files. This allows different tasks defined in the GUI to have a different state. e.g. so that a when a task for one channel has processed today's files a different task for a different channel still remembers to process today's files.

-scan_hvr_out

Scan Scheduler log file hvr.out. Command hvrmaint writes a summary of HVR errors detected in this file to its output and to any emails that it sends.

-scan_channel=chn

Only scan for general errors and errors in specified channel(s) chn.

-scan_location=loc

Only scan for general errors and errors in specified locations(s) loc.

-scan_ignore=patt

Ignore log records which match specified pattern patt (can be regular expression).

-test_scheduler

Check that HVR Scheduler is actually running using hvrtestscheduler. If option -stop is also defined then this test is performed before the HVR Scheduler is stopped. If option -start is supplied then hvrmaint always checks that the HVR Scheduler is running using a test, regardless of whether or not option -test_scheduler is defined.

-check_logfile_growth

Check that logfile hvr.out has grown in size since the last time hvrmaint was run. If this file has not grown than an error message will be written.

-task_group=group

Task group allows different hvrmaint tasks to share the same state. So a nightly task that processes log files and gives a warning if the latency is >1 hour can use the same 'offset state' as a task that runs during the day which gives a warning if latency is >1 minute.

-latency_limit=dur

Check for replication latencies and consider jobs over the limit erroneous. Value for dur can be specified in one of the following formats:

  • N d or N day or N days - Indicates number of days
  • N h or N hour or N hours - Indicates number of hours
  • N m or N min or N mins or N minute or N minutes - Indicates number of minutes
  • N s or N sec or N secs or N second or N seconds - Indicates number of seconds
  • HH:MM:SS[.SSS] - Indicates time format with hour, minute, second, and millisecond
  • MM:SS[.SSS] - Indicates time format with minute, second, and millisecond

Milliseconds [.SSS] is ignored by this parameter.

-latency_channel=chn

Only check latencies of jobs in specified channel(s) chn.

-latency_location=loc

Only check latencies of jobs in specified location(s) loc.

-stop

Stops HVR Scheduler.

-start

Starts HVR Scheduler.

-quiesce_grace=secs

If jobs are still running when the HVR Scheduler must stop, allow seconds secs grace before killing them. The default is 60 seconds. This parameter is passed the HVR Scheduler using the -q option.

-archive_files=patt

Move any files in directory $HVR_CONFIG/log/hub with pattern patt to the archive directory. Files which are not matched by patt are deleted.

-archive_keep_days=N

Retain files in archive directory for N number of days. The retained files are deleted after N number of days. Requires option -archive_files. If this option is not specified, then archived files are kept indefinitely.

-archive_compress

Compress HVR Scheduler log files while moving them to the archive directory. For a Windows hub, this option can only be used if command gzip has been installed.

-journal_keep_days=n

Retain HVR journal files in directory $HVR_CONFIG/jnl for n number of days. The files are deleted after n number of days. These files are written by integrate jobs if parameter Integrate /JournalRouterFiles is defined.

-output_verbose

Prints latency information and error summaries to hvrmaint output.

-email_to=addr1[;addr2]

Email output from hvrmaint output to addr1 [and addr2]. Requires either option -smtp_server or option -mailer. Multiple email addresses can be specified, either using a single -email_to option with values separated by a semicolon or using multiple -email_to options.

-email_from=addr

Specify a sender address addr in email header.

-email_only_when_errors

Send an email if hvrmaint encountered an error itself or detected an HVR error while scanning hvr.out or the latency limit is exceeded.

-email_only_when_errors_or_warnings

Send an email if hvrmaint encountered an error itself or detected an HVR error or warning while scanning hvr.out or the latency limit is exceeded.

-error_limit=N

HVR errors reported is limited to N number. This option prevents the generated emails becoming too large. Default is 1000.

-smtp_server=server

SMTP server to use when sending email. Value server can be either a node name or IP address. Requires option -email.

-smtp_user=user

Username user for authentication SMTP server if needed.

-smtp_pass=pass

Password pass used for authentication on the SMTP server if needed.

-mailer=cmd

Mailer command to use for sending emails, instead of sending them via an SMTP server. Requires option -email. String %s contained in cmd is replaced by the email subject and string %a is replaced by the intended recipients of the email. The body of the email is piped to cmd as stdin. E.g. on Linux: -mailer=/bin/mail -s %s %a

-slack_webhook_url=url

A webhook for a Slack channel in company MyCorp looks like https://hooks.slack.com/services/xxxx/yyyy.

To generate such a webhook, sign into Slack, then navigate to Apps ▶ Manage apps ▶ Custom Integrations ▶ Incoming WebHooks ▶ Add Configuration.

-slack_channel=chn

Hvrmaint will send the message to the specified slack channel chn. You can also send the message to a user (@username).

-send_slack_only_when_errors

Send a slack message if hvrmaint encountered an error itself or detected an HVR error while scanning hvr.out or the latency limit is exceeded.

-send_slack_only_when_errors_or_warnings

Send a slack message if hvrmaint encountered an error itself or detected an HVR error or warning while scanning hvr.out or the latency limit is exceeded.

-snmp_notify

Send SNMP v1 traps or v2c notifications. The -snmp_community option is required. See $HVR_HOME/lib/mibs/HVR-MIB.txt

-snmp_version=vers

Specify SNMP version. Value vers can be 1 or 2c (default).

-snmp_heartbeat

Send a hvrMaintNotifySummary notification, even if there was nothing to report.

-snmp_hostname=host

SNMP agent hostname host. Defaults to localhost.

-snmp_port=port

SNMP agent trap port. Defaults to port 162.

-snmp_community=str

Community string str for SNMPv1/v2c transactions.

-disable

hvrmaint is running but alerts are not generated. This can be useful during a maintenance window when channels are being modified or stopped. An alternative is to stop running hvrmaint during the maintenance window and restart it after, but this can generate a lot of alerts caused by the maintenance.

-hub=hub

Hub database for HVR Scheduler. This value has form user/pwd (for an Oracle hub), inghub (for an Ingres hub database), or hub for a (SQL Server hub database). For Oracle, passwords can be encrypted using command hvrcrypt. In HVRGUI, this option is available inside the Text tab.

-sched_option=schedopt

Extra startup parameters for the HVR Scheduler service. Possible examples are -uuser/pwd (for a username), -hsqlserver (for the hub class) or -cclus/clusgrp (for Windows cluster group). In HVRGUI, this option is available inside the Text tab.

-output=fil

Append hvrmaint output to file fil. If this option is not supplied, then output is sent to stdout. Output can also be sent to an operator using option -email. In HVRGUI, this option is available inside the Text tab.

-env=NAME=VALUE

Set environment variable. This option can be repeated to set multiple variables such as $HVR_HOME, $HVR_CONFIG, $HVR_TMP, $II_SYSTEM, $ORACLE_HOME etc. In HVRGUI, this option is available inside the Text tab.

Examples

Unix & Linux

  • On Unix, hvrmaint could be scheduled to monitor the status of HVR every hour and also to restart the HVR Scheduler and rotate log files at 21:00 each Saturday. The environment for such batch programs is very limited, so many -env options are needed to pass it sufficient environment variables.

    Two option files are prepared. The first option file /usr/hvr/hvr_config/files/hvrmaint_hourly.opt will just check for errors and contains the following:

    -hub=hvr/!\{s8Dhx./gsuWHUt\}!			# Encrypted Oracle password 
    -sched_option=-h oracle
    -env=HVR_HOME=/usr/hvr/hvr_home
    -env=HVR_CONFIG=/usr/hvr/hvr_config
    -env=HVR_TMP=/tmp
    -env=ORACLE_HOME=/distr/oracle/OraHome817
    -env=ORACLE_SID=ORA817
    -email_from=hvr@prod.mycorp.com
    -email_to=bob@mycorp.com;jim@mycorp.com
    -email_only_when_errors
    -snmp_server=snmp.mycorp.com
    -output=/usr/hvr/hvr_config/files/hvrmaint.log
    -scan_hvr_out
    

    The second option file /usr/hvr/hvr_config/files/hvrmaint_weekly.opt will restart the HVR Scheduler and rotate the log files each week.

    -hub=hvr/!\{s8Dhx./gsuWHUt\}!				# Encrypted Oracle password 
    -sched_option=-h oracle
    -env=HVR_HOME=/usr/hvr/hvr_home
    -env=HVR_CONFIG=/usr/hvr/hvr_config
    -env=HVR_TMP=/tmp
    -env=ORACLE_HOME=/distr/oracle/OraHome817
    -env=ORACLE_SID=ORA817
    -email_from=hvr@prod.mycorp.com
    -email_to=bob@mycorp.com;jim@mycorp.com
    -email_only_when_errors
    -snmp_server=snmp.mycorp.com
    -output=/usr/hvr/hvr_config/files/hvrmaint.log
    -scan_hvr_out
    -stop
    -archive_files=hvr.out				# Only archive log file hvr.out
    -archive_compress
    -archive_keep_days=14		 	 	# Delete files after 2 weeks
    -journal_keep_days=4
    -start
    

    The following lines are added to crontab for user hvr (these should be single lines without wrapping):

    0 * * * * /usr/hvr/hvr_home/bin/hvrmaint /usr/hvr/hvr_config/files/hvrmaint_hourly.opt 
    0 21 * * * /usr/hvr/hvr_home/bin/hvrmaint /usr/hvr/hvr_config/files/hvrmaint_weekly.opt
    

    Alternatively the following line could be added to crontab for root:

    0 21 * * 6 su hvr -c /usr/hvr/hvr_home/bin/hvrmaint /usr/hvr/hvr_config/files/hvrmaint_weekly.opt
    

    Instead of scheduling hvrmaint on its own, it could also be used as part of a larger nightly batch script run by root which halts the HVR Scheduler and DBMS before doing a system backup. This batch script would roughly look like this:

    su hvr -c /usr/hvr/hvr_home/bin/hvrmaint /opt/hvrmaint.opt -stop -scan_hvr_out -archive_files=hvr.out<br />
    su ingres -c /opt/ingres/utility/ingstop 	# Stop DBMS
    backup -f/dev/rmt/0m				# Perform system backup<br />
    su ingres -c /opt/ingres/utility/ingstart 	# Restart DBMS<br />
    su hvr -c /usr/hvr/hvr_home/bin/hvrmaint /opt/hvrmaint.opt -start

Windows

  • On Windows, hvrmaint can be run as a Windows Scheduled Task.

    The configuration steps are as follows:

    1. Click Start ▶ Settings ▶ Control Panel ▶ Scheduled Tasks.
    2. Create a new Scheduled Task by clicking File ▶ New ▶ Scheduled Task.
    3. Double-click Add Scheduled Task.
    4. On the Tasks tab enter the hvrmaint command line in the Run field, for example:

      c:\hvr\hvr_home\bin\hvrmaint.exe 
      c:\hvr\hvr_config\files\hvrmaint_hourly.opt
      
    5. On the Schedule tab, configure when the hvrmaint script should run.
    6. When ready click the OK button.
    7. A dialog now appears requesting Windows account information (username and passwords). Enter this information as requested.

    A sample option file for hourly monitoring could be:

    -hub=hvr/!{s8Dhx./gsuWHUt}!			# Encrypted Oracle password 
    -sched_option=-h oracle
    -env=HVR_HOME=c:\\opt\\hvr_home
    -env=HVR_CONFIG=c:\\opt\\hvr_config
    -env=HVR_TMP=c:\\temp
    -env=ORACLE_HOME=c:\\distr\\oracle\\OraHome817
    -env=ORACLE_SID=ORA817
    -email_from=hvr@prod.mycorp.com
    -email=bob@mycorp.com;jim@mycorp.com
    -email_only_when_errors
    -snmp_server=snmp.mycorp.com
    -output=c:\\opt\\hvr_config\\files\\hvrmaint.log
    -scan_hvr_out<br /><br />
    

    A sample option file for weekly restart of HVR on Windows would be:

    -hub=hvr/!{s8Dhx./gsuWHUt}!				# Encrypted Oracle password 
    -sched_option=-h oracle
    -env=HVR_HOME=c:\\opt\\hvr_home
    -env=HVR_CONFIG=c:\\opt\\hvr_config
    -env=HVR_TMP=c:\\temp
    -env=ORACLE_HOME=c:\\distr\\oracle\\OraHome817
    -env=ORACLE_SID=ORA817
    -email_from=hvr@prod.mycorp.com
    -email=bob@mycorp.com;jim@mycorp.com
    -email_only_when_errors
    -snmp_server=snmp.mycorp.com
    -output=c:\\opt\\hvr_config\\files\\hvrmaint.log
    -scan_hvr_out
    -stop
    -archive_files=hvr.out				# Only archive log file hvr.out
    -archive_keep_days=14		 	 	# Delete files after 2 weeks
    -journal_keep_days=4
    -start

Sample Output

From: root@bambi.mycorp.com
To: bob@mycorp.com; jim@mycorp.com
Subject: hvrmaint detected 7 errors (323 rows in fail tables) for hub hvr/ on bambi
2017-11-01T21:00:01-06:30 hvrmaint: Starting hvrmaint c:\tools\hvrmaint.opt -hub=hvr/ -stop -start
2017-11-01T21:10:21-06:30 hvrmaint: Stopping HVR Scheduler 4.4.4/5 (windows-x64-64bit).
2017-11-01T21:10:33-06:30 hvrmaint: Scanning d:\hvr_config\log\hvr\hvr.out (2017-11-01T21:00:03-06:30).
2017-11-01T21:11:13-06:30 hvrmaint: 7 errors (323 rows in fail tables) were detected during scan.
2017-11-01T21:12:33-06:30 hvrmaint: 3 capture jobs for 1 location did 606 cycles.
2017-11-01T21:12:59-06:30 hvrmaint: 6 integrate jobs for 2 locations did 400 cycles and integrated 50 changes for 3 tables.
2017-11-01T21:13:53-06:30 hvrmaint: Archiving 9 log files to d:\hvr\archivelog\hvr_20050209.
2017-11-01T21:16:23-06:30 hvrmaint: Purging 0 archive directories older than 14 days.
2017-11-01T21:18:29-06:30 hvrmaint: Starting HVR Scheduler 4.4.4/5 (windows-x64-64bit).

-------------------- Summary of errors detected during scan-----------------
F_JD1034_RAISE_ERROR_P3 occurred 6 times between 2017-11-01T19:43:52-06:30 and 2017-11-01T20:14:24-06:30
F_JJ106E_TIMEO_DB occurred 1 time at 2017-11-01T21:10:03-06:30

-------------------------- Errors detected during scan---------------------------
 2017-11-01T21:10:03-06:30: hvrscheduler: F_JJ106E_TIMEO_DB: Wait for response from database slave exceeded the 600 seconds defined by 'timeo_db' attribute. Waiting...
 2017-11-01T19:43:52-06:30: channel-cap-d01: F_JD1034_RAISE_ERROR_P3: Error as raised by user during pl/sql procedure statement on Oracle SID.

----------------------- End of errors detected during scan -----------------------
 2017-11-01T21:19:01 hvrmaint: Sending e-mail to bob@mycorp.com; jim@mycorp.com

Files

HVR_CONFIG





log






hubdb




HVR Scheduler log files.



hvr.out




Main Scheduler log file.



.hvrmaint_state




hvrmaint state file.


logarchive





hubdb





YYYYMMDD





hvr.out




Archived Scheduler log file. These files are created if hvrmaint option -archive_files is defined and deleted again if option -archive_keep_days is defined.


hvr.out.gz




Archived Scheduler log file if -archive_compress is defined.