Description of Services
Consulting assistance

Oracle GoldenGate for Big Data (GGBD)

In order to use GoldenGate in a Golden Gate Hub Configuration

Assistance with creating additional Oracle GoldenGate for Big Data processes needed:

•  Manager
•  Extract
•  Replicat

• Assistance with architecture design of GGBD implementation with existing client Hub architecture
• Evaluation of existing environment, produce readiness document of any outstanding issues to prerequisite to implementation
• Develop prerequisite checklist to address prior to implementation
• Identification of Java components and all configuration/parameter files
• Assistance/Review of data movement plan
• Assistance with sample GGBD parameter files
• Assistance with test / development setup in existing Hub environment
• Assistance and identification/documentation  of components, files, parameters necessary  for setup of GGBD
• Assistance with installation of GGBD and associated processes through the GG CLI
• Assistance with any conflict detection and resolution
• Assistance and documentation of resiliency recipe
• Assistance with functional testing / implementation
• Setup of monitoring for GGBD
  •  Monitoring extracts to the point of transfer
  • Configuration of new replications
• Assistance/documentation with and for GoldenGate troubleshooting for GGBD

Assumptions

• Design should be consistent with the Maximum Availability Architecture
• All processes detailed above will be documented in detail, with examples that have been tested.

https://docs.oracle.com/en/middleware/goldengate/big-data/12.3.2.1/gbdin/understanding-java-adapter-and-oracle-goldengate-big-data.html#GUID-EDFEF2A7-ADBA-4B1A-87F1-AEFD2061AE9D

Oracle Big Data 12.3.2.1

Like extract altering, you can also alter the replicat process to reposition to start over replication process from a trail file, Along with reposition with specific extseqno , you can also reposition to specific date and time especially useful when you regenerate extract trail files or replication abended for some reasons.

Start Stop Replicat start replicat rep3

stop replicat rep3 Starting and Stopping Replicat Process

stop replicat r* stop/start replicat process or group that start with r*

start replicat r*

Kill Replicat kill replicat rep3 Kill the replicat process when the stop command does not respond

Lag Replicate lag replicat rep3 Shows how much lag replicat is from the extract.

From particular extract sequence
ALTER REPLICAT rep3, EXTSEQNO 53 Useful when you know which sequence the replicat stop and restart from that point

From particular RBA address
ALTER REPLICAT rep3, EXTRBA 0

ALTER REPLICAT rep3, EXTRBA 1001 Useful when the extrail is available and you want to read from particular place to
avoid duplicity of data

0 - means from header

n - means from particular that point.
From Particular date

ALTER REPLICAT rep3, BEGIN 2011-01-07 08:00:00

Reads and start replicate from particular date and time
Unregister Unregister replicat rep3 Rather delete if you want to unregister the replicat from database, so ti removes the information from the goldengate metadata.
Delete delete replicat rep3 deletes the replicat from process
cleanup cleanup replicate rep3 cleansup records from checkpoint table , does not delete the replicat

Similar to archive log clean up, one should also think about purging goldengate trail files.

By default, Goldengate manager does not purge them unless the purgeoldextracts parameter is not specified in the mgr param file.

Syntax for purging trail files

PURGEOLDEXTRACTS trail [, USECHECKPOINTS | NOUSECHECKPOINT ] [, MINKEEP_rule MINKEEPHOURS|MINKEEPDAYS|MINKEEPFILES]

So now the question is, "how often the manager purges the trail records", by default it is 10 MINS as per CHECKMINUTES default value. Even you do not specify this 10 mins is the default.

For example, in the following the manager process purges the old extracts.

The USECHECKPOINTS parameter tells manager process to check if trail file is already applied or not in Checkpoint table. Do NOT use NOUSECHECKPOINT, since it will delete all the trail files without considering the trail file.

.ggsci

./ggsci

info all

EDIT PARAMS MGR

### Add the bold line to your manager Param file ###

PORT 7809

USERID ggs_owner, PASSWORD ggs_owner

PURGEOLDEXTRACTS /data/ggate/dirdat/ex, USECHECKPOINTS

This is now GoldenGate for Big Data

#!/bin/ksh
############################################################################
# Script Name..: gg_common.ksh
# Description..: Common GG Hub functions for inclusion to all driver scripts
# Author.......: Michael Culp
# Date.........: 04/23/2010
# Version......: .65
# Modified By..: Michael Culp
# Date Modified: 06/07/2017
# Comments.....: Change this library to make mods to all functions
# Schema owner.: N/A
# Dependent on.: N/A
# Script type..: function library
# Function list:
############################################################################

############################################################################
#
############################################################################

gg_init()
##########################################################
# These variables should be set prior to setup
# Call this function anytime the variables need to be
# set for values that will be utilized in other functioons
# gg_init should be used to determine which environment
# needs to be setup
#########################################################
{

# Take a parameter for the instance
gginst=$1

# Do not change the following 2 variables
#########################################################
ggcomver=".92"
ggcomdt="10/18/2018"

# set -xv
# This will set the ggini file based on the hostname
#########################################################
initflnm=`hostname -a`".ggini"

echo "Display the init file name"
echo $initflnm

# Test to see if the is an init file first
# if so read it, otherwise set these variables

#   future enhancement check for various scripts standard names
echo
echo
echo "Checking for the existance of the ggini initialization file.."
echo

# if [ -f /oracle/admin/scripts/ggcommon/`hostname -a`.ggini ]; then
if [ -f /oracle/admin/scripts/ggcommon/`hostname -a`.gginix ]; then
   echo "....file exists...."
   echo
   echo " The ggini file was read ....Y"
   echo
   echo "....reading variables from the .ggini file....."
   echo
#   . ./`hostname -a`.ggini
   . /oracle/admin/scripts/ggcommon/`hostname -a`.ggini
else
   echo "....file does not exist"
   echo
   echo " The ggini file was not read ...."
   echo
   echo "....reading variables from the gg_init function....."
   echo
   #########################################################
   # Set the Oracle inventory location
   #########################################################
   orainv="/oracle/oraInventory"

   echo
   echo "Running gg_init function()......."
   echo

   #######################################################
   # GoldenGate operational directory
   # This needs to be set during build of additional
   # GoldenGate instances
   #######################################################
   # gg_dir="/oracle/product/gg123"
   # gg_dir="/oracle/product/gg123ms"
   # gg_dir="/oracle/product/gg122"

   # gg_dir="/oracle/product/ggvad001"
   # gg_dir="/oracle/product/ggvad001"
   # gg_dir="/oracle/product/ggvad001"
   # gg_dir="/oracle/product/ggtxd003"
   # gg_dir="/oracle/product/ggtxd002"
   # gg_dir="/oracle/product/ggtxd001"
   # gg_dir="/oracle/product/ggtxd006"
   # gg_dir="/oracle/product/ggtxd007"
   gg_dir="/oracle/product/ggvat004"

   # gg_dir="/oracle/product/gg121"

   export GGS_HOME=$gg_dir

   export OGG_HOME=$GGS_HOME

   #######################################################
   # GoldenGate version
   # We support GG 12.1, 12.2, 12.3 currently
   #######################################################
   ggver="12.3"
   # ggver="12.2"
   # ggver="12.1"

   # export DBFS_MNT=/dbfs

   #######################################################
   # GoldenGate source version
   #######################################################
   ggsrcver="12.1"

   #######################################################
   # ggflnm - is the zip file name of the GG distro
   # This is the name of the goldengate zip files
   #######################################################
   # File below is 12.3.0.1.2
   ggflnm="123014_fbo_ggs_Linux_x64_shiphome.zip"

   # File below is 12.3.0.1.2
   # ggflnm="123012_fbo_ggs_Linux_x64_shiphome.zip"

   # File below is 12.2
   # ggflnm="V100692-01.zip"

   # File below is 12.3 Micro Service Architecture
   # ggflnm="123010_fbo_ggs_Linux_x64_shiphome.zip"
   #######################################################

   #######################################################
   # ggdir_dist - directory of the distro zip file
   # This is the name of the directory where the distribution GoldenGate software is stored
   # Choose one and uncomment it
   #######################################################
   ##### GoldenGate ver. 12.3
   # ggdir_dist="/ora01/Oracle_SW/gg123/"
   ##### GoldenGate ver. 12.3.0.1.4
   ggdir_dist="/ora01/Oracle_SW/gg123/123014/"
   ##### GoldenGate ver. 12.3 MicroService Architecture
   # ggdir_dist="/ora01/Oracle_SW/gg123ms/"
   # ggdir_dist="/ora01/Oracle_SW/gg122/"
   # ggdir_dist="/ora01/Oracle_SW/gg121/"

   #######################################################
   # gginstdir - Location of the installer
   # GoldenGate installer directory
   #######################################################
   gginstdir=$ggdir_dist"fbo_ggs_Linux_x64_shiphome/Disk1"

   #######################################################
   # ggrspfldir - Response file directory
   # Response file directory
   #######################################################
   ggrspfldir=$ggdir_dist"fbo_ggs_Linux_x64_shiphome/Disk1/response"

   #######################################################
   # ggrspfl - response file name
   # Response file name for install of the GG binaries
   #######################################################
   ggrspfl="oggcore_mrc.rsp"

   #######################################################
   # XAG version
   #######################################################
   # xagver="7.1b"
   # xagver="8.1b"
   xagver="9.1"

   #######################################################
   # This is the name of the directory where the XAG software is stored
   #######################################################
   # xagdir_dist="/ora01/Oracle_SW/xag71/"
   # xagdir_dist="/ora01/Oracle_SW/xag81b/"
   xagdir_dist="/ora01/Oracle_SW/xag91/"


   #######################################################
   # XAG Ops directory
   # This is the operational directory for XAG
   #######################################################
   # xagdir="/oracle/product/xag_71"
   # xagdir="/oracle/product/xag_81b"
   xagdir="/oracle/product/xag_91"


   #######################################################
   # XAG home directory
   #######################################################
   XAG_HOME=/oracle/product/xag_91
   XAG_LOG=/oracle/product/xag_91/log/nodename


   #######################################################
   # This is the name of the xag zip file
   #######################################################
   # xagflnm="xagpack81b.zip"
   xagflnm="xagpack91.zip"

   # Read the GG environment files
   # . ./ggoraenv.env

fi

XAG_SD=/oracle/admin/scripts/ggcommon

CRS_SD=/oracle/admin/scripts/crsdbfs
DBFS_SD=/oracle/admin/scripts/clle_dbfscommon
GG_SD=/oracle/admin/scripts/ggcommon
ASM_SD=/oracle/admin/scripts/asmcommon
DG_SD=/oracle/admin/scripts/dgcommon


############################################################################
# Show the variable status dashboard,  comment this line to not display it
############################################################################
gg_init_dbrd

}

gg_init_dbrd()
########################################################
# This is a display dashboard that shows the values of
# Environmental variables
# used in various functions
########################################################
{

echo
echo "Change variables in gg_common.ksh gg_init...."
echo "Display with gg_init_dbrd function .........."
echo

echo
echo "..gg_common.ksh ver. $ggcomver Date $ggcomdt"
echo "....Oracle Inventory location................: " $orainv
echo "....GoldenGate Version.......................: " $ggver
echo "....GoldenGate Source Version supported......: " $ggsrcver
echo
echo "================= Installation Parameters ======================="
echo "....GoldenGate Filename......................: " $ggflnm
echo "....GoldenGate software distribution dir.....: " $ggdir_dist
echo "....GoldenGate Response File Dir.............: " $ggrspfldir
echo "....GoldenGate Response File ................: " $ggrspfl
echo "....GoldenGate Installer directory...........: " $gginstdir
echo "....GoldenGate Ops  Directory................: " $gg_dir
echo "....GoldenGate Home..GGS_HOME................: " $GGS_HOME
echo "....GoldenGate Home..OGG_HOME................: " $OGG_HOME
echo
echo "....XAG Version..............................: " $xagver
echo "....XAG Filename.............................: " $xagflnm
echo "....XAG Dist directory.......................: " $xagdir_dist
echo "....XAG Ops directory........................: " $xagdir
echo "....XAG Home directory.......................: " $XAG_HOME
echo ".......XAG scripts directory.................: " $XAG_SD
echo "....XAG Log Directory........................: " $XAG_LOG
echo
echo "================= DBFS Parameters =============================="

echo "================= Scripts Directoriess ========================="
echo ".......CRS.................: " $CRS_SD
echo ".......DBFS................: " $DBFS_SD
echo ".......GoldenGate..........: " $GG_SD
echo ".......XAG.................: " $XAG_SD
echo ".......DataGuard...........: " $DG_SD
echo "================================================================"

echo
echo "....LD_LIBRARY_PATH..........................: " $LD_LIBRARY_PATH
echo
echo "....PATH.....................................: " $PATH
echo

echo "Installation directories......................."
echo
ls -lt /oracle/product/


# echo "Show ldd for libraries........................."
# ldd -d extract
# echo "....end transmission..........................."

}

gg_cr_env()
###############################################################################
# Create the ggoraenv.env file
###############################################################################
{

#
# Environment
#

################################################################
# usedate=`date +%a`
################################################################
usedate=`date +%d-%h-%Y`

echo $usedate

echo
echo "...Creating ggoraenv_"$usedate".env...."
echo

# echo > ggoraenv.env
cat << EOS > ggoraenv_$usedate.env

# export ORACLE_HOME=/oracle/product/
# export ORACLE_SID=
# export LD_LIBRARY_PATH=

# GGS_HOME=/oracle/product/gg12.1
# GGS_HOME=/oracle/product/12.2/gg_2
# export GGS_HOME=/u01/app/oracle/product/12.2/gg_2
# export GGS_HOME=$gg_dir
# GGS_HOME=\$gg_dir
# export GGS_HOME=$gg_dir
export GGS_HOME=/oracle/product/gg122

export GG_SITE=01
# export GG_SITE=02
# export GG_SITE=03
# export GG_SITE=04
# export GG_SITE=05
# export GG_SITE=06
# export GG_SITE=07
export GG_LOC=TX
# export GG_LOC=VA

# if there is only one DB related to this GG home,
# set NEW_ORACLE_SID to avoid constant switch between a DB env and its GG env
# export NEW_ORACLE_SID=DBFSXD011; . ~/.std_profile

# otherwise, set NEW_ORACLE_SID to dummy if there are multiple replicated databases related to same GG home
# export NEW_ORACLE_SID=dummy; . ~/.std_profile

# uncomment next three lines if  ORACLE_SID is dummy
# export ORACLE_HOME=/oracle/product/11.2.0/db_1

export LD_LIBRARY_PATH=$ORACLE_HOME/lib/$JAVA_HOME/lib/amd64/server:$LD_LIBRARY_PATH
export PATH=$ORACLE_HOME/bin:$PATH

export PATH=$GGS_HOME:$PATH
export LD_LIBRARY_PATH=$GGS_HOME:$LD_LIBRARY_PATH

# for OEM12c GG monitoring
# export JAVA_HOME=/oracle/product/12.1.0/oem_1/agent/core/12.1.0.2.0/jdk/jre
# export PATH=$JAVA_HOME/bin:$PATH
# export LD_LIBRARY_PATH=$JAVA_HOME/lib/amd64/server:$LD_LIBRARY_PATH

TZ=UTC
export TZ

#############################################
## For GI Agent (XAG)
##############################################
# export XAG_HOME=/oracle/product/xag71
# export XAG_HOME=/oracle/product/xag81b
# export XAG_HOME=/oracle/product/xag91
# export PATH=$XAG_HOME/bin:$PATH

##############################################
# Change the prompt
##############################################
PS1="\\
\${PWD} \\
\${SNAME} [\${ORACLE_SID}] [GG_site$GG_SITE"_"$GG_LOC]-> "
export PS1

# alias ggsora12='. $HOME/.ggsora12_env;cd $GGS_HOME'
# alias ggsora12='. /ora_rman/ora013/mrc/gg/ggoraenv.env;cd $GGS_HOME'
# alias xag='. $HOME/.ggsora12_env;cd $XAG_HOME'
# alias ggstatus='$XAG_HOME/bin/agctl status goldengate piperd01_oggapp'

echo "GGS_HOME is : " $GGS_HOME

# export TMPLT=/oracle/product/11.2.0/db_1/assistants/dbca/templates
export TMPLT=/oracle/product/12.1.0/db_1/assistants/dbca/templates
export ADMSCR=/oracle/admin/scripts
export APXSCR=/oracle/admin/scripts/apex
export GGADMSCR=/oracle/admin/scripts/ggcommon
export APXADMSCR=/oracle/admin/scripts/apex

alias admscr='cd $ADMSCR'
alias apxscr='cd $APXADMSCR'
alias tmplt='cd $TMPLT'
alias ggadmscr='cd $GGADMSCR'

alias gi='cd /oracle_crs/product/11.2.0/crs_2/log/$HOST'
alias ocssd='cd /oracle_crs/product/11.2.0/crs_2/log/$HOST/cssd'
alias crsd='cd /oracle_crs/product/11.2.0/crs_2/log/$HOST/crsd'

GGS_HOME=/oracle/product/12.2/gg_2
alias gh='cd $GGS_HOME'
alias gd='cd /ggate'
#############################################
# For GoldenGate
#############################################
export GGS_HOME=/oracle/product/12.2/gg_2
export PATH=$GGS_HOME:$PATH
export LD_LIBRARY_PATH=$GGS_HOME:$LD_LIBRARY_PATH


EOS
}

 

autostart vs autorestart in goldengate

 

We are having goldengate setup 11.2.0.1.17 between oracle rac db of versions 11.2.0.4 on Linux platform.

We are trying to setup action script via ggsapp service to start/stop all processes automatically.

 

We had come across these 2 parameters but had some concern.

As mentioned in GG reference, AUTOSTART parameter to start one or more Extract and Replicat processes automatically when Manager starts.

AUTORESTART  parameter to start one or more Extract and Replicat process to be restarted by Manager after a failure

 

While testing, I stopped one of extract manually and it didn't come up automatically. This seems to correct for me

as autostart will take into effect only when manager starts and autorestart will work only when process gets failed.

There is no parameter which can bring up process automatically if its stopped manually other than some shell script with some code.

Please correct me.

Also will there any negative impact at any situation if we add autostart in mgr.prm

Here are some good GoldenGate scripts I have found

https://www.dropbox.com/s/grewqwmna0hlvl1/GoldenGate_Monitoring_Scripts_v1.06.zip?dl=0&file_subpath=%2Fgg_mon

GoldenGate 18c Download

Automated Trail File Recovery

GoldenGate Basic Config