If you have installed the GTK2 bundle (as described in the installation guide, ISP 8.1.12, AIX 7.2) and probably updated the RPMs to a more recent level (that is not the problem) then you run into this problem:
GUI Installer/Instance Installer
There exists a library problem between AIX and AIX Toolbox libraries.
If you start the IBM Installation Manager GUI:
-----[/tmp/install.nnnnnnnn.log]---------------
00:00.53 ERROR [main] com.ibm.cic.agent.internal.ui.AgentUI reportStatus
Could not load SWT library. Reasons:
/daten/TSM/8.1.12.100/im32/configuration/org.eclipse.osgi/766/0/.cp/libswt-pi-gtk-4630.a (0509-022 Cannot load module /opt/freeware/lib/libgdk-x11-2.0.a(libgdk-x11-2.0.so.0).
0509-150 Dependent module /usr/lib/libXrender.a(libXrender.so.1) could not be loaded.
0509-152 Member libXrender.so.1 is not found in archive
0509-022 Cannot load module /daten/TSM/8.1.12.100/im32/configuration/org.eclipse.osgi/766/0/.cp/libswt-pi-gtk-4630.a.
0509-150 Dependent module /opt/freeware/lib/libgdk-x11-2.0.a(libgdk-x11-2.0.so.0) could not be loaded.)
swt-pi-gtk (Not found in java.library.path)
..... cut ....
-----[/tmp/install.nnnnnnnn.log]---------------
The IBM Installation Manager GUI does need libXrender.so.1 from linXrender.a.
This libary is provided by AIX and by the AIX Toolbox.
Normally the AIX library is the first library found in the library path.
The installer is unfortunately not linked against the correct library
(with an absolute path).
The ISP Server Initialization Wizard dsmicfgx does need the AIX version of the library.
Before you can start the ISP software GUI installer:
export LIBPATH=/opt/freeware/lib:$LIBPATH
./install.sh -g
Before you can use dsmicfgx:
export LIBPATH=
/opt/tivoli/tsm/server/bin/dsmicfgx -g
This is not what the user is expecting.
These problems could be avoided if the install.sh does include the correct LIBPATH to the AIX Toolbox. Then both installers (software/SIP instance) would work without any further user interaction.
It is also very bad, that the referenced GTK2 bundle destroys your installed AIX Toolbox by downgrading several already installed packages.
Currently the system is at AIX 7.2 TL5 SP2 level with the following RPM packages installed:
# rpm -qa
tcl-8.4.7-3.ppc
expect-5.42.1-3.ppc
bzip2-1.0.8-2.ppc
db-5.3.28-1.ppc
libiconv-1.16-1.ppc
pkg-config-0.29.2-2.ppc
pcre-8.44-1.ppc
sqlite-3.32.3-1.ppc
perl-5.30.3-2.ppc
xz-5.2.5-1.ppc
sqlite-devel-3.32.3-1.ppc
pcre-devel-8.44-1.ppc
make-4.2.1-5.ppc
libtextstyle-devel-0.20.2-1.ppc
db-devel-5.3.28-1.ppc
m4-1.4.17-1.ppc
libXrender-0.9.8-2waixX11.ppc
libgcc-6.3.0-1.ppc
libmpc-1.0.3-1.ppc
shared-mime-info-1.6-2.ppc
pixman-0.34.0-1.ppc
lzo-2.10-2.ppc
harfbuzz-1.2.7-2.ppc
pango-1.40.1-2waixX11.ppc
libgcc8-8.3.0-5.ppc
libstdc++8-8.3.0-5.ppc
gmp-6.1.2-1.ppc
libmpc-1.1.0-1.ppc
mpfr-devel-4.0.2-2.ppc
libgomp8-8.3.0-5.ppc
gcc8-cpp-8.3.0-5.ppc
gcc8-c++-8.3.0-5.ppc
gcc-8-1.ppc
libgomp-8-1.ppc
libstdc++-devel-8-1.ppc
bash-doc-5.1.4-1.ppc
expat-devel-2.2.9-2.ppc
ncurses-6.2-2.ppc
gettext-0.20.2-1.ppc
zlib-1.2.11-2.ppc
xz-libs-5.2.5-1.ppc
libpng-1.6.27-3.ppc
libjpeg-9d-1.ppc
libffi-3.2.1-3.ppc
info-6.7-1.ppc
libnghttp2-devel-1.41.0-1.ppc
jbigkit-2.1-1.ppc
libwebp-1.0.2-1.ppc
libwebp-tools-1.0.2-1.ppc
libtiff-devel-4.2.0-1.ppc
glib2-2.56.1-3.ppc
gdk-pixbuf-2.35.1-3waixX11.ppc
gdbm-devel-1.18.1-1.ppc
AIX-rpm-7.2.5.1-1.ppc
tk-8.4.7-3.ppc
df1000e21410f103-00012000040025700030-1.noarch
libtextstyle-0.20.2-1.ppc
libunistring-0.9.10-1.ppc
readline-8.0-2.ppc
slang-2.3.2-1.ppc
gdbm-1.18.1-1.ppc
python-2.7.18-1.ppc
xz-lzma-compat-5.2.5-1.ppc
slang-devel-2.3.2-1.ppc
readline-devel-8.0-2.ppc
xz-devel-5.2.5-1.ppc
libunistring-devel-0.9.10-1.ppc
bzip2-devel-1.0.8-2.ppc
sqlite-doc-3.32.3-1.ppc
libXcursor-1.1.14-1.ppc
libstdc++-6.3.0-1.ppc
mpfr-3.1.2-3.ppc
atk-2.20.0-3.ppc
freetype2-2.8-1.ppc
cairo-1.14.6-2waixX11.ppc
libXft-2.3.2-4waixX11.ppc
libgcc-8-1.ppc
libstdc++-8-1.ppc
mpfr-4.0.2-2.ppc
libstdc++8-devel-8.3.0-5.ppc
gmp-devel-6.1.2-1.ppc
sed-4.8-1.ppc
gcc8-8.3.0-5.ppc
gcc-cpp-8-1.ppc
gcc-c++-8-1.ppc
libmpc-devel-1.1.0-1.ppc
bash-5.1.4-1.ppc
expat-2.2.9-2.ppc
fontconfig-2.11.95-4.ppc
libxml2-2.9.10-1.ppc
gettext-devel-0.20.2-1.ppc
zlib-devel-1.2.11-2.ppc
ncurses-devel-6.2-2.ppc
libpng-devel-1.6.27-3.ppc
libjpeg-devel-9d-1.ppc
libffi-devel-3.2.1-3.ppc
libnghttp2-1.41.0-1.ppc
jbigkit-libs-2.1-1.ppc
jbigkit-devel-2.1-1.ppc
libwebp-devel-1.0.2-1.ppc
libtiff-4.2.0-1.ppc
libxml2-devel-2.9.10-1.ppc
glib2-devel-2.56.1-3.ppc
gtk2-2.24.30-3waixX11.ppc
This combination does work with the software installer (after setting LIBPATH) and with the instance configuration utility (after removing LIBPATH).
This should be forwarded to the ISP development support.
Please reopen with IBM Support. This is most likely a defect.