Thank you for your help. 
Yes, 5.1 is definitely out of service (and soon 5.2 will be too, if it is not already), therefore an upgrade to 5.3, ML6 (CSP3) would be a good idea.
Still there are a few points i'd like to mention as possible problems:
1) As far as i know the sshd daemon is started not on the commandline but via the system resource controller (SRC). Try to issue the command:
startsrc -s ssh
If there is a error message like "the ssh subsystem is not on file" you might not have executed the correct installation procedure
2) On AIX 5.3 (and on 5.1 too, IIRC) you need to install the rpm-package openssl.0.9.7g-1 prior to install ssh/sshd. Issue
rpm -q openssl
and look if the output is like "openssl-0.9.7g-1". If not this might be the source of your problems.
3) Have you checked if you have installed the correct package? There are *several* packages to be installed via installp, issue:
lslpp -l openss*
the output should be like:
# lslpp -l openss*
Fileset Level State Description
----------------------------------------------------------------------------
Path: /usr/lib/objrepos
openssh.base.client 4.3.0.5300 COMMITTED Open Secure Shell Commands
openssh.base.server 4.3.0.5300 COMMITTED Open Secure Shell Server
openssh.man.en_US 4.3.0.5300 COMMITTED Open Secure Shell
Documentation - U.S. English
openssh.msg.en_US 4.3.0.5300 COMMITTED Open Secure Shell Messages -
U.S. English
Path: /etc/objrepos
openssh.base.client 4.3.0.5300 COMMITTED Open Secure Shell Commands
openssh.base.server 4.3.0.5300 COMMITTED Open Secure Shell Server
I hope this helps
bakunin
PS: ah, and a final point: do NOT use bash to start daemons in AIX. Usually it doesn't matter, but why take the risk? On AIX Korn Shell is used throughout the system and you should stick to that.
bash-2.05a# startsrc -s ssh
0513-085 The ssh Subsystem is not on file.
bash-2.05a# rpm -q openssl
openssl-0.9.7l-2
bash-2.05a# lslpp -l openss*
lslpp: 0504-132 Fileset openssh-4.7p1 not installed.
lslpp: 0504-132 Fileset openssh-4.7p1.tar not installed.
lslpp: 0504-132 Fileset openssl-0.9.7l-2.aix5.1.ppc.rpm not installed.
lslpp: 0504-132 Fileset openssl-0.9.8g not installed.
lslpp: 0504-132 Fileset openssl-0.9.8g.tar not installed.
That means you haven't installed ssh correctly, as i have suggested. You find a complete HOW-TO about how to obtain and install SSH correctly on AIX 4.3.3, 5.1, 52 and 5.3 here.
good - met the requirements
Seems like you just have to install SSH correctly (that is: using AIX methods) and your problem should be solved.
bakunin
I downloaded the file openssh-4.1p1_51.tar.Z, I extracted its content and ran "inutoc"....
bash-2.05a# pwd
/tmp/openssh
bash-2.05a# ls
Customer_README openssh.msg.CA_ES openssh.msg.EN_US openssh.msg.HU_HU openssh.msg.ko_KR openssh.msg.ru_RU openssh.msg.ZH_CN
openssh-4.1p1_51.tar openssh.msg.cs_CZ openssh.msg.es_ES openssh.msg.it_IT openssh.msg.KO_KR openssh.msg.RU_RU openssh.msg.zh_TW
openssh.base openssh.msg.CS_CZ openssh.msg.ES_ES openssh.msg.IT_IT openssh.msg.pl_PL openssh.msg.sk_SK openssh.msg.Zh_TW
openssh.license openssh.msg.de_DE openssh.msg.fr_FR openssh.msg.ja_JP openssh.msg.PL_PL openssh.msg.SK_SK openssh.msg.ZH_TW
openssh.man.en_US openssh.msg.DE_DE openssh.msg.FR_FR openssh.msg.Ja_JP openssh.msg.pt_BR openssh.msg.zh_CN .toc
openssh.msg.ca_ES openssh.msg.en_US openssh.msg.hu_HU openssh.msg.JA_JP openssh.msg.PT_BR openssh.msg.Zh_CN
now, when I try to install it with "smitty install", it says:
#---------------------------------------------------------------------
# No installable software products were found on the media.
#---------------------------------------------------------------------
I tried with with "." from that directory and with absolute path /tmp/openssh/ .......
I specified "openss.base" and the installatione went succesfully but....
bash-2.05a# startsrc -s ssh
0513-085 The ssh Subsystem is not on file.
bash-2.05a# lslpp -l openss*
Fileset Level State Description
----------------------------------------------------------------------------
Path: /usr/lib/objrepos
openssh.license 4.1.0.1 COMMITTED Open Secure Shell License
openssh.man.en_US 4.1.0.1 COMMITTED Open Secure Shell
Documentation - U.S. English
openssh.msg.en_US 4.1.0.1 COMMITTED Open Secure Shell Messages -
U.S. English
lslpp: 0504-132 Fileset openssh-4.1p1_51.tar not installed.
lslpp: 0504-132 Fileset openssh.base not installed.
lslpp: 0504-132 Fileset openssh.msg.ca_ES not installed.
lslpp: 0504-132 Fileset openssh.msg.CA_ES not installed.
lslpp: 0504-132 Fileset openssh.msg.cs_CZ not installed.
lslpp: 0504-132 Fileset openssh.msg.CS_CZ not installed.
lslpp: 0504-132 Fileset openssh.msg.de_DE not installed.
lslpp: 0504-132 Fileset openssh.msg.DE_DE not installed.
lslpp: 0504-132 Fileset openssh.msg.EN_US not installed.
lslpp: 0504-132 Fileset openssh.msg.es_ES not installed.
lslpp: 0504-132 Fileset openssh.msg.ES_ES not installed.
lslpp: 0504-132 Fileset openssh.msg.fr_FR not installed.
lslpp: 0504-132 Fileset openssh.msg.FR_FR not installed.
lslpp: 0504-132 Fileset openssh.msg.hu_HU not installed.
lslpp: 0504-132 Fileset openssh.msg.HU_HU not installed.
lslpp: 0504-132 Fileset openssh.msg.it_IT not installed.
lslpp: 0504-132 Fileset openssh.msg.IT_IT not installed.
lslpp: 0504-132 Fileset openssh.msg.ja_JP not installed.
lslpp: 0504-132 Fileset openssh.msg.Ja_JP not installed.
lslpp: 0504-132 Fileset openssh.msg.JA_JP not installed.
lslpp: 0504-132 Fileset openssh.msg.ko_KR not installed.
lslpp: 0504-132 Fileset openssh.msg.KO_KR not installed.
lslpp: 0504-132 Fileset openssh.msg.pl_PL not installed.
lslpp: 0504-132 Fileset openssh.msg.PL_PL not installed.
lslpp: 0504-132 Fileset openssh.msg.pt_BR not installed.
lslpp: 0504-132 Fileset openssh.msg.PT_BR not installed.
lslpp: 0504-132 Fileset openssh.msg.ru_RU not installed.
lslpp: 0504-132 Fileset openssh.msg.RU_RU not installed.
lslpp: 0504-132 Fileset openssh.msg.sk_SK not installed.
lslpp: 0504-132 Fileset openssh.msg.SK_SK not installed.
lslpp: 0504-132 Fileset openssh.msg.zh_CN not installed.
lslpp: 0504-132 Fileset openssh.msg.Zh_CN not installed.
lslpp: 0504-132 Fileset openssh.msg.ZH_CN not installed.
lslpp: 0504-132 Fileset openssh.msg.zh_TW not installed.
lslpp: 0504-132 Fileset openssh.msg.Zh_TW not installed.
lslpp: 0504-132 Fileset openssh.msg.ZH_TW not installed.
I have no openssh server installed...??
aarghhhhhhh....
You need to install the openssh.base too. Use "smitty install_all" to get to the installation menu and specify "." as the installation medium/path (presuming your current directory is where the images are). Go to the field "Software to install" and select the packages you want to install (base, license and all the messages you want to have on the machine), then select "COMMIT software updates", "EXTEND file systems if space needed" and "ACCEPT new license agreements", deselect "PREVIEW only?", "SAVE replaced files" and "Preview new LICENSE agreements" (who'd read them, anyways?) and start the installation.
Read the output of installp and analyze it. Maybe you see prerequisite failures or something such, which prevent the installation.
Hope this helps.
bakunin
The problem is that I select the field "Software to install" and I press tab for a list. A window pops up and says "Use PF4 or Esc+4 to generate a list" but if I press F4 or Esc+4 doesn't list anything at all... I only can use "openssh.base" in that field.
Here's the output of the installation with "openssh.base" (it says it is already installed)...
Before command completion, additional instructions may appear below.
geninstall -I "a -cgNqwXY -V2 -J" -Z -d . -f File 2>&1
File:
I:openssh.base.client 4.1.0.1
I:openssh.base.server 4.1.0.1
+-----------------------------------------------------------------------------+
Pre-installation Verification...
+-----------------------------------------------------------------------------+
Verifying selections...done
Verifying requisites...done
Results...
WARNINGS
--------
Problems described in this section are not likely to be the source of any
immediate or serious failures, but further actions may be necessary or
desired.
Already Installed
-----------------
The number of selected filesets that are either already installed
or effectively installed through superseding filesets is 2. See
the summaries at the end of this installation for details.
NOTE: Base level filesets may be reinstalled using the "Force"
option (-F flag), or they may be removed, using the deinstall or
"Remove Software Products" facility (-u flag), and then reinstalled.
<< End of Warning Section >>
FILESET STATISTICS
------------------
2 Selected to be installed, of which:
2 Already installed (directly or via superseding filesets)
----
0 Total to be installed
Pre-installation Failure/Warning Summary
----------------------------------------
Name Level Pre-installation Failure/Warning
-------------------------------------------------------------------------------
openssh.base.client 4.1.0.1 Already installed
openssh.base.server 4.1.0.1 Already installed
COMMAND STATUS
Command: OK stdout: yes stderr: no
Before command completion, additional instructions may appear below.
geninstall -I "a -cgNqwXY -V2 -J" -Z -d . -f File 2>&1
File:
I:openssh.base.client 4.1.0.1
I:openssh.base.server 4.1.0.1
+-----------------------------------------------------------------------------+
Pre-installation Verification...
+-----------------------------------------------------------------------------+
Verifying selections...done
Verifying requisites...done
Results...
WARNINGS
--------
Problems described in this section are not likely to be the source of any
immediate or serious failures, but further actions may be necessary or
desired.
Already Installed
-----------------
The number of selected filesets that are either already installed
or effectively installed through superseding filesets is 2. See
the summaries at the end of this installation for details.
NOTE: Base level filesets may be reinstalled using the "Force"
option (-F flag), or they may be removed, using the deinstall or
"Remove Software Products" facility (-u flag), and then reinstalled.
<< End of Warning Section >>
FILESET STATISTICS
------------------
2 Selected to be installed, of which:
2 Already installed (directly or via superseding filesets)
----
0 Total to be installed
Pre-installation Failure/Warning Summary
----------------------------------------
Name Level Pre-installation Failure/Warning
-------------------------------------------------------------------------------
openssh.base.client 4.1.0.1 Already installed
openssh.base.server 4.1.0.1 Already installed
but
startsrc -s ssh
0513-085 The ssh Subsystem is not on file.
bash-2.05a# lslpp -l openss*
Fileset Level State Description
----------------------------------------------------------------------------
Path: /usr/lib/objrepos
openssh.base.client 4.1.0.1 COMMITTED Open Secure Shell Commands
openssh.base.server 4.1.0.1 COMMITTED Open Secure Shell Server
openssh.license 4.1.0.1 COMMITTED Open Secure Shell License
openssh.man.en_US 4.1.0.1 COMMITTED Open Secure Shell
Documentation - U.S. English
openssh.msg.en_US 4.1.0.1 COMMITTED Open Secure Shell Messages -
U.S. English
Path: /etc/objrepos
openssh.base.client 4.1.0.1 COMMITTED Open Secure Shell Commands
openssh.base.server 4.1.0.1 COMMITTED Open Secure Shell Server
bash-2.05a# startsrc -s ssh
0513-085 The ssh Subsystem is not on file.
At this point, can it be really a bug...?
Did you upgrade your ML as everyone working this issue for you has suggested?
I have to admit i made a typo somewhere up this thread. The ssh service is called sshd and the group is ssh. So the command to start it is
startsrc -g ssh
or
startsrc -s sshd
The first one starts the group ssh, the scond one starts the single service sshd. I'd like to apologize. If both are not working, what is the output of
lssrc -a | grep ssh
bakunin
I just finished the upgrade, I still have to reboot but "startsrc -s sshd" goes well.... Hope that there will be no problem after reboot... 
Thank you to everyone involved...