I installed OpenSolaris 2009.06 on VirtualBox recently and I needed to login with a vnc viewer such as tightvnc remotely. You should note that the following is not the most secure way of accessing opensolaris since I am setting it with root.
The first thing you need to do is to edit your custom.conf file with the following settings:
# vi /etc/X11/gdm/custom.conf
The xvnc service needs to be enabled:
# svcadm enable xvnc-inetd
# svcs -a | grep xvnc-inetd
online 19:05:10 svc:/application/x11/xvnc-inetd:default
Set your vncpassword:
This would give you some kind of security.
If you want your sessions to be persistent, then you can edit xvnc-inetd service:
# inetadm -m svc:/application/x11/xvnc-inetd:default wait=”true”
Restart your xvnc-inetd service
# svcadm restart xvnc-inetd
You can also change your screen resolution with the following:
# inetadm -m svc:/application/x11/xvnc-inetd:default exec=”/usr/X11/bin/Xvnc -inetd -query localhost -once passwordFile=/root/.vnc/passwd -geometry 1280x800x24″
Just edit your -geometry flag according to your needs.
From this point on, all you need to do is to use your vnc viewer application to access the system remotely.