Hudson, Our Build Server Host
Our primary build box is an OSX XServe; we do this so we can run OSX VMs (which, according to Apple, is only legal when your host operating system is OSX). We're running Hudson as our build engine.
You can access the Hudson web instance here: http://xserve.openqa.org:8080
By default, you have read-only access. But you can login with the username "openqa" and our shared secret "openqa" password. If you don't know it and you need it, talk to Patrick.
You should also be able to ssh into the xserve box. You'll need your own personal login name and password. Again, Patrick can set you up with a login; you should always use your own login/password (not a common/shared one) to ssh to the xserve machine.
With that said, there are two shared users.
- builduser: This is the user that runs the Hudson instance. This user has no password. You can "sudo su - builduser" to become builduser.
- admin: The administrator of the box; like root. You can access this user's password with "sudo cat /var/root/ADMIN_PASSWD". (Hopefully it's safe to tell you this, because if you've got sudo/root access, we're already screwed). Please be careful with this password; keep it secret, keep it safe.
Our VMs
The VMs are being run under VMWare, on the main GUI console of the OSX server. To administer the VMs, you need to access the machine's GUI console using VNC. This is a bit tricky, because it requires port forwarding over SSH (aka SSH tunneling).
TODO: Explain port forwarding and VNC
The xserve host is configured to respond to VNC requests on the standard port 5900, but only when VNC is initated from localhost. The VNC password is the same as the "admin" user password (above).
So one easy way to contact the host GUI consle is to run a command like this:
ssh -L 5910:localhost:5900 xserve.openqa.org
Then you can connect to "localhost:10" with your VNC client to see/manipulate the host GUI console. There, you can interact directly with all of the slaves using the VMWare Console.
But interacting with a VM through the VMWare GUI console over VNC is mind-numbingly slow; you'll want to set up additional forwarding rules so you can VNC into the slave VMs directly.
Our Slaves
(NOTE that these IP addresses are dynamic, and may change. If this documentation is wrong, you'll have to use the Host VNC to manually manipulate the machines to find their IP addresses.) (TODO: Can Hudson just tell us the IP addresses?)
- osx-vm-1
- 192.168.25.129:5900
- password is the "openqa" password
- Should be logged in as a local shared user named "openqa", whose password is the "openqa" password.
- ubuntu-vm-1
- 192.168.25.131:5900
- password is the "openqa" password
- Should be logged in as a local shared user named "openqa", whose password is the "openqa" password.
- xp-vm-1
- 192.168.25.128
- NO VNC, connect over RDP (aka Windows Remote Desktop, aka Terminal Services) on port 3389.
- username "Administrator", password is the "openqa" password
Here's a nice ~/.ssh/config entry
Host qax
Hostname xserve.openqa.org
# xp-vm-1 Remote Desktop
LocalForward 3399 192.168.25.128:3389
# osx-vm-1 SSH (port 1122) and VNC (display port 10)
LocalForward 1122 192.168.25.129:22
LocalForward 5910 192.168.25.129:5900
# ubuntu-vm-1 SSH (port 1123) and VNC (display port 11)
LocalForward 1123 192.168.25.131:22
LocalForward 5911 192.168.25.131:5900
# vista-vm-1 Remote Desktop (port 3400)
LocalForward 3400 192.168.25.130:3389
If you're on Unix/Linux, you can set up all of your forwarded ports like this:
ssh -L 3394:192.168.25.128:3389 -L 5910:localhost:5900 -L 5911:192.168.25.129:5900 -L 5912:192.168.25.131:5900 xserve.openqa.org
Then you can use use RDP to connect to localhost:3394, connect VNC to localhost:10 to access the host VNC, and connect to localhost:11 and :12 for the OSX and Ubuntu VMs respectively.
You can test that your port forwarding is working correctly by attempting to telnet to the appropriate ports.
Restarting Hudson
To restart Hudson, ssh in to the xserve as yourself, then "sudo su - builduser". The builduser's home directory contains "hudson.sh" which will launch Hudson in the background under "nohup". You might want to kill Hudson first. Find its process id using "ps -eaf | grep hudson", then kill the process id with "kill -9".
If the slaves are still up, they will automatically reconnect to the master Hudson instance when it restarts. If the slaves are down, you may need to (re)start them separately.
Restarting the Hudson Slaves
Hudson will automatically start up the "osx-local-headless" slave just by firing up a child process. The other children must be fired up by hand by logging into the respective machine's consoles via VNC/RDP, and firing up a JNLP agent.
The easiest way is to login to the slave boxes and navigate a browser here: http://xserve.openqa.org:8080/computer/, click on a node name (e.g. vista-vm-1) and click on "Launch slave agent".
The second easiest way (which is sometimes easier on Linux) is to run a command like this:
$ javaws http://xserve.openqa.org:8080/computer/ubuntu-vm-1/slave-agent.jnlp
TODO: launch slave agents on startup. Note that we must NOT launch the slave agents headlessly, or we won't be able to launch our browsers and gather screenshots.
Once you've started Hudson, you may also need to start up "dsmp", described below.
Our Maven configuration
DSMP: Maven proxies and reproducibility
To guarantee a reproducible build, we need to blow away Maven's local repository with every build. We need to do that because otherwise we may accidentally depend on old artifacts we have built without realizing it. (For example, we may accidentally depend on an old dead version of Selenium Core.) Unfortunately, Maven keeps both artifacts YOU make in the same directory structure as artifacts you DOWNLOAD, which is a horrible design flaw.
Blowing away the local repository every time forces Maven to download hundreds of components with every build; since the Internet's reliability is uncertain, this means that Maven will usually fail (at random) to build with a clean local repository, unless you also use a caching proxy server.
So, every Maven slave in our system is also running DSMP, the Dead Simple Maven Proxy. DSMP proxies Maven downloads from the Internet.
Alternatives:
- We could run just one central instance of DSMP that all of the slaves share, but DSMP may not be perfectly reliable in a multi-user environment, so we just run one copy per slave.
- We could try to use Nexus as our Maven proxy. But Nexus isn't really a "proxy" in Maven terms; it's actually a "mirror." There's no way to use a "mirror" to guarantee that all Maven repositories are proxied; the only way to do that is with a regular old HTTP "proxy" like DSMP provides.
- There is no alternative to blowing away the local repository without jeopardizing reproducibility. No command line flag will clean out your own built artifacts; the only way to be sure is to start clean.
This is a very condensed version of a complex problem; if you have more questions about this, ask Dan.
Nexus
We have a nexus box running at nexus.openqa.org, which is where our users download our software. The administrative login is "admin" and the password is the openqa password.
The build boxes themselves need to "deploy" to nexus.openqa.org. ("Deploy" in this context is the Maven term meaning "distribute build artifacts." It probably would have been better if they'd called that something else, like "upload" or "dist".) To do this they need to login to the nexus boxes using HTTP basic authentication. The login for that is "deployment" and the password is the openqa password.
settings.xml
As a result of DSMP and nexus deployment, the settings.xml file should look something like this (with the password obfuscated)
<settings>
<proxies>
<proxy>
<host>localhost</host>
<port>9999</port>
<nonProxyHosts>nexus.openqa.org</nonProxyHosts>
</proxy>
</proxies>
<servers>
<server>
<id>openqa-releases</id>
<username>deployment</username>
<password>**********</password>
</server>
<server>
<id>openqa-snapshots</id>
<username>deployment</username>
<password>**********</password>
</server>
<server>
<id>openqa-thirdparty</id>
<username>deployment</username>
<password>**********</password>
</server>
</servers>
</settings>