This is documentation on the development environment and other issues around Formever.
INTRODUCTION
Formever is designed to be as self contained and independent as possible. To that end everything Formever needs is in a single directory ‘formever’. This directory will be on development and test machines. There is only one code base in Formever. All of the different uses (the server running on Linux serving clients, the code running on those clients, the technical admin program that manages the swarm (the collection of servers that handles all the account) are in the same code base with different main programs that are run. This greatly increases reliability as things like constants never get out of step because all Formever code, running for whatever reason, uses the same static data inside the Java code.
All the scripts that build parts of Formever or that do some maintenance task (mainly in bash) are in the code base and hence under git source code control. They will all be discussed below.
To build Formever the development environment has to run on Linux, Windows and MacOS. This is necessary as you need to be able to test out problems on different platforms and be able to run the debugger if needed. Windows is a bit of the odd man out and it’s special issues will be discussed below.
As simple as I’ve tried to make it there is a lot to explain and I’m not sure if there is some kind of rationale that dictates in what order things should be discussed. However I will plunge in and do the best I can.
SOFTWARE TOOLS
All servers and the main development computer run Ubuntu 16.04. This is a LTS release and will be updated until 2021. There is no reason why a newer version of Ubuntu couldn’t be used. I have tried to make the software as simple as possible and only use basic functions. So there should be no problem to move to a later version.
Presently I’m using Java 10 from Oracle. There is a lot of confusion about licensing right now and Oracle is pushing GraalVM as what people who don’t want a commercial license should use. Presently I can’t run on Java11+ as the server cannot seem to initialize fonts. So I moved back to 10. When the latest version of GraalVM is released to support 13? (or something more than 8 which it does right now). That will be a good time to see if it works for Formever running on the server.
There are several publicly available libraries that are bundled with Formever.
- Apache POI – to allow MS Office documents to be read and written
- Apache PDFBox = to allow pdf s to be read and written
- JavaMail – so that email can be sent
- Crypto shouldn’t be there (it used to be) but it is now in the core Java libraries
Standard linux stuff – especially ssh (and associated commands like scp).
The development environment (IDE) is Intellij’s IDEA. I use the free community edition which is written in Java and runs on all the platforms. It has really nice git integration and a good development features.
I use git for source code control. Right now I am using a separate DO (DIgital Ocean) server (gcs) to hold the git repository. Right now the repository for John’s app development is on that serever. The current intent is to move them to github.
That is it for tools. It is intended to be a small set of highly reliable tools to keep everything reliable and as simple as possible.
DEVELOPMENT COMPUTERS
There are three development computers (Linux, Windows & Mac).
Linux
This is the main development machine (running Ubuntu 16.04 at present). It has the formever
director under the home directory ~/formever. As we shall see this directory occurs on all the development computers and the swarm servers. As much as possible the structure is kept the same on all computers.
I run Gnome classic desktop interface. It is simple and easy to use.
Presently I am running this on an Acer lapop – it has an i7 chip (few years old) , 16GB of memory and 128GB SSD. Passowrd is the same as the Mac (same as Corin’s Mac).
IDEA is installed in a separate directory underneath the home directory. This is also where the various jdks are (only one is being used by there are often several there for different testing). There are also some IDEA configurations in various directories (the default locations set by IDEA).
IDEA has it’s own build in jdk – which is changed when you update IDEA. In the Project settings of IDEA you point to the JDK directory you want to use and all the libraries (pdfbox, poi etc.) that you want to use in your code.
IDEA is an icon on the desktop that is run by being double clicked.
This laptop is currently used for monitoring. There is a bash command to run the "TechnicalAdmin" program in a terminal window. Technical Admin has a main entry point. The Techncial admin has many commands to monitor and manage all the aspects of a Formever swarm. There is a .monitorCloudLog command that every minute gets an update of the log on the server and reflects it. The log display is color coded with red being bad and green, blue and black normally good.
All formever stuff is in ~/formever. I will cover the directory structure separately as it has commonality between the different computers.
However the ~/.ssh directory is the standard ssh (it is easier to leave it in the standard location rather than figure out how to default to another one) and there is a config
file there that associates a name with the IP address etc. of the remote server. Swarm servers are S0,S1 … (production swarm C-PROD) and X0,X1, … (test swarm X-TEST). Right now there is only S0 and X0 as the swarms are singlets at the moment. There is an ssh name (gcs) for the git server, although it is rare to need to directly log into it. There are also ssh names for the two WordPress servers formever.org (wp1) and formever.com (wp3).
In the .ssh file are the various public and private keys. Each swarm server has a public key installed (id_rsa_swarm.pub) which is paired with the private key here (id_rsa_swarm). The swarm public key is intalled by the Server Service (DO or AWS) when the swarm server is created. The ability to install a public key when being created is an essential feature of any such service that is to be use by Formever.
In the .ssh directory are keys for the mail and web servers as well.
There is another important directory ~/formeverbuild – this is where the two jars (formevercloud.jar formeverlocal.jar) that are built by the build script (build.sh in bashscripts) are put. The TechnicalAdmin program when updating the servers will take the jar files from this directory. The copy_jars_to_dropbox.sh script will copy the files in this directory to dropbox (formever_builds/current_build) which is where the scripts that build installers on Mac and Windows will get them to include in the installers. There is also two tiny text files that are build version.txt and uploaddatetime.txt which have the version in them and the date time of upload respectively. Only the Windows installer build copies these to the download directory on formever.org so the date time and verision on the download page come from the Windows installer build and the Mac has no impact on it.
Inside the formever directory is a sub-directory ‘locals’ under this are a series of directories ‘local-0’ ‘local-1’ etc. Each of these directories looks identical to a server directory (under /formever on swarm server). In that it has a
~/formever/locals/local-3/server-0/accounts/account-local directory (for example).
Systems (account-local is the usual way of calling them when they are being tested running in local mode).
In the IDE are configurations for different locals. There is a command line setting that identifies the local to Formever and forces it to look in that directory for the database. So in that way you can run testing and debugging on many systems at the same time. There is no limit to how many locals you can have. Just make a new directory and copy a local configuration and update the local number.
Under the formever directory is a backup directory. It has sub directories s0 and x0 for the two swarm directories. Under these are server-0 directories that have the accounts directory which has all the accounts on that server (eg. account-iatse118). There is a backup_all.sh script in the bashscripts directory (~/formever/dev/com/thinkingworks/bashscripts). This does an rsync and backs up all the servers (2 at the moment).
Remember the data that is backed up from servers is encrypted but local accounts are not. The following command will decrypt an account and copy it to a local (it is a command under Technical Admin – FevTechAdmin – and hence starts with a period)
.decryptAccountFiles iatse118@s0 local3
(The @s0 is a short cut to specify the long path down to the account directory in the backup directory – you can give the path explicitly if you wish – to reference the test swarm you use x0 of course).
The ~/formever/dev directory is the working directory for git. All the code is in the packages com.thinkingworks.*
There are two other reverse domains in the code base dumster (not used) and imgScalr – which is source code imported into do image scaling efficiently.
The git branches are named with bug numbers. So I try and make each change relate to an issue or feature request in the bug database. develop is the main branch and the work pattern is to start a branch, fix the bug and then when all seems okay commit the changes, return to develop branch and merge in those changes and delete that branch that was merged. Of course, some branches take a long time and have to be left for a while . In those cases I ususaly commit an interim commit before returning to develop and don’t merge it . I wait until that branch is ready until finally merging it into develop.
I push develop up to the git server when it changes.
I rarely use other branches on Windows and Mac as most of the time i just pull develop to keep them up to date. Some of the script files or assets impact installers (like the readme.rtf which is used by the Mac installer and should indicate the latest version number and an install notes).
Presently we are not uploading Linux installers and are not running Formever as a client on Linux. This could be done but there is no demand so far for Linux. So the script that generates the Linux download is probably out of date and needs fixing.
Windows
Windows is developed on my MS Surface.
The main use of this machine from a development point of view is to build the Window installer. There are three scripts ont he desktop (these are Windows bat scripts). The slip stream one builds the slip stream download/installer. There is a test one that just builds the installer into a drop box directory. There is a production script that builds a production build. The slipstream build copies the installer as the slipstream download on formever.org and the production script copies a production installer to the download directory.
The c:\formever directory is pretty much a copy of the linux one and all formever directories. It has the dev sub directory which is the git directory. As commits are pushed (from the git branch develop) they are pulled via IDEA on Windows thus keeping the codebase synchronized.
All the scripts use for the build are in bashscripts (probably not the best name) directory in the codebase. There is also the windows_installer_build.bat script that builds a windows installer. It takes a parameter to indicate whether to build SlipStream or Production. It generates an exe from a vbscript formever_local.vbs that runs Formever on Windows. The reason for this is to avoid anti-virus detection that raw bat files attract. The vb script does the standard loop around formever that checks for updates and handles new jdk updates. Basically if there is an update Formever exits and the script will loop back load in the new jars and possible a jdk and re run Formever. This is done on the swarm servers and on the Mac as well but they use bash instead of VBS.
The NSIS installer there. All the build stuff is in the codebase under git.
The code signing certs are there in bashscripts as well. So the build script just runs a bat files.
When a client installs Formever it is put in their user directory in the standard manner. users/<user name>/AppData/Roaming/formever – in that directory is a server-directory.
The code is installed in Program Files (x86). As it is 64 bit code it should be in Program Files.
There is a directory in the formever user directory called bin. This is where the newer jars are copied to by the server if the code being run on the client machine is a version lower than the one on the server. when Formever is run the mod date of those jars are compared to the ones installed in program files and the latest one is loaded.
MAC
The Mac is also used mainly for building the Mac installer. There is a complicated script (in bashscripts) that builds the Mac version putting in all the icons and setting all the hidden Mac switches. Just like the Windows installer build it gets its jar files from dropbox to put them into the installer.
There is a Formever directory on the Mac, of course. It is actually in ~/Projects rather than ~ directly (historical reason lost in time).
One note is that currently the scp command in the script doesn’t copy the installer to the download directory. Haven’t figured this out yet. You have to run the scp command directly in a terminal window and then it works. (Why the identlcal command doesn’t work in the script is a mystery – has something to do with the .ssh config file not being found).
The Mac is used to build the installer and sometimes to test Mac only issues. There aren’t many so development on the Mac is not done very often.
The Mac installs formever into Applications and it has its own sandbox directory to play in. Similar to Windows there is a bin directory that has jar files whose mod dates are compared to the ones in the Application contents and the latest is used.
SWAM SERVERS
Right now we are working with two swarms. Swarms are collections of co-operating computers that heartbeat (not the O/S command but a Formemver TCP/IP feature) each other.
Right now both swarms are singletons.
These two swarms are C-PROD and X-TEST.
The C and the X are used to name the servers the server in C-PROD is C000000L000 and the one in X-TEST is X000000L000
The number is the server number (0 for both). The L000 suffix is for "life" if the server is rebuilt because it failed the life will be incremented e.g. L001 etc.
When I first built the swarm I wanted to handle the case when a server died and put in code to automatically ‘repair’ the swarm by starting up a new server and copying the accounts to it from backup on other servers.
That has turned out not to be needed. these servers are so reliable it never happens. If there is a hardware issue the service detects it and moves our server to another virtual machine. This is just seen as a reboot and everything else (IP address most importantly) remains the same. I made the servers handle reboots so they just start up again and reload the accounts and are ready.
There is a formever directory on the server. It is attached to the root rather than the home. i.e. /formever
Inside this directory is the jre directory which has the java jre. There is a dev directory that has the two jar files (formevercloud.jar formeverlocal.jar). The local jar is here because it has to be downloaded to clients when they log in with an older version.
There is a jre_on_deck directory where you copy a new jdk. when the server is updated it copies that jre to the jre directory so it becomes the running one. This is done as you don’t want to try and replace the running ones while running.
There is an update directory where the new jar files are placed and the script loop in formever.sh which is running Formever will copy them to dev before re-running Formever.
There is the server-0 directory. This is the same structure as all Fomever installations – whether the local directories on a development machine or a standard client install.
There could be server-1 etc. directories as well. If they are here they are being used to backup that server on this server. This is not done right now as Formever level backup is turned off. Right now I’m am rsyncing to my linux box with the backup_all.sh script.
Formever on a swarm server runs as an initd process. This is a very simple use of initd. It is set to run Formever on system start.
You can start or stop Formever as follows:
sudo systemctl stop formever
sudo systemctl start formever
So for example to do this on the X-TEST server you would
ssh x0 – this logs into the X-TEST server
and then execute the systemctl commands. You can of course say reboot which will kick you off and reboot the server.
Remember the current servers are all Digital Ocean ( called droplets in their patois).
Thinkingworks has a DO account. You can create new servers there (although I have code in TechAdmin to do this and so don’t don’t that normally for swarm servers).
It is a good idea to got to DO and create images of the servers (snapshots). You don’t have to power them down to do this I have had no problem. You should do this fairly often and before a production update. The snapshot gives you the abilityt ot restore the server to exactly that state. You can see my naming convention from existing snapshots.
I don’t have any AWS servers at present. They are more expensive and more complicated.
There is an interface to a server service that allows Formever to use it when implemented for that service. They are all a bit different so you have to custom write one each time you want to add another (IAAS?) service.
For each server there is a swarm directory which is kept in memory and is a list of all the accounts on the server. The swarm servers all talk to each other (when there is more than one) and can find out what server an account is one if it isn’t one of theirs.
There is a swarm server location facility. In the thinkingworks dns (managed under a hover account). There is a obscure url fqz___… (or something like that) Against that we list the IP address of one of the production servers and one of the test servers. So with a simple call to DNS ( a Java library function) we can get the IP address of a swarm member. We can then send it a ‘management’ request to ask to the IP of the fellow swarm member that has an account.
Once we have the IP it is stored in a local properties file so that it can be used directly as people sign into the same account all the time. If the server is changed the connection will fail and then the DNS will be used to get a server to query fo rthe new IP address of the server having that account. This means we can move accounts between servers without any disruption.
THE BUILD PROCESS (LINUX Computer)
This is a brief description of the build process.
1. You have a bunch of bug fixes and enhancements
2. The are all done in branches, committed and merged into develop
3. develop is pushed to the gcs server (git)
4. when a slipstream is deemed ready to release for testing then in develop four files are updated
a) the FormeverVersion class has the slipstream number incremented and the build number incremented. (the build number is what controls the test for a new version – we are are around 1340 right now – it just keeps on going up). Remember a sliptstream number of 0 means a production build.
b) the readme.rtf is updated to reflect the latest version and build number
c) the copy_jars_to_dropbox is updated to save the previous jars into a directory labelled with the version and slipstream (this is done so we have copies of all the jars we have published both slipstreams and productions) You can look at the previous_builds subdirectory of current_builds to see how all the previous ones have been named.
d) the build_windows_installer.bat is updated to put the version, ss and build into the meta data on the exe – there is no really good reason for doing this but it has become a habit.
5. Now these changes are committed as a slipstream release or a production release (depending). Basically if you look back through the git log you will see the commit messages for these going back many versions.
6. Push to gcs (this is just going to the version control/git menu and doing a push. The IDE is all set up with the server name and ssh keys).
7. Go to a terminal window and run the build.sh script in bashscripts. You should cd to ~/formever/dev/com/thinkingworks/bashscripts first
8. when that is finished run the copy_jars_to_dropbox
9. It is a good idea to allow a bit of time for dropbox to sync on Windows and Mac. So go to the IDEA and run the techamdin program (I usually run it in debug mode).
10. The admin command is
.updateFormeverCode X-TEST 0 (for slipstream)
.updateForemverCode C-PROD 0 (for production)
This will copy the jars to the server and restart the server so the start with new code. If someone is logged into that swarm it will go through an 11 minute process to get them off (messages and finally forced lgoout).
11. Check the monitoring terminal session (it should be running the runtechadmin.sh script (just run one from the bash history). then run the techadmin command .monitorCloudLog C-PROD 0 (or .monitorCloudLog X-TEST 0 for test swarm). These are best to just keep running all the time.
You should see the server update and finally settle with ‘initialization finished’
12. Go to windows and run the test or production script depending. It will build the installer and upload it to the website.
13. Go to Mac and run one of the script commands on the desktop (test or production) – remember the scp command (secure copy) has an issue and has to be redone manually.
All done – the new slipstream or production release is ready to go.
