FarragoBuild
From Eigenpedia
Contents |
Introduction
This document provides instructions on how to build and run Farrago and its dependencies (including Fennel).
Supported Platforms
So far, the Fennel C++ library only builds and runs on Linux, Mac OS X, and Windows. This means that if you want to get a fully functional Farrago build, you have to be running one of these operating systems. However, the Java portion is likely to run on other platforms where JDK 1.5 is available. You can use it to test parsing, validation, optimization, and the metadata aspects of DDL, as well as queries against foreign data accessible via Java (a sample HSQLDB source is provided for testing JDBC, and the Freebase MQL plugin demonstrates access to a webservice).
So, if you're working on a task that doesn't require Fennel, you can do the development anywhere. If you need Fennel, then you have two options:
- use your own Linux, Mac or Windows box if you have one
- work on porting Fennel to your platform!
Important Windows Note
If you are using Windows, please read and follow the Windows setup instructions before going any further.
Get the Source Code
There are several ways to get the Farrago source code:
- download a full Eigenbase source release
- pull the latest code from the Eigenbase Perforce server
- pull the latest code from the anonymous-access Subversion replica, as described in PerforceToSubversionReplication (svn co svn://perforce.eigenbase.org/trunk/open/dev)
- Note that this is experimental but so far so good. However, it is read-only, so it's not useful if you plan to commit changes directly (that requires Perforce). It is quite useful if you want to submit patches only and get someone with Perforce access handle the commit for you.
If you just want to take a look-see through the codebase, you can use one of
- p4web (with full revision history)
- Subversion http (latest code only)
If you have downloaded a full Eigenbase source release (or checked out via Subversion), you can skip the rest of this section about Perforce and continue on with #Initial Build; note that in this case we'll refer to the top-level directory created when you unpacked the release archive as dev in this and other developer docs.
If you're new to Perforce, you might want to read the official Perforce Quick Start. If you don't already have a p4 client program, download the appropriate binary from Perforce (you only need the client, not the server).
There are a number of ways to configure Perforce depending on the platform (on Windows through the GUI, on Unix through environment variables or a .p4env file). The essential information you need to provide is:
- P4PORT=perforce.eigenbase.org:1666
- P4USER=your-sourceforge-user-name
- P4CLIENT=your-user-name.your-hostname
In order to make changes, you'll need to send your username to the Farrago project admin and request write access. If you don't have a Sourceforge account yet, go get one. For read-only clients (like if you just want a snapshot of the latest code), use the predefined username "guest" (the server license has a user limit, but there's no limit on the number of clients, so feel free to create multiple workspaces on as many machines as needed).
Once you've got Perforce set up, run p4 client (or edit your default clientspec from P4WIN). Map your view to match this example:
View:
//open/dev/thirdparty/... //YOURCLIENT/dev/thirdparty/...
//open/dev/fennel/... //YOURCLIENT/dev/fennel/...
//open/dev/farrago/... //YOURCLIENT/dev/farrago/...
We'll refer to the YOURCLIENT/dev local filesystem directory as dev in this and other developer docs.
While editing your client, you should also set the LineEnd attribute to share, especially if you are developing on Windows or using Samba. This prevents DOS CRLF's from getting checked in.
Finally, run p4 sync to get all the source files and thirdparty distributions into your client workspace. This may take a while because some of the thirdparty distributions are large.
Initial Build
Before proceeding, make sure you have JDK 1.5 or higher installed, with JAVA_HOME set to the JDK installation root.
For the first build in a new sandbox, there's a lot of one-time setup that needs to be done, including building some of the third-party components from source. This process is automated by the script dev/farrago/initBuild.sh. Change to the dev/farrago directory and run the initial build as follows:
initBuild.sh --with-fennel or initBuild.sh --without-fennel
If you're building without Fennel, this should be fairly quick (under ten minutes on most machines); with Fennel it may take quite a bit longer. If all goes well, you'll be left with a complete Farrago build for your platform. Otherwise, you'll need to dig into the script and its output to see what's going wrong. If you can't figure it out, ask for help on the farrago-developers mailing list. You should probably include the last 100 lines of build output including the error.
Fennel Build
The Fennel build is documented at FennelBuildSystem.
If you need help with problems in the C++ portion of the build, use the fennel-developers mailing list.
Environment Variables
The initial build script takes care of setting some necessary environment variables internally, but once it completes they are left unset. For subsequent runs and builds of Farrago, you will need to make sure these are set. Scripts are provided for this purpose. You can run them automatically from a login script such as .bash_profile, or run them manually before working with Farrago:
source dev/fennel/fennelenv.sh dev/fennel source dev/farrago/farragoenv.sh dev/thirdparty
As usual, you only need the Fennel script if Fennel support is enabled. Don't execute these scripts directly; you must use . or source in order for the environment variable settings to take effect for subsequent commands.
Incremental Builds
Once you already have a working build, updating to the latest code (assuming you're using the Perforce server) requires following these steps:
- p4 sync. You'll probably want to examine the output to see what has changed (or you can ignore it and blindly carry out the rest of the steps unconditionally).
- If anything under dev/thirdparty/... has changed, cd to that directory and run make all. New thirdparty components and upgrades to new versions are fairly rare. Upgrades to C++ components usually require you to repeat the original initBuild.sh process.
- If anything under dev/fennel/... has changed (and you're building with Fennel support), cd to that directory and run make. If it has been a long time since your last sync, you may get errors, in which case you should run make clean and then try again. If you still get errors, see below on forcing a full rebuild.
- cd to dev/farrago and run ant test to make sure you'll be working from a good build (WARNING: this will rebuild your catalog automatically, destroying any data in your Farrago database). To avoid running the tests, run ant createCatalog instead.
If none of the C++ components under thirdparty have been upgraded, and you'd like to force a rebuild of everything else, skip the steps after the sync and instead run
cd dev/farrago initBuild.sh --with-fennel --skip-fennel-thirdparty-build
This will recapitulate the original full build, but skipping the time-consuming thirdparty C++ components.
Catalog Dump
At any time, you can get a dump of the catalog by running ant exportCatalog; the output will be an XMI file in dev/farrago/catalog/xmi/FarragoCatalogExport.xmi. This can be useful for debugging the effects of DDL execution.
Catalog Backup/Restore
The ant createCatalog task (invoked automatically as part of ant test) creates a backup of the initial catalog (and database if Fennel is enabled). If your catalog gets corrupted, you can recover with ant restoreCatalog. This is a lot faster than ant createCatalog (which is another way to get back to a good state).
Interactive SQL
You can execute interactive SQL from a command-line via the open-source JDBC sqlline project. Just run the sqllineEngine script from dev/farrago. After connecting, try select * from sales.emps; (but if you disabled Fennel, don't be disappointed when the result set comes back empty).
For information about hacking sqlline, see SqllineDocs.
Tracing
See FarragoTracing for instructions on tracing Farrago.
Unit Testing
See FarragoTests for information on Farrago unit tests.
Debugging
The developer FAQ contains some useful information on debugging, including how to set up Eclipse.
A standalone debugger (JSwat) is also provided in the thirdparty distribution. Run ant jswat.sqlline or ant jswat.isql to use this debugger for execution of interactive SQL commands. For sqlline, input/output is performed through the JSwat Output pane at the bottom left of the screen. The junitDebug script can also be used to debug one unit test at a time. If you want to use another debugger, the jswatDebug ant macro definition in dev/farrago/buildMacros.xml might be helpful as a template for passing the right startup information. Also, as part of a successful build, the full classpath is always dumped to the file dev/farrago/classpath.gen.
--Jvs 20:56, 15 July 2009 (EDT): I don't think anyone has used JSwat for a while, so it may no longer be working.
Client/Server Operation
So far, all of the execution modes discussed have been embedded engine mode, where the Farrago engine is loaded directly into a single process such as sqlline. This is the most useful for debugging, but scripts are also available to test client/server mode. Run the farragoServer script to start the server in a shell (after startup, it waits listening for connections; enter the !quit command in that shell to shut it down). From another shell, run sqllineClient to connect and execute commands.
Coding
Please read the CodingConventions documentation. Before starting development projects, consider using private branching.
Feedback
If you have trouble with any of this, or have other questions, use the farrago-developers mailing list.
