bhansley

Windows 7, Wired and Wireless networks

2010-10-05T09:38:00.006-04:00

I've been on Windows 7 (64-bit) for a few months now, and just recently moved to a new Lenovo T510. The move, using Windows backup and restore, was downright painless and just worked.

One issue I had with the new setup that was frustrating me was that, when docked, the laptop would keep a strong preference for wireless connections over wired ones. Apparently, Windows 7 defaults to using some customized route preference calculation metric, and had my wireless at 90 and my wired connection at 900-ish. (Lower scores indicate stronger affinity.)

You can see your current, auto-assigned routing metrics by typing netstat -r from a command prompt, and looking in the 'Metric' column of hte IPv4 Route Table.

When I'm docked, I want wired to take precedent, as the wired connection has a static IP, and makes certain dev and admin tasks possible that cant' be done around here on a regularly-shifting DHCP address on wireless - things like sending SMTP mails without credentials, the way servers get to.

So, using an articles from Lifehacker and Palehorse as a starting point, I set forth to change my interface metrics to suit my needs.

Starting with Control Panel\Network and Internet\Network and Sharing Center, click the wired connection under Connections:
Then click 'Properties'.
Then select 'Internet Protocol Version 4 (TCP/IPv4)' and click 'Properties'.
Click 'Advanced' at the bottom of the window.
Un-check the 'Automatic Metric' box, and enter a new routing metric here. Lower numbers are higher priority. I used a routing metric of 10 for my wired connection and 200 for my wireless, and everything seems to be behaving exactly as I want.
Repeat from #1 selecting your wireless network connection.
Reboot when done to have the new assignments take effect.

As above, you can verify your assignments by typing netstat -r from a command prompt, and looking in the 'Metric' column of hte IPv4 Route Table.

A word of warning: you may want to check with your networking people before you do this. Routers can assign this metric, and overriding what they assign may have unintended consequences for how you access your networks.

Bill

Daylight Savings Time in SQL Server

2010-05-13T13:42:00.006-04:00

We're running a vendor system that stores every date/time object in GMT. This isn't a bad thing, but it does require converting GMT -> Local time in every place we used it: select statements, where clauses, corelated subqueries, etc. To do this we started out years ago with a simple Function plus a database table to store DST dates.

That function looked like this:

CREATE FUNCTION [dbo].[ConvertGMT]
(@MYDATE DATETIME)
RETURNS DATETIME AS
BEGIN
DECLARE @OFFSET INT
DECLARE @YEAR INT
DECLARE @DST_BEGIN DATETIME
DECLARE @DST_END DATETIME
DECLARE @RETURNED_DATE DATETIME

SET @YEAR = YEAR(@MYDATE)
SET @DST_BEGIN = (select DST_BEGIN from dbo.DST_LOOKUP where YEAR_LOOKUP = @YEAR)
SET @DST_END = (select DST_END from dbo.DST_LOOKUP where YEAR_LOOKUP = @YEAR)

IF (@MYDATE BETWEEN @DST_BEGIN AND @DST_END)
SET @OFFSET = -4
ELSE
SET @OFFSET = -5

SET @RETURNED_DATE = DATEADD(hour,@OFFSET,@MYDATE)
RETURN @RETURNED_DATE
END

and the associated table looked like this:

DST_BEGIN DST_END YEAR_LOOKUP
4/4/1999 2:00:00 AM 10/31/1999 3:00:00 AM 1999
4/2/2000 2:00:00 AM 10/29/2000 3:00:00 AM 2000
4/1/2001 2:00:00 AM 10/28/2001 3:00:00 AM 2001
4/7/2002 2:00:00 AM 10/27/2002 3:00:00 AM 2002
4/6/2003 2:00:00 AM 10/26/2003 3:00:00 AM 2003
4/4/2004 2:00:00 AM 10/31/2004 3:00:00 AM 2004
4/3/2005 2:00:00 AM 10/30/2005 3:00:00 AM 2005
4/2/2006 2:00:00 AM 10/29/2006 3:00:00 AM 2006
3/11/2007 2:00:00 AM 11/4/2007 3:00:00 AM 2007
3/9/2008 2:00:00 AM 11/2/2008 3:00:00 AM 2008
3/8/2009 2:00:00 AM 11/1/2009 3:00:00 AM 2009
3/14/2010 2:00:00 AM 11/7/2010 3:00:00 AM 2010
3/13/2011 2:00:00 AM 11/6/2011 3:00:00 AM 2011

This worked well enough for a while, but it cost a couple of table hits for every conversion. That adds up when you're doing 6 conversion per row returning a couple hundred rows a few times a minute. This benchmarked at ≅15seconds to return 45,000 at 3 conversions / row in a simple select.

Our next step was to eliminate the table lookup. That was simple and inelegant, and looked like this:

ALTER FUNCTION [dbo].[ConvertGMT] (@MYDATE DATETIME)
RETURNS DATETIME AS
BEGIN
DECLARE @OFFSET INT
DECLARE @YEAR INT
DECLARE @RETURNED_DATE DATETIME

SET @YEAR = YEAR(@MYDATE)

IF @YEAR = 2010 AND @MYDATE between 'Mar 14 2010 2:00AM ' and 'Nov 7 2010 3:00AM ' SET @OFFSET = -4
ELSE IF @YEAR = 2009 AND @MYDATE between 'Mar 8 2009 2:00AM ' and 'Nov 1 2009 3:00AM ' SET @OFFSET = -4
ELSE IF @YEAR = 2008 AND @MYDATE between 'Mar 9 2008 2:00AM ' and 'Nov 2 2008 3:00AM ' SET @OFFSET = -4
ELSE IF @YEAR = 2007 AND @MYDATE between 'Mar 11 2007 2:00AM ' and 'Nov 4 2007 3:00AM ' SET @OFFSET = -4
ELSE IF @YEAR = 2006 AND @MYDATE between 'Apr 2 2006 2:00AM ' and 'Oct 29 2006 3:00AM ' SET @OFFSET = -4
ELSE IF @YEAR = 2005 AND @MYDATE between 'Apr 3 2005 2:00AM ' and 'Oct 30 2005 3:00AM ' SET @OFFSET = -4
ELSE IF @YEAR = 2004 AND @MYDATE between 'Apr 4 2004 2:00AM ' and 'Oct 31 2004 3:00AM ' SET @OFFSET = -4
ELSE IF @YEAR = 2003 AND @MYDATE between 'Apr 6 2003 2:00AM ' and 'Oct 26 2003 3:00AM ' SET @OFFSET = -4
ELSE IF @YEAR = 2002 AND @MYDATE between 'Apr 7 2002 2:00AM ' and 'Oct 27 2002 3:00AM ' SET @OFFSET = -4
ELSE IF @YEAR = 2001 AND @MYDATE between 'Apr 1 2001 2:00AM ' and 'Oct 28 2001 3:00AM ' SET @OFFSET = -4
ELSE IF @YEAR = 2000 AND @MYDATE between 'Apr 2 2000 2:00AM ' and 'Oct 29 2000 3:00AM ' SET @OFFSET = -4
ELSE IF @YEAR = 1999 AND @MYDATE between 'Apr 4 1999 2:00AM ' and 'Oct 31 1999 3:00AM ' SET @OFFSET = -4

ELSE IF @YEAR = 2011 AND @MYDATE between 'Mar 13 2011 2:00AM ' and 'Nov 6 2011 3:00AM ' SET @OFFSET = -4
ELSE IF @YEAR = 2012 AND @MYDATE between 'Mar 11 2012 2:00AM ' and 'Nov 4 2012 3:00AM ' SET @OFFSET = -4
ELSE IF @YEAR = 2013 AND @MYDATE between 'Mar 10 2013 2:00AM ' and 'Nov 3 2013 3:00AM ' SET @OFFSET = -4
ELSE IF @YEAR = 2014 AND @MYDATE between 'Mar 9 2014 2:00AM ' and 'Nov 2 2014 3:00AM ' SET @OFFSET = -4

ELSE SET @OFFSET = -5

SET @RETURNED_DATE = DATEADD(hour,@OFFSET,@MYDATE)
RETURN @RETURNED_DATE
END

The performance boost with this version was amazing. It processed the same 45,000 x 3 conversion / row query in ≅3 seconds. Across all our GUI and report calls, that was a noticeable performance boost. But, it looked like something someone in CS101 would write, and it still required occasional updates to add new years and keep current.

The final step - and current solution - started with a DST discussion at MSSqlTips that used some pre-calculated offsets to dynamically calculate DST start and stop dates based on the DOW of the start and end months. This worked well for dates 2007 and later, but earlier dates had a different date pattern. I continued the same model that Tim Cullen started with the MSSqlTips post and used a set of precalculated offsets for 2006 and earlier starts, and dynamically calculated DST end dates.

The final function looks like this:

CREATE FUNCTION [dbo].[ConvertGMT]
(@MYDATE DATETIME)
RETURNS DATETIME AS
BEGIN
DECLARE @OFFSET INT
DECLARE @YEAR INT
DECLARE @DST_BEGIN DATETIME
DECLARE @DST_END DATETIME
DECLARE @RETURNED_DATE DATETIME

SET @YEAR = YEAR(@MYDATE)

declare @DSTStartWeek smalldatetime, @DSTEndWeek smalldatetime

if @YEAR >= 2007
BEGIN
set @DSTStartWeek = '03/01/' + convert(varchar,@YEAR)
SET @DST_BEGIN = case datepart(dw,@DSTStartWeek)
when 1 then
dateadd(hour,170,@DSTStartWeek)
when 2 then
dateadd(hour,314,@DSTStartWeek)
when 3 then
dateadd(hour,290,@DSTStartWeek)
when 4 then
dateadd(hour,266,@DSTStartWeek)
when 5 then
dateadd(hour,242,@DSTStartWeek)
when 6 then
dateadd(hour,218,@DSTStartWeek)
when 7 then
dateadd(hour,194,@DSTStartWeek)
end

set @DSTEndWeek = '11/01/' + convert(varchar,@Year)
SET @DST_END = case datepart(dw,dateadd(week,1,@DSTEndWeek))
when 1 then
dateadd(hour,2,@DSTEndWeek)
when 2 then
dateadd(hour,146,@DSTEndWeek)
when 3 then
dateadd(hour,122,@DSTEndWeek)
when 4 then
dateadd(hour,98,@DSTEndWeek)
when 5 then
dateadd(hour,74,@DSTEndWeek)
when 6 then
dateadd(hour,50,@DSTEndWeek)
when 7 then
dateadd(hour,26,@DSTEndWeek)
end
END
ELSE
BEGIN
set @DSTStartWeek = '04/01/' + convert(varchar,@YEAR)
SET @DST_BEGIN = case datepart(dw,@DSTStartWeek)
when 1 then
dateadd(hour,2,@DSTStartWeek)
when 2 then
dateadd(hour,146,@DSTStartWeek)
when 3 then
dateadd(hour,122,@DSTStartWeek)
when 4 then
dateadd(hour,98,@DSTStartWeek)
when 5 then
dateadd(hour,74,@DSTStartWeek)
when 6 then
dateadd(hour,50,@DSTStartWeek)
when 7 then
dateadd(hour,26,@DSTStartWeek)
end

SET @DSTEndWeek = '11/01/' + convert(varchar,@Year)
SET @DST_END = dateadd(hh,3,dateadd (dd,(1 - datepart(dw,@DSTEndWeek)),@DSTEndWeek))

END

IF @MYDATE between @DST_BEGIN and @DST_END
SET @OFFSET = -4
ELSE
SET @OFFSET = -5

SET @RETURNED_DATE = DATEADD(hour,@OFFSET,@MYDATE)
RETURN @RETURNED_DATE
END

This code has one change from Tim's code - it now ends at 3am in November, for 2007 and later. This is what we have in production now. It's just as performant as the previous version (≅3 sec for the benchmark query), but it has the advantage of never needing updating for new years. It's more elegant than it's predecessors, but I have no doubt that it could be done better.

Use, share, comment.

-Bill

(Thanks to the Wikipedia DST Article for clarity on this.)

Apache, SSL and Tomcat Clustering

2009-06-26T14:04:00.003-04:00

With help from the Apache HTTPd docs, the Apache Tomcat docs, and lots of Google-help, I've set up a failover-friendly load balanced server pair that seriously ups our game in terms of high availibility, and boosted performance as a by-product.

Background (skip this if you just want to get to the server setup details)

My group does custom web applications for use the Operating Rooms. We interface with multiple clinical and administrative systems and show, collect, and share data that's critical to the operations and to patient safety. Our longest running application, ORview, is a Java web app that collects pre-operative assessments and post-operative assessments, provides airport monitor-style big screen views throughout the OR area, and provides other billing, reporting and QA/QI functionality. Our newest application is RequestOR, which collects posting requests for new case postings, and shares, via HL7, this data with GE's Centricity Periopertive Manager and IDX. Our old ORview prod server was at end-of-life, and we were needing a new server to deploy RequestOR on so we designed, spec'd and installed a new server cluster that'll meet the needs of both of our major applications, and give us some room to migrate some of our more minor applications (Java and JRuby).

Server Hardware

Qty 2 - IBM HS21 Blade Server, Dual Xenon Quad-core @ 3.0GHz, 8GB PC2-5300 RAM, 146G SAS HDD.

Server Software

Red Hat Enterprise Linux Server release 5.3 (Tikanga)

Java SE Runtime Environment 64-bit (build 1.6.0_13-b03)

Apache HTTPd 2.2.3 (httpd-2.2.3-22.el5_3.1) with mod_ssl (mod_ssl-2.2.3-22.el5_3.1) and mod_proxy_ajp (as part of the httpd install)

Apache Tomcat 6.0.18

Network/DNS Configuration

The key to doing multiple SSL-secured applications is that each unique SSL certificate NEEDS it's own IP address and host name to bind to. (There are some budding ways around that but none of those were mature enough going into this process to be a viable production option.)

requestor: 10.20.215.226, 10.20.215.228 - configured as dns round-robin

orview2: 10.20.215.227, 10.20.215.229 – configured as dns round-robin

Those IPs get distrubuted to each server that's hosting that application, along with each server having their own ip.

server 1

10.20.215.223 – blade1

10.20.215.226 – requestor

10.20.215.227 – orview2

228.0.0.23 - multicast

server 2

10.20.215.224 – blade2

10.20.215.228 – requestor (dns rr)

10.20.215.229 – orview2 (dns rr)

228.0.0.23 - multicast

Apache HTTPd Configuration

First, the explanation. There's a default port 80 host (in black) that just answers requests on the machine's unique IP. We use this index.html to point to a simple machine ident. The RequestOR section is next (in blue). It defines a port 80 host that redirects all requests to the port 443 (ssl) version of itself. The second virtual host in the blue is the ssl version. This section contains config info for the SSL certificates, and a reference to balancer://ajpCluster/requestor as handler for all requests to this virtualhost. The ORview2 section (in red) largely duplicates this configuration for the second ssl application on these servers. The second server's config files are identical, except the IPs are changed to match that server's configuration.

<VirtualHost *:80>
DocumentRoot /var/www/html
ServerName blade1.foo.edu
</VirtualHost>
# RequestOR virtual hosts
<VirtualHost 10.20.215.226:80>
DocumentRoot /var/www/html/requestor
ServerName requestor.foo.edu
Redirect permanent / https://requestor.foo.edu/
</VirtualHost>
<VirtualHost 10.20.215.226:443>
DocumentRoot /var/www/html/requestor
ServerName requestor.foo.edu
SSLCertificateFile
/etc/pki/tls/certs/requestor.foo.edu.crt
SSLCertificateKeyFile
/etc/pki/tls/private/requestor.foo.edu.key
SSLEngine on
SSLProtocol all -SSLv2
<Location />
ProxyPass balancer://ajpCluster/requestor
stickysession=JSESSIONID
</Location>
# ErrorLog logs/dummy-host.example.com-error_log
# CustomLog logs/dummy-host.example.com-access_log common
</VirtualHost>
#ORview2 virtual hosts

<VirtualHost 10.20.215.227:80>
DocumentRoot /var/www/html
ServerName orview2.foo.edu
Redirect permanent / https://orview2.foo.edu/
</VirtualHost>
<VirtualHost 10.20.215.227:443>
DocumentRoot /var/www/html
ServerName orview2.foo.edu
SSLCertificateFile /etc/pki/tls/certs/orview2.foo.edu.crt
SSLCertificateKeyFile
/etc/pki/tls/private/orview2.foo.edu.key
SSLEngine on
SSLProtocol all -SSLv2
<Location />
ProxyPass balancer://ajpCluster/orstat stickysession=JSESSIONID
</Location>
</VirtualHost>

Apache mod_proxy_ajp Configuration

This is the config for the ajp load balancer. Tomcat on each server is set to listen for AJP requests on port 8009. This config file (the same on each server) tells the AJP balancer about the cluseter composed of TomcatA and TomcatB. In the absence of any other details, it'll default to sending new requests to the least loaded Tomcat server, and sending requests from existing sessions to the server that's been handling them. That's what the stickysessions attribute makes happen. Proxy listeners are configured for each application, same color scheme as above.

LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
#
# When loaded, the mod_proxy_ajp module adds support for
# proxying to an AJP/1.3 backend server (such as Tomcat).
# To proxy to an AJP backend, use the "ajp://" URI scheme;
# Tomcat is configured to listen on port 8009 for AJP requests
# by default.
#
<Location /balancer-manager>
SetHandler balancer-manager
</Location>
<Proxy balancer://ajpCluster>
BalancerMember ajp://blade1.foo.edu:8009 route=tomcatA
BalancerMember ajp://blade2.foo.edu:8009 route=tomcatB
</Proxy>
<Location /requestor>
ProxyPass balancer://ajpCluster/requestor stickysession=JSESSIONID
</Location>
<Location /orstat>
ProxyPass balancer://ajpCluster/orstat stickysession=JSESSIONID
</Location>

Tomcat's server.xml

The server.xml is generally a good sized file where most of the defaults are just fine. I've excerpted the relevant bits here that I had to change to get clustering working.

A connector is defined in the <server> portion of the file. It should be enabled by default.

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />

There should be one <engine> element. Edit it to include the unique name used for this in the proxy-ajp config file - tomcatA in this case. The other server's server.xml looks the same except for tomcatB here.

A cluster element is defined inside the <engine> element. The important thing to set here is the multicast address (in red) to be used by Tribes to synchronize session information across the servers in the cluster. The FarmWarDeployer (in blue) is experimental and (as of when this was written) isn't ready for prime-time.

<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
channelSendOptions="6">

<Manager className="org.apache.catalina.ha.session.BackupManager"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true"
mapSendOptions="6"/>

<Channel className="org.apache.catalina.tribes.group.GroupChannel">
<Membership className="org.apache.catalina.tribes.membership.McastService"
address="228.0.0.23"
port="45564"
frequency="500"
dropTime="3000"/>
<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="auto"
port="5000"
selectorTimeout="100"
maxThreads="6"/>

<Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
</Sender>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"/>
</Channel>

<Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;"/>

<Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
tempDir="/tmp/war-temp/"
deployDir="/tmp/war-deploy/"
watchDir="/tmp/war-listen/"
watchEnabled="false"/>

<ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>

Conclusion

It's completely possible to get high-availibility load balancing and clustering working for Apache's HTTPd and Tomcat under Linux. The performance and fault-tolerant benefits are completely worth it. and thanks to a lot of work done by a lot of dedicated people, it's pretty easy to get set up and running. I send my thanks to every site I Googled figuring out how to get this working - there are too many to count. If you have questions or corrections, please post them here, and I'll do what I can to help figure them out.

Unix File Permissions

2007-07-16T11:15:00.001-04:00

A useful tidbit that I just discovered: if you want to set unix file permissions for the group to match the owner it's as simple as:

chmod g=u filename

This just saved me boatloads of time.

Discovered at: comp.unix.questions

-Bill

Harnessing Capistrano

2007-05-17T19:34:00.000-04:00

Harnessing Capistrano - RailsConf2007 Tutorial
Jamis Buck

Slide aand demos available here.

Started out as a Deployment tool

Can also use it for:
+ monitoring tool across all servers (ps, df, uname, etc.)
+ server maintenance (mounts, symlinks, ...)
+ troubleshooting

A basic config file and a demo.

A config using a gateway and a demo.

A cool description and demo of 'cap -e shell' which creates and caches a connection to each deployment server. Can be scoped by role or host.

Cap 2.0 adds namespaces for tasks. Sort of a way of grouping like tasks. (eg. cap tail:showuploadlog groups the showuploadlog task into the tail namespace.

Can do variables.

And transactions, so you can make sure tasks complete on all servers or it rolls them all back - no inbetween. No good way to recover if a rollback fails.

All sort of options for including other cap files. This was always optional in Cap 1.x, but there is no default config in Cap2.0.

You can also script-check dependencies (cap deploy:check). That looks damn useful.

scrpit/spin tell cap how to start your application Many times i's script/process/spawner

cap deploy:cold <- first time deploy

cap deploy:rollback
cap deploy:migrations
cap deploy:web:disable
cap deploy:web:enable
cap deploy:cleanup <- clean up all but the last 5 releases

In addion to standard a version control-based checkout, you can do other types: export, copy, remote cache, or you can roll your own. (Set by :deploy_via option.)

There are all sorts of nifty options to get Cap to work with all sorts of version control systems.

Lots of helpers: run, sudo, stream, connect, find_task, etc.

In the ways of advanced usage, there are:
+ before_ and after_ events (before_deploy: run_tests)
+ custom callbacks: (email notifications, etc.) with complex rules
+ staging environments: you can script deploys to behave differently based on target

JRuby on Rails Tutorial

2007-05-17T17:51:00.000-04:00

JRuby on Rails - RailsConf 2007 Tutorial

Advantages:
Take advantages of existing Java libraries
Run on Java infrastructure
Supports Java's multibyte/unicode strings.
Support Ruby's thread API, one native thread = one system thread, good multicore use.
True concurrent threading.

Great progress in the past year. From barely running rails 1.1 last year to running all of rails (except some lowlow demand dark corners) now, with good performance on everything but RDoc.

Most Gems just work. Anything pure Ruby (or with a JRuby port) runs.
Webrick works.
JRuby port of Mongrel works.

Differences:
1-Database support
Pure Ruby drivers work - mySQL
All the JDBC drivers you'd want to use work. (Yay!) (Some need custom coding to support migrations.
JDNI for connection pooling (More yay!)

2-No Native Extensions
Unless there's a port.
Mongrel - done
Hpricot - done
Database support - some done, some in progress
RMagick - in progress

3-Command-line Performance
Very good (possibly faster) once you're running, but typical java-slow startup performance.

Deployment
1. Mongrel works well. No process forking, process management
But why? Use Java App Servers via GoldSpike (or rails-integration) plugin.
2. Build WAR files for Rails apps.
One plugin, pure Ruby, out comes a deployable WAR file.
3. Glassfish server gem. (Sort of a "Pack" in the box implementation.)
Not yet. But soon.

Migrating existing Rails apps to JRuby/Rails
Be aware of the currently unsupported features.
1. Database support
- MySQL is great
- Derby & HSQL work well. Small embedable DB.
- Postgres - Few out of 1000+ tests.
- Oracle - starting to get attention
- MS SQL Server & DB - Need help, haven't really been worked on much.
Migrations mostly work well, tricky on some DBs that don't have all features.
Fixtures work well, parts of rails test. Issues generally YAML rather than DB issues.
2. Native Extentions
Option 1 - Use something else, aka don't do it.
Option 2 - Use an equivalent Java library. (binds you to jruby)
Option 3 - Port the library yourself.
Option 4 - Port by wrapping a java library.
3. Deployment Options
- Mongrel: works, but not the most efficient
- Existing WebApp Server: good concurrency, clustering, resource pooling
- Grizzly/GlassFish v3 option: lightweight, gem mongrel-like install

Cascading Drop-downs in Rails

2007-01-22T22:24:00.000-05:00

We were looking for a solid example of cascading dynamic drop-down select lists to use in our rails application, and found the web sorely lacking in solid examples. We found a very good start at http://www.railsweenie.com/forums/2/topics/767 but it wasn't complete enough, or didn't work entirely. So my very good buddy and co-worker Sheri and I figured this out, and got it working for our app, and wanted to document it here in the hopes that it helps someone else. Here's the nutshell version:

It's probably already there, but make sure this line is in your standard_layout.rhtml:

<%= javascript_include_tag :defaults %>

Add this function def to application_helper.rb:

def update_select_box( target_dom_id, collection, options={} )
# Set the default options
options[:text] ||= 'name'
options[:value] ||= 'id'
options[:include_blank] ||= true
options[:clear] ||= []
pre = options[:include_blank] ? [['','']] : []
out = "update_select_options( $('" << onclick="BLOG_clickHandler(this)" class="blsp-spelling-error" id="SPELLING_ERROR_5">dom_id.to_s << "'),"
out << "#{(pre + collection.collect{ |c| [c.send(options[:text]), c.send(options[:value])]}).to_json}" << ","
out << "#{options[:clear].to_json} )"
end

This calls update_select_options which needs to go into application.js:

function update_select_options( target, opts_array, clear_select_list ) {

if( $(target).type.match("select" ) ){ // Confirm the target is a select box

// Remove existing options from the target and the clear_select_list
clear_select_list[clear_select_list.length] = target // Include the target in the clear list

for( k=0;k <>
obj = $(clear_select_list[k]);
if( obj.type.match("select") ){
len = obj.childNodes.length;
for( var i=0;i <>
}
}

// Populate the new options
for(i=0;i <>
o = document.createElement( "option" );
o.appendChild( document.createTextNode( opts_array[i][0] ) );
o.setAttribute( "value", opts_array[i][1] );
obj.appendChild(o);
}
}
}

Add something like this to the form.rhtml (changing the name of the observable field as appropriate):

<%= observe_field 'item[facility_id]', :frequency => 0.5,
:update => 'location_id', :url =>
{ :controller => 'item', :action=> 'refreshLocation' },
:with => "'facility_id=' + escape(value)" %>

Add something like this to the controller:

def refreshLocation
@facilities = Facility.find(:all)
@facility = Facility.find(params[:facility_id])
@locations = Location.find_all_by_facility_id(params[:facility_id])
render :update do |page|
page << text =""> :description} )
end
end

This tidbit in the form.rhtml is the ultimate target of all this work (this is the drop down we want to refresh)

Location:

<%= select_tag "item[location_id]", options_from_collection_for_select(@locations,:id,:description) %>

If I missed any code attributions from the various sources we pieced this together from, I'm sorry. Write me and I'll make good and give attribution where appropriate.

If you have any questions about this, post 'em and I'll take my best shot at answering.

And finally, thanks Sheri! Couldn't have done it without you!

Rails Projects Abound

2006-12-10T21:36:00.000-05:00

Okay, abound is a little strong. But we've got one new one and one to-finish Ruby on Rails project on our plates at work. The new project is a quickie tracking app for all inventory items containing human or animal tissue that get used in the operating rooms. Apparently, it's a JCAHO requirement that, as of January 1, 2007, all hospitals have to track all items that contain any amount of tissue - we've always had to track tissue-based implants: heart valves, bone chips, etc. The idea that was being floated was for a MS Access app, but I've grown way too tired of supporting those, plus with Rails we can now code a web app as fast as we can do a desktop Access app, all things being equal.

Politically speaking, it'll be very cool for us to pull this off in a two-week sprint, and as simple and straight-forward as it is, I'm thinking we'll be done much sooner, even with teaching Ruby and Rails to a couple of long-term Smalltalk/Java/Struts programmers. Technically speaking, the more Rails code we can get in production, the happier I'll be.

The other Rails app on our near horizon is to spend some time finishing Tart, the request management app that I've been learning and twiddling with for over a year now. It's an sort of an issue tracker that incorporates multiple approvals (IRB and departmental), and is tuned to our request process. It'll be good to get it launched and out there in people's hands.

Oh, the update I promised you in the last post? She interviewed very well, blew the socks off everyone she met, and she started on our team last week.

-Bill

Tags: RubyOnRails, Rails

Falling in love all over again

2006-10-25T16:04:00.000-04:00

You know, when you're hiring, and the perfect resume comes across your desk, you know it? Well, it happened today. Literally the perfect candidate fell into our laps. I talked to her, and she sounds like exactly (really, exactly.) what I'm looking for. She's got struts experience. She's got hibernate experience. She's got a serious value for test-driven development. Hell, she's got values as a programmer! She likes, understands, and wants to work in an Agile shop. From talking to her she's got a pretty laid-back personality. I think I'm in love.

Her interview is set up for Friday, and I'm really curious to see how she's going to click with the team.

Well, world.... I'll keep you posted on how it goes.

-Bill

Team Building as Magic

2006-09-04T07:54:00.000-04:00

I've recently been tasked with staffing our group back up to it's fullest potential. Recent comings and goings have left us with 2 staffed and 4 open positions, not including mine. As I've been wading through resumes and talking with candidates something struck me: building a team from near-scratch like this has a great resemblance to the strategic card games Magic and Pokemon. To start a game in one of these, you sort through your collection of cards (offensive, defensive, special use, environmental, etc.) and build a game deck of 15 or so cards that you play that game with. Part of the gameplay is luck, but far more of it is strategy, hinging on how you pick what cards go in your deck and how you play those cards as the game progresses.

Building a team for success at work is no different. I have dozens of likely candidates as potential choices, each with their technical and personal strengths and weaknesses. The team that I put together will determine our success over the coming months and years. In addition to core technical competencies, I'm putting emphasis on professional developer talents such as testing and agile team experiences, and I'm banking that that emphasis will serve us well. Other traits I'm finding valuable are a passion for craft, meaning I want people to be passionate about doing things the right way (over just getting them done), and people skills over technical skills, meaning that I don't want an uber-guru who keeps everyone around themselves ticked-off or can't communicate. Ideally you want someone with all these skills and all the ideal traits in one package (or even a whole team of folks like that), but those folks as rare and hard to find as heros in Magic. I think a solid heterogeneous team with just the right blend of skills in a team that can work well together is far more realistic and achievable.

I'm off to build my winning deck.

OSCON 06 Day 5 - Morning Sessions

2006-07-28T17:31:00.000-04:00

Session 1
Open Source Performance Monitoring Tools, Tips and Tricks for Java
Matt Secoske
matt@secosoft.net

Your project requires performance monitoring/planning/goals when your business requires it. You can do this through profiling, which is a focused look at system execution. To plan for performance: 1) determine your goals, 2) create testing scenarios, 3) determine monitoring / profiling needs. 4) Integrate this into your development process, and finally 5) integrate it into your production environment.

In planning, you need to know:

expected total number of clients

expected peak total number of clients

most common tasks these clients will be doing

acceptable response time

how long will the data stay around

and what you want to monitor:
hardware (web/app/db servers) cpu, memory, cache hit %, disk and network speed.
java specific (gc, app metrics)

JUnitPerf decorates JUnit test, great for benchmarking particular tests or test cases while refactoring, it's good at continuous performance testing, but not so good as a deployed monitoring solution. It's also available in JUnit 4.0 as an annotation.

The Grinder is a clusterable performance tester. It can do stress, load, capacity and functional testing. It can proxy traffic for recording and playback later as part of tests.

Apache JMeter does stress, load, capacity and functional testing. It's not clusterable and doesn't do proxy recording. It does have a plugin architecture for customization.

Log file analysis is another way to monitor performance. Since you're writing to disk it can have a noticeable negative impact on the performance you're trying to monitor. It requires changes to your source code, and doesn't accurately reflect how expensive your operation was, only the time required to execute it. If you're committed to logging, Aspects is a recommended way of doing it (AspectJ, AspectWerkz, Java Interactive Profiler, GlassBox Inspector).

JFluid / NetBeans Profiler - part of Sun's new JVM profiling tool, and also part of the NetBeans Profiler Extension. It supports local and remote profiling, and provides limited JVM support (mainly 5.0 and up).

On the other side, there's Eclipse TPTP. It does local and remote profiling, and requires a JVM agent for remote use.

Matt then did a brief demo of TPT in Eclipse, and a deeper demo of the NetBeans profiler, exploring many of the in's and out's.

He closed with a few tips and tricks:

Put in just enough metrics to get your performance measurements

Performance Test != Production

Real world data + real world usage patters + near-production environment = accurate benchmarks

Keep a little monitoring in production.

Session 2
Hacking Your Home Phone System (Year 2) - aka. Does the Phone Work Today?
Brian Aker

Terms:
PBX - private branch exchange
FXO - receives signal
FXS - generates signal
DMARK - demarcation point
PSTN - public switched telephone network

On wiring: put boxes in every wall, with 3/4 conduit (the largest you can fit
in a wall). Metal boxes survive better and are easier to mount. Run
electricity separately and cross at 90deg angles. More than 360deg worth of
conduit turns makes it hard to run cable later. Finally, don't leave boxes
empty for inspections - it worries inspectors.

Asterisk is an open source project that creates all sorts of phone
functionality in software (voicemail, conferencing, PBX, etc.) Digium makes
the best hardware (and funds Asterisk development). Skip the cheaper cards, and
get the TDM400P cards. Be careful of the FXO / FXS ports, as plugging the
wrong one into the wrong place can blow up the card.

Phone Instruments
BudgetTone: $40, cheap, cheap sounding, voice CID
Snom 190: $299, entirely scriptable via web services, can talk to multiple voip
services, no buttons for lines
Polycom SoundPoint: $200, good sound, buttons for lines, PoE, slow to boot

Analog to Digital Devices
Sipura 2000: 2port FXS analog adapter, $70
Sipura 3000: FXS/FXO bi-directional adapters, $70

The Computers: Any linux box will work.

The Software: Asterisk is hard to setup. The extensions file is the key to it
all. Plan on it taking a while.

The next generation is Asterisk@Home. The CD boots, wipes the drives, and
installs a working Asterisk based on Centos. It probably needs to be tweaked
after it installs, but it works. Recently they changed the name of the product
to Trixbox (which comes with SugarCRM built in).

High-availability: MySQL Cluster - set up a second machine, cluster them.

Mashups:

Front-door solenoid: unlock your doors via your phone

ipkall: free phone numbers!! (dial-a-song, a special number for certain
callers)

dial-a-monkey network

Livejournal's mod_mp3: good for creative content, freaks out business people

AIM Bot: with the follow-to-phone feature, it sends IM messages when messages arrive (configurable for certain folks, if that suits you.

Hardware Vendors
Digium - http://www.digium.com/
Polycom - http://www.polycom.com/
Snom - http://www.snom.com/
Sipura - http://www.sipura.com/

Links
http://voip-info.org/
http://www.asterisk.org/
http://www.planetasterisk.org/
FreeWorld Dialup - http://www.freeworlddialup.com/
http://krow.livejournal.com/ <- Speaker's blog.

-Bill

Tags: OSCON06, Asterisk

OSCON 06 Day 4 - Afternoon Sessions

2006-07-28T12:20:00.000-04:00

Session 3
Outer Joins for Fun and Profit
Bill Karwin

I've never fully understood Outer Joins. Although I can use them, I'm far more comfortable falling back to the (+) from Oracle and += from MSSQL to get what I want. Bill started explaining outer joins with set theory and Venn diagrams, which open source databases support which kind of outer joins. Then Bill launched into several examples, each explaining concepts of increasing complexity. Beyond the obvious stuff, you can do things like mimic a NOT IN subquery on platforms (MySql4.0) that doesn't support subqueries. You can also do a 'Greatest row per group' subquery without having to use the max() function. This lets you return more rows than just the max'd one. In english, its "show the row for which no other row exists with a greater date for the same product".

One of the more interesting demo/solution bits he showed was sudoku solving with SQL (using outer joins, of course...).

Session 4
Using the Google Web Toolkit
Bruce Johnson and Bret Taylor

Ajax, the same old arguments: nothing to download, every app is only a URL
away, desktop-like functionality. In reality, it auto-reinstalls every
full-page load, so it'd better be small. It's sorta secure; it's so dang hard
to get it working at all that security is almost an afterthought. There's a
plethora of technologies, on it's own platforms with it's own quirks - it's no
wonder developers hate Ajax.

Looking at Ajax from a Java viewpoint, Google set out to leverage their Java
knowledge, and make Ajax front-ends that are still very webby. They came up
with the idea of translating Java into Javascript. They actually pulled this
off, and it really works. GWT can run in two mode: hosted mode (the whole app
runs in the jvm as java - this gives you java-based debugging - which is
beautiful and useful!) and native mode (that runs in a os-native web browser).

GWT moves all the stateful session logic to the client. This enables stateless
load balancing and (thus) server clustering. It also leaves as much UI-only
stuff on the client, requiring no server round-trip. GWT provides leverage for
the solutions - you can take any solution and wrap it in a class and have it
to reuse later.

GWT, as Java, brings static type checking (static types... lalalalala....) and
all the benefits and drawbacks that typically come with it. There are code reuse advantages: create your ajax libraries as jars, reuse your code.

They provide a fantastic palette of UI widgets that make cross-platform rich
UI creation pretty simple, and one less thing to worry about.

The Web Toolkit comes along with a RPC library as well. It's dead simple, and
lets you create objects remotely, and serialize them back to the browser.

History and linking are ususally a casualty of Ajax. AWT lets you (with a pretty simple set
of calls) provide a full history support and linking. It really is dead
simple!

But wait! There's more!! There's JUnit support for testing of both the Java
and Javascript sides!

Downloadable at http://code.google.com/webtoolkit/
and the GWT Widget Library at http://gwt-widget.sourceforge.net/ and more.

Wow. GWT is cool. I can't wait to get it and play with it.

Session 5
Taming an Audience with Laser and Snake
Robert Stephenson

The basic idea is that you have a webcam, an iSight in Robert's case, and a
laserpointer. You point at the screen with the laserpointer, and the iSight
and your software work like a mouse to control the computer running the
presentation. In Robert's case, he leveraged the AppKit (a python interface to
Apple system calls) to call the CocoaSequenceGrabber to capture frames. Robert
chose a set of RGB factors to pick an appropriate range of laser colors to
have the app look for.

-1.1r + 2.0g -1.1b > 0.9

But this didn't work so well. The problem is that that there's not enough
resolution in the sRGB color space for the camera to be able to resolve the
dot of the laser pointer. And/or there's not enough range in the sRGB gamma
when ends up clipping the intensity of the laser dot. Owing to the way that
the camera's automatic gain control tries to normalize the camera exposure,
Robert tweaked the camera's sensitivity settings to compensate and was
able to reliably track the laser about the screen.

He then, with some ObjectiveC, C, Python and AppleScript, he was able to
control PowerPoint (or was it Keynote). Robert also added capabilties to
ajust keystoning and skewing of the image for the camera.

Session 7
Web Heresies
Avi Bryany

Seaside is a web development framework written in Smalltalk. Chances are
you're not going to go out and use Seaside. That's okay with Avi. He just
wants to talk about the heresies of web development. Avi's first idea is that
the HTML files belong to the developer, and that designers work with the CSS
files. He did a brief apples to apples comparison of language semantics so we
non Smalltalk people could read his examples. Naming things, while a good idea
when absolutely necessary, should be avoided if it can. You can use the object
that the field is rendering for to source it's own information. If you don't
name your objects, they get numbers, and a behind-the-scenes hash maps the
numbers back to their objects and their accessors for each numbered field.
Kinda confusing, but I can see the reasons why this would be a good idea.

An aside from Avi's talk: Seaside is based on Squeak Smalltalk, which I've been meaning to check out.
From my Smalltalk days, grumble11grumble years ago, I remember the Virtual
machine being HUGE and somewhat slow, and hard to crack for outside use. I'm
really curious to give Seaside a shot, even if just to see how it does what it
does. I'm intreagued as heck about how a Smalltalk web app server would fit
together and work.

Avi went on to cover a few demos, showing source code, and running a few hello
world-type things. A question that came up (as part of his talk) is how would
you serialize a session? The short answer is that you don't. Just use session
affinity instead. The cost of a server failing and taking a session with it
tends to be very low, so (essentially) screw it. Use sessions like crazy. Put
anything in a session you want. Really. Anytime. (I'm guessing this is another
herasy.) Try on the idea that you can also save the current execution point
and put it in the session. This would mean that you can throw the current
execution stack in the session (and in this case putting it in the URL) as the
form is displayed. When the form gets posted, you pull the execution stack out
of the session and continue execution from where it was, thus processing the
form. Think about that one and let that one sink in. Does your brain hurt yet?
It should. (This overhead isn't that much, but amounts to about 1M / current
user.)

Another Seaside goodie is that you can change your page load order by changing
the method order in your Smalltalk code. Relying on Smalltalk gives you some
really interesting powers that I've never seen in web applications before.
Your server image is still a fairly full-featured image, including the ability
to vnc into the sever image and interactively debug it. That's another "Think
about it." moment. That's cool.

Q: What's the lineage of Seaside?
A: WebObjects + Paul Graham = Seaside. It's also similar to Tapestry.

Q: What have been the changes to Seaside over time?
A: Seaside used to have a templating system. That was ditched 3 years
or so ago in favor of the programamtic html creation. There were also a few
minor architectural changes over time as well.

Q: Ajax?
A: Yes! http://scriptaculous.seaside.st/

-Bill

Tags: OSCON06, SQL, Outer Joins, Smalltalk,seaside,GWT,Google Web Toolkit

OSCON 06 Day 4 - Morning Sessions

2006-07-27T15:14:00.000-04:00

OSCON '06 - Thursday Morning Sessions

Session 1
Building Rails to Legacy Applications
Robert Treat

Of course greenfield applications are the easiest to use in Rails, but what about existing schema? How do you get rails to coexist and work successfully with Rails?

One option is to use your database to help fix the problem. Views can make legacy schema look far more rails-friendly. He demo'd that in PostgreSQL, you can create a rule that lets view write-backs work. Once you set these insert/update rules, you can treat your view as rails-friendly tables and thing should be far easier.

Using the database requires less knowledge of Ruby and Rails, and it does make things look like more what you expect to see from book examples. It does have the major drawback of having to support two schema.

The other option is to make rails smarter about how it That is, you can use ruby code to manipulate Active Record to match our data model. First off, edit your environment.rb to set

pluralize_table_names = false

. Another adaptation to make is to tell your model to use a differently-named primary key. That way rails knows to look for foo_id rather than just id.

Slides at http://www.brighterlamp.org/

Session 2
I'm 200, You're 200
David Sklar
sklar@sklar.com

In Web 2.0, we make lots of assumptions about other services being up, and who we can depend on and who we can't. So, what is your app dependent on? What contingencies do you have if it's not there.

Dependency is needinging something that's not yours (eg. physical control, organizational control, intellectual control). This includes server dependencies (content, per-machine hardware and software, internal- and external-network calls, content created by others), code dependencies (who wrote it, how it works, who wrote the documentation, who knows where the documentation is wrong.), business dependencies (who supplies your feed, how many of your folks are in the National Guard, what are your copyright and patent risk, what's in your SLAs and are the penalties helpful?).

You can mitigate real-time dependencies with planned modes of degradation based on data freshness, application features and read/write data. Avoid live web service calls when possible, instead making calls offline, which lets you sanity check and cache the results. You can also create a local data store for when the remote service has blips. To degrade features gracefully segment your app into non-interdependent parts. Further, you can build code with the idea of having pluggable external dependencies - eg use map provider Y instead of G, or ad protocol G instead of X. APP, S3 and "ad html" are already starting to fill that need. Monitoring your dependencies to know when to switch needs to happen.

In summary, examine your dependencies and your risk exposure, and plan for and mitigate those risks accordingly.

Slides at http://www.sklar.com/

-Bill

Tags: OSCON06, Ruby on Rails

OSCON 06 Day 3 - Afternoon Sessions

2006-07-26T20:11:00.000-04:00

Session 1
Driving Rails Deep into the Back Office
Fernandez

Financial presures to reduce costs are big.
Be ready for the build vs. buy decision because it will come up.
Obstacles: "good enough" legacy systems, BI tools, reporting packages/
Better and faster in rails? Yes!

One idea: the trojan application.

The Race approach: parallel or same teams do the same project in rails and java/.net/whatever.

The Pilot: do a small bit of an app as a proof of concept.

The Rescue: catch a failing project and redo it in rails

The Undercut: (risky) Come it, and because you're so productive you can do it in far less time than other solutions.

Case Study:
the "PCS" Project
-Small team
-Three months
-Replaced homegrown PL/SQL solution
-DSL-centric solution

Lesson 1: optimize and raise your levels of abstraction
How? Know what each piece of the stack does and doing a custom DSL.

Lesson 2: Rails really breathes life into XP
All the benefits of XP really click with Rails.

Lesson 3: Don't sweat performance and scaling. Most back-office sysmts have relatively few users.

Session 2
Streamlined
Stuart Halloway
Relevance LLC

In the beginning, there was ruby and rails...
Along came Streamlined, which replaces the basic scaffolding.

Streamlined is generates pages with a basic stylesheet, ajaxified creation/edit/update/delete, on the fly filtering of all fields, sort by clicking column headers, export to csv/xml,

New 'app/streamlied' directory that contains view code (that looks amazingly like rails models). It picks up object relationships from the rails models, and reflects them in the ajaxy goodness. But the new streamlined directory contains all the view-related stuff and putting it in one place. All the views (.rhtmls) are generated and fully open to being customized. If there's no customization, it

All the CRUD stuff gets refactored from each and every controller up to the streamlined_controller. All are overridable if need be. Or, if deleted entirely, there's an automatic fallback to a functional generic version.

Streamlined is Alpha
0.02 is out now
0.03 is out next Monday

Streamlined produces:
-Production-ready Enterprise scaffolding.
-Generic enterprise CRUD
-Simplicity of ActiveRecord for views and controllers

Generator options fall into 3 categories:
-semantic (--no-relationships, --no-views)
-look (--no-header, --no-about)
-theme (css=vendor.css or whatever)

UI Options
-relationship - choose view and summary
-user_columns - which columns to display

Licensed under MIT license.

Download http://www.streamlinedframework.org/
Documentation: same place
Submit patches: http://collaboa.streamlinedframework.org/

Things to look at:

jmatter - naked objects project

Session 3
Ruby for Java Programmers
Ugo Cei

Why?
-Leveraging exsiting Java libraries, source and infrastuctre

How?
-RubyJavaBridge http://arton.no-ip.info/collabo/backyard/?RubyJavaBridge
Works best if you wrap Java methods in simpler wrappers.

No mappomg for ruby iterators on java collections.

No date convertors

Weak ejb getter/setter property converters.

-SWIG - http://www.swig.org/
Jakarta POI, nifty library for manipulating Microsoft OLE2 Office fiiles uses SWIG to provide Ruby bindings. (http://jakarta.apache.org/poi/poi-ruby.html)

-JRuby - http://jruby/codehaus.org
Not a bridge, but a 100% Java interpreter written in Java
Gives ruby access to all Java libraries.
No access to ruby extensions in C
"Almost" able to run RubyGems and Rails
Quick Development pace.
Still slow compared to C Ruby, but quoting Charles O. Nutter "I think it's now very reasonable to say we should beat C Ruby performance by the end of the year."

Can use ruby "each" on Java collections

Data type conversions

Full support for JavaBean properties

Using the IRuby interpreter, you can call Ruby methods from Java.

-XML-RPC - It's possible to run the java and ruby process independently and enable them to communicate.

-SOAP - It's possible to run the java and ruby process independently and enable them to communicate.

Slides at http://www.sourcesense.com/transfer/ruby_for_java_programmers.pdf

-Bill

Tags: OSCON06, OSCON06, ruby, streamlined, jruby, Ruby on Rails

OSCON 06 Day 3 - Morning Keynotes and Sessions

2006-07-26T15:16:00.000-04:00

Keynotes

Of the morning keynote speakers, Tim O'Reilly had the most interesting and thought-provoking stuff to share. His talk keyed around 5 points that are he thinks are shaping the Open Source landscape today.

Architectures of Participation (aka. Web2.0)

"When the best leader leads, the people say 'We did it ourselves!'"

Open Source Licenses are Obsolete

Asymetric Competition

Operations as Advantage

"Being on someone's platform is becoming the same as being hosted on their infrastructure

Open Data

Random links from Tim's talk

Seaside - A web app framework in Squeak Smalltalk

Ning - A build-your-own-webapp for the masses

OpenFount Web2.0 app service that creates 2.0ishness via GWT and uses Amazon S3 for storage.

StumbleUpon - A new way of discovering web sites.

Session 1
Metaprogramming Java with HiveMind and Javassist
Howard Lewis Ship
hlship@gmail.com

Metaprogramming - Writing programs to write programs. Traditionally at compiletime: lexx, yacc, XDoclet (javadoc-style comments that affect program build), AspectJ.

Solution 1: Source Code Generation - Generate more source code at build time (XDoclet, ejbc) Awkward to write test to code that may not exists. Has a more complex build cycle.

Solution 2: Aspect-Oriented Programming - Metaprogramming with AspectJ: weave your code in with ohter code in an "aspect". Cotrol how inital code and aspect code works together. It changes your classes at build-time by adding method calls interspersed with your code.

Runtime Metaprogramming: Leave existing classes alone, and create new classes at runtime. Factories use configuraiton to create new classes and instantiate them. Done by Annotations.

HiveMind: Inversion-of-control container, much like Spring. Provides lifecycle to services: injection of dependencies, notification of lifecycle events. Driven by XML configuration. AOL via interceptors wrap implementations.

Javassist: APO library: Locat classes into memory as CtClass, modify them: add, remove, change, convert into Class objects. Don't have to learn bytecode ... Java-like syntax. (Part of JBoss)

HiveMind wrappers: Doesn't change existing classes, lots of proxies. ClassFactory is a simplified API wrapper around ClassFactory. Create new ClassFab instance, and add constructior, interfaces, fields to that instance.

Metaprogramming and Design Patterns - meta makes it easy to build code that implements useful design patterns easily (and more extisibly) on the fly.

-Bill

Session 2
Embedding a Database in the Browser
David Van Couvering
Database Technology Group - Sun Microsystems
david.vancouvering@sun.com

A database? In a browser? Why?
This is useful because....: the mobile user, keep personal data off the server, and it provides a fast local web cache.

What do you need to make this work: Java, embedable, small-footprint, standards compliant, secure, and provide automatic crash recovery.

Why a database? Standard, portable API (ODBC), ACID semantics, flexible data model, powerful query capability, and works with lots of tools, using existing skills.

What is Apache Derby? 100% Java Open Source relational database, http://db.apache.org/derby. 2M jar, with options to get it down to 500k.

Is this AJAX? Well, not really. All data is stored in the local derby implementation. Maybe LJAX?

To actually use this, your code has to be cert signed (to get acces to the local filesystem for the local databae). Can self-sign or get a real cert from the usual places. Your code then needs to be wrapped in a PrivilegedAction block.

Mapping Data to Fields: Java API with XML abstraction, call JDBC from Javascript, Java Persistance API, dojo storage abstraction,

Essentially: SQL results -> XML -> js into DOM fields.

You can even run this off of a USB stick. The derby format is portable and can be ready by any derby.jar. You can encrypt databases for security.

Alternate solutions: mozStorage (browser specific, intended for internal Mozilla use), dojo.storage (with Flash)

Future Directions: Web server in browser, Synchronization, Implement a dojo StorageProvider.

-Bill

Tags: OSCON06

OSCON 06 Day 2 - Afternoon Tutorials

2006-07-25T20:06:00.000-04:00

John Paul Ashenfelter - Rock-solid Web Development: Testing Web Apps

Take-home lesson #1 : Testing gives you confidence in your code and application.

Basic of Testing
Hierarchy of testing:
1. None.
2. Ad hoc testing - depends on people, not reproducible.
3. Unit testing
4. Bodies - help desk, users, managers.. have people beat on it.
5. Bodies + test plan - directs people on what to test.
6. Automated test plans.- gives the computer the boring, tedious parts.

Types of Testing: Low Level Code
Low-level testing done by developers to make sure an object behaves the way the spec says it should.

Done by programmers

Unit test are best examples

Specific functionality testing in isolation.

Types of Testing: Application Level Testing (also Functional or Integrated Testing)
Done by non-dev people... QA and UA types.
Also may involve automated testing.
Examples are browser interactions.

Types of Testing: System Level
Includes Load, Performance and Stress tests.

Types of Testing: User Level
Testing the stories from the cards - usability and acceptance testing.

See also: conformance, security, and failover testing.

Who does testing: You. Dev team, QA team, Help Desk. Not users / customers.

Take-home lesson #2: Do not let your users do your testing.

Getting started? If you're starting from ground zero (lotsa code, no tests) you can add tests as you write new code, add tests that demonstrate reported bugs, and add test instead of clicking through the app yet again.

There's also (from http://use.perl.org/~amoore/journal/30215)
1) boiling the frog - start slowly
2) play ping-pong - write a test for some code that someone's else wrote
3) maybe a rachet? - keep improving, and make sure you keep bettering your test standards.

Take-home lesson #3: Good programmer write tests.

Functional Testing with Selenium
Easy to use, runs in many browsers, exposes browser specific issues, straightforward to automate.

Speaking seleneese: you build tests in html as 3-column tables. The command language has Actions, Accessors, and Assertions. And there are locators and patterns.

Actions: anything the user can do, there's a selenium action for.

Accessors: examine the state of browser/application, usu storeSomething each has several (often 6) related assertions.

Assertions:
assertSomething - aborts on a failure
verifySomething logs failures and continues
waitForSomething waits until a timeout or condition (ajax)
There are also the inverse of all these.

Seleneese locations: it can find things by id, name, identifier, link, dom, and xpath.

Basic test structure is a html 3-column table: command | argument | argument, where the second argument is often blank. The first line is often a comment.

There's also an IDE wich can record sessions show test, and allow editing of existing / recorded tests.

Where this gets cool is that you can have something (Java, Rails, perl, etc.) generate the 3-column HTML that is your test. Doing this, you can build test that contain decisions, loops, and database referencing.

Take-home lesson #4: Selenium will save you time.

Continuous Integration and Automation
Integration with Cruise Control and your testing is a good and worthwhile thing
to do.

Use dbUnit to set up your database before tests and reset it when you're
finished. It's also useful for testing stored procedures and the like. You
know, the smarts you put into db code.

Take-home lesson #5: If you repeat it, automate it.

Load testing with Grinder
Load tests - how does the app behave with more than one user using it.
Performance tests - load it and measure performance metrics at typical usage
levels.
Stress tests - crank up the load until it breaks.

The Grinder (Use V3 and not V2. The Beta label is bogus and is really only
related to documentation):
Agent: that run the tests
Console: coordinate the tests in a central location
TCPProxy: records the test using a browser.

Personal note: Grinder does indeed rock - we've used it for some load-related
troubleshooting. But I realize we could be using it for lots more.

Talk slides are at http://www.ashenfelter.com/ or http://transitionpoint.com/

Apologies for this one being so erratic and unedited. This is pretty raw dump of my notes from the session.

-Bill

Tags: OSCON06, testing

OSCON 06 Day 2 - Morning Tutorials

2006-07-25T15:11:00.000-04:00

Stuart Halloway - Ajax on Rails

Stuart's smart and articulate, not to mention that he's a hometown boy.

Many of the recent successful companies (google, yahoo...) didn't come from vendor-supplied solutions. They evolved on Open Source, by having a good idea, being early to the solution space, and pursuing it tenaciously. AJAX (and likely on Ruby on Rails) is a great enabler for

97% of AJAX traffic on the web is html. But he lies. He pulls number out of his, erm.... thumb.

Prototype, the Library
It provides low level support for dynamic web apps, hiding browser oddities. It's used by Scriptaculous and Rico, and was driven and inspired by Ruby on Rails - the method names in it a RoRish already. It does xhr completely and provides some JS extensions. It does a bit of DOM and CSS/Behavior stuff, and in Stuart's model, it does 'view-centric' ajax (sending mostly html to the client).

Next we demo'd a basic type-ahead search app (think GoogleSuggest). There is lots of space for compromise, from doing full ajax (lots of server interaction) to pulling all results to the browsers and filtering things entirely on the client in js; there's all sorts of space for compromises in between. Doing ajax is about taking granularity from one page at time to much smaller mini-requests. For this demo, we talked through the Rails call to do this, then looked at the js that was generated as a result of that call.

Random tool aside: The Green Checkmark is the Firebug Firefox extension. Get it. Use it. It even has a Javascript debugger. Good mention too for the Web Developer toolbar.

The server side of the demo is in Rails; it does the database-y bit that does the search, then returns a rendered partial to the page with the results.

There are several useful XHR helper methods: link_to_remote, form_remote_tag, remote_form_for, observe_field, observe_form, submit_to_remote. These do a lot of common, useful, helperish stuff that you probably want to already be doing anyway.

Degradable Ajax
It's pretty straightforward to have an Ajax app behave like a conventional web1.0 app, albeit without the fancy typeheads and puffs and shrinks. If the ajax widgets call url foo, and the form submit url is foo, the action behind foo can be smart enough to know whether the request came from an ajax control, in which case it returns a partial, or if the form submits, it processes and returns the entire page.

Scriptaculous
Scriptaculous is a effects and widget library that builds on Prototype. It, too, is very Ruby- and Rails-ish in it's naming and parameters. The next demo and code review was a autocomplete field with Scriptaculous. The nextnext demo is a scriptaculous drag-and-drop demo. These libraries make it so darn easy those things that used to be such a pain.

RJS
All the code so far (and both libraries) falls flat at changing more than one section of the page at once. RJS does this, and it does it by sending js back to the browser to be executed. To get started do rake rail:update:javascripts. Add javascript_include_tag :defaults in your template page to get all the right libraries in your page. When you call something like this:


if @saved
  page.visual_effect(:blind_up, 'model_form', :duration=>0.5)
  page.replace_html 'model_error', 'Saved!'
else
  page.replace_html 'model_error', error_messages_for('player')
end
page.delay(0.5) { page.redirect_to(:action=>'list') }

rather than the obvious-looking calls back to the client. It generate javascript and sends it back to the client to make these things happen.

Streamlined
Stuart, after doing DHH's 7 minute rails demo in 3 minutes, regenerated and threw the same screens up using Streamlined, his companies gee-whiz ajaxy rails generator. It's great. It's beautiful. It's wicked easy. And they're officially launching it here at OSCON (tomorrow, I think). What Stuart demo'd was just a little tiny taste of what it can do (as witnessed at RailsConf), and he'll going to floor everyone tomorrow at the conference session.

Random tool aside #2: JavasScript Shell another useful Firefox extension. It lets you run arbitrary js against your page on the fly.

More on Protype
At a more raw javascript level, the prototype library helps make js behave very much like a real oo language, and helps smooth over the difference between different vendors browsers. Getting down to js at this level, while painful at time, is necessary to work around certain kinds of problems. Prototype's usefulness shouldn't be underestimated. It's not just for visual DOMish javascript, but it adds considerable Object (big O) support to javascript.

Technologies from this talk to look more at:

Streamlined - A fantastic ajax scaffold generator for rails.

Prototype Window Class - a js library for doing windows.

ARTS Testing Extension - good tool for testing rjs code

Stuart's presentation slides and sample code is available at codecite.com

-Bill

Tags: OSCON06, Ajax, RubyOnRails, Javascript

OSCON 06 Day 1 - Afternoon Tutorials

2006-07-24T20:12:00.000-04:00

Mastering VIM - Damien Conway

Alright, I've been in the tutorial session 5 minutes and I've given up on taking coherent notes. He's so smart, and so fast, and jumps around and covers material so quickly. I strongly prefer vi to emacs (I usually never install it). Over time, I've drifted away from vi, (I'm taking notes in and am just getting used to KDE's Kate) thinking, "Oh, it's too hard to worry with... Kate (or whatever) is so much easier". Damien has proven me wrong, and has shown me so many easy, powerful things to do with vim, that I'm excited and curious to give it a another go.

My favorite bits were:

Using do-it-yourself marks and vim's native marks to make jumping and editing easier.

The idea of binding a key to the :nohilite to hide the search hilighting when you're done.

Actually *getting* cut(d)/copy(y)/paste(p) by the vi keys (I always wimp and drop back to mouse/xwin click-drag-middleclick to do it)

]p to paste something at the current level of indentation

Branched undo in vim7, producing the "Trousers of Time"

Autocompletion!! (Well, I knew that it has always done it for filenames, but it can do it for vim commands and body text as well.) This gives us language-specific completions!

You can use vim as a file browser!

set autowrite - always save before quit

:options|resize - browse and set all options, then you can :mkvimrec to make a .vimrc to save all your current settings.

set shiftround

Damien's Total Tabular Control function. Very nice.

Automatic, self-cleaning backup files.

Visual Block Mode - swoon.

vipJ - Join all the lines in a paragraph together.

Abbreviations, tho they're not as good as insertion maps because they don't need the extra space after. This can be good and bad.

:nmap <Space> <something useful> - Make normal-mode space do something useful.

It was a great talk (a very difficult choice over the Asterix talk), and he's put together 50 tips - each a broad topic group of commands - in the handouts that are the real take-home goodies from this talk.

And a quick word on Damien: he's a great presenter. I got hooked on him at last year's OSCON, where he redefined Perl to work in Latin, with Roman numerals for results.

-Bill

Tags: OSCON06

OSCON 06 Day 1 - Morning Tutorials

2006-07-24T18:18:00.000-04:00

Scalable Internet Architectures - Theo Schlossnagle
Intro and Useful Points - (I'll backfill this intro as soon as I make it to my notes...)

Practicals - The problem: Scalable static image serving - eg. an in-house Ikami. We looked at a vendor solution, +/- of it. Next we did a build-your-own, looking at each of the pieces, and scaling the solution bigger and smaller. We talked through configuring each of the pieces, including a cool look at how to decide which image server is closest to the user geographically. By using multiple DNS servers, one near each image serving cluster. Things should converge... but they don't. Routes change too quickly for this to work. This does work if you use DNS Shared IP (AnyCast). All 3 servers all claim to serve the same IP - you'll always get to the closest server, with no convergence time. You can use a similar technique (if you own enough of the right, big, colocated parts) to make your system DDOS-resistant.

Logging - Again, we started with a hypothetical configuration and defined goals = multiple servers, real-time log analysis and reaction. Next he covered (dis)advantages of distributed logging, passive logging (sniffing), leading us to multicast logging. (JMS == perfect multicast logger at the app level) You can add multiple subscribers and loggers on the fly. This enables active and passive monitors, special purpose analyzers, and write-to-disk tasks.

Caching Architectures - caching can benefit application performance (and perceived performance) greatly. Theo covered layered cache, integrated cache (in app), data cache (in data store), write-thru cache. As before, he created an example case with example system configuration and goals of the exercise. In our sample bloggy-newsy app, when an article request comes in, we check to see if that page exists on the server, and if not, we fetch it from the db and create the page locally, so it exists for all future requests - much like we already do at Duke with patient discharge dates.

Tiered architectures - Theo started with a background on tiered design. Then he evolved into a worthy rant against traditional tiering. (Expensive people and $$-wise, hard to predict need and scale up/down.) A healthy replication system that allows any type of machine (web server) to answer any request. Scaling then just requires adding (or removing) like servers as needed. This sounds like the like of stuff that mongrel enables. Need more capability? Just stack it on like lego bricks.

Database Replication (part of Tiering) - It's a hard thing to deal with and do well. This is not warm-(or cold-) failover - this is true db replication (clustering). Multimaster replication is a log way off. Master-slave stuff is ready to use now. He made a mention of multi-vendor database replication - using mysql to cache and crunch 100 distributed instances, all dumping their data to a single Oracle backend.

The Right Tool for the Job - aka. How to Do it Wrong For our previous sample newsy-bloggy app, the customer wants a "which 30 users last loaded this page" and "which pages did a user load in the last 30 minutes". He talked through a mysql implementation of this, and how poorly it scales and how ludicrous the implementation would be. Next we created a custom app (a Skiplist structure in C) that hooks the logging stream (from above) and crunches it on the fly.

Q&A Session
Q: File systems?
A: There's GFS (at a cost)... Lustre is good. None are perfect, avoid them all if you can.

Q: Does spread have a performance cost?
A: No, negligible.

Q: I have huge sessions that resist distribution.
A: Go optimize them, make it smaller or compress it. Another option is to subdivide the session among different urls, and only get the big/expensive ones when you need them.

Q: Massive databases, how do you scale them?
A: Most obvious way is to federate (subdivide by geography/age/etc) to different databases.

Technologies from this talk to look more at:

spread - a group message service

whackamole - provides load-balancing and failover

mod_log_spread -

spreadlogd

Splash

Tags: OSCON06

Welcome!

2006-07-22T16:07:00.000-04:00

I've had this blog for a long while and have yet to post anything to it, intending for this to be my "professional" blog (counter to my thought-of-the-moment personal blog), with all my geeky inspirational wisdom, and the right time never seems to come. Last month it was very close, with RailsConf '06 in Chicago, but the network support for the conference and the hotel were laughable, and my blogging urges were thwarted yet again.

Now, I'm on my way to Portland, Oregon for OSCON '06. I came last year and had a blast. I got to see DHH do his Rails intro in person, I got to hear Why do a show, I got to ride a Segway, I got to meet a fair number of cool, geeky folks, and I got exposed to all sorts of wonderful, cool, nerdly ideas than I never would have if I hadn't come.

This year, I'm going without my wife. While it was great exploring Portland and the surrounding environs with her, I think I missed out on some fun after-hours social events. This year (for better or worse....) I get to be as geeky as I want to be, 24x5. I'm really curious what kind of ideas I'm going to end up with by the end of this.

So, welcome to my professional blog. I'm looking forward to sharing my experiences as a software development team lead, an explorer of new technologies, a Ruby on Rails enthusiast, and as a "veteran" software developer.