Recipe to see changes in repo in real time

Today, I was playing with Markdown in a README.md file for a repository and thought it’d be helpful if I could see my changes in real time (this was a junk repo I was going to delete, so I wasn’t worried about a bunch of commits). Just thought I’d blog this for my own future reference.

Create a script to commit changes and reload Safari…

update.sh

#!/bin/bash

# Commit changes
git add README.md
git commit -m "formatting"
git push

# Reload Safari
osascript -e 'tell first window of application "Safari" to do JavaScript "window.location.reload(true)" in current tab'

Install fswatch (Mac’s version of inotifywait), a tool to monitor for changes to a file or folder.

brew install fswatch

With the following command, any change to our file will get committed and the page in Safari will be reloaded.

fswatch -o README.md | xargs -n1 ./update.sh

OpenMRS Data Model Browser

Ever since the beginnings of OpenMRS, we’ve used the data model as a reference and as a teaching tool.  As the number of tables has grown, it has become harder to keep the data model diagram updated.  I also wanted an easy way to search for tables, columns, or foreign keys.  So, I created dbtohtml to generate an easily browsable, standalone, singe-page HTML view of the data model.

@should do test-driven development

I recently described the @should taglet created by OpenMRS that helped the community adopt and sustain better testing practices.  Mário asked a good question about test-driven development (TDD):

While I don’t think we’re doing much TDD in the OpenMRS Community at this point, it would be great to evolve this direction.  The real question is: will the @should tags that helped us start testing our code become an impediment to TDD?  I don’t think so.

Let’s try a simple example to see how we could be TDD-ish with @should tags.  Imagine that we want to be able to get the age in years of a person:

class Person {
  Integer getAge(Date onDate) {
    return 0; // TODO: return age
  }
}

Before we write any code, we describe the expected behavior.  To keep the example brief, I’ll just describe a couple expected behaviors:

class Person {
  /**
   * Returns person's age in years.
   * @should return null for date before birthdate
   * @should not round up age
   */
  Integer getAge(Date onDate) {
    return 0; // TODO: return age
  }
}

Next, we invoke the Behavior Test Generator plugin to automatically do the busy work of generating the skeleton for our unit tests.

class PersonTest {
  void getAge_shouldReturnNullForDateBeforeBirthdate() {
    // TODO: write unit test
  }
  void getAge_shouldNotRoundUpAge() {
    // TODO: write unit test
  }
}

So, now we can write our unit tests and see them fail, like any newborn tests in TDD would do.  Granted, in this example, you don’t technically start with the test code, but you can start with describing behavior (using @should tags) prior to writing code and using those tests to drive development.  So, yes, we start with @should tags; however, @should tags, can precede any actual code, since they are effectively shorthand for the tests we are writing before coding.

@should do behavior-driven testing

In 2008, when OpenMRS was struggling to adopt better test-driven development practices, I was lucky enough to read Dan North’s Introducing BDD.  As Dan says:

How true!  For example, it’s common to see something like this when you start creating unit tests:

public class PatientTest {
  public void testPatient() {
    // test stuff here
  }
}

The next question is, what gets tested in a method called “testPatient”? I suppose the only wrong answer is “nothing.” But the problem is there are an infinite number of right answers… because “testPatient” doesn’t say anything about the behavior. As Dan points out, simply replacing the word “test” with the word “should” is a game changer. Let’s try again, except this time we will use “should” in our method name:

public class PatientTest {
  public void addIdentifier_shouldNotAddIdentifierThatIsInListAlready() {
    // make sure an identifier isn't duplicated
  }
}

It’s much easier to guess what will be tested inside that unit test’s method. That’s good… but it gets better. Dan’s suggestion of “should” not only places the focus on behavior, it also automagically forces testing to be scoped to a specific behavior, since any developer who sees a method name wrapping onto its third line instantly knows she is going about testing the wrong way and will look for help. Dan gives a great justification for this approach… but he had me at should.

Given Dan’s insight into using “should” instead of “test” to drive BDD, the trick was figuring out how we could engrain this approach within the OpenMRS community.  After some discussion, we came up with an idea that I’m still proud of today and I believe has helped us adopt a better testing culture.  Here’s what we did…

@should Javadoc tags

Testing is often filled with cookie-cutter code and requires additional effort that is difficult to sustain.  We wanted to find a way to overcome both of these challenges.  What we needed was a trivially easy way to generate behavior-focused tests.  So, we invented the @should Javadoc tag to allow developers to describe expected behaviors within the Javadoc and then we paid someone to develop an IDE plugin to auto-generate the test methods from existing method names.

Now that we have the @should tag, let’s take on more stab at testing.  Imagine you are writing some code for the Patient object…

public class Patient {
  public void addIdentifier(PatientIdentifier patientIdentifier) {
    // ...
  }
}

You know that an identifier shouldn’t be added twice for the same patient, so you simply state that behavior in the Javadoc:

public class Patient {
  /**
   * @should not add identifier that is in list already
   */
  public void addIdentifier(PatientIdentifier patientIdentifier) {
    // ...
  }
}

That’s it.  You’re already doing BDD!  Now, you tell your IDE to generate any missing unit tests for Patient and it automatically generates this method stub for you in the appropriate location:

public class PatientTest {
  public void addIdentifier_shouldNotAddIdentifierThatIsInListAlready) {
    // write your test here
  }
}

The IDE plugin automatically derives the proper location and method name from your @should tag and the associated method.  Now you can focus on testing that specific behavior without having to worry about any cookie-cutter code and adopting BDD is as simple as writing a Javadoc comment.

Benefits of using the @should Javadoc tag

• Low threshold to start testing behavior: just write a brief sentence starting with @should in your Javadoc.
• @should allows the documentation of behaviors to generate your unit tests.
• We can programmatically track test coverage by scanning Javadoc tags on methods along with corresponding test methods.
• There’s a low learning curve: “when you write a method, describe its expected behavior in @should tags, use the plugin to auto-generate the test methods, and fill in the blanks.”

Final Thoughts

We still have a long way to go down the road to full BDD, but I was very happy with our first step.  Over the years, the @should tag has become a handy tool for establishing a behavior-driven culture of testing; in fact, it has helped us adopt testing in general.  For any Java-shop that is wondering “How do we get our developers to start testing their code?”, I would strongly encourage you to read Dan North’s writings and consider adopting the @should Javadoc tag.

Related Resources

• Dan North’s Introducing BDD
• Example of using @should tags and the test automatically generated.
• Source for should taglet.
• Source for the Eclipse BehaviorTestGenerator plugin and IntelliJ GenerateTestCase plugin.

Freezing and Thawing Droplets in a DigitalOcean

DigitalOcean has been a game-changer for me.  Why create another space-hungry VM locally when you can spin up a new machine in 60 seconds on DO?  And it gets better: Tugboat.  Now, I can manage my droplets from the command line.  Since, I typically use (and reuse) droplets like I did local VMs, I often want to set a droplet aside for a while (maybe weeks or months) and return to it later.  Fortunately, DO provides a way to put snapshots into cold storage and then retrieve them later.  But freezing and thawing droplets wasn’t easy enough.  I made a suggestion to DO, but I doubt it’ll be implemented anytime soon (if ever), so I used Tugboat and some Groovy scripts to roll my own.

Here is what I was looking for:

Freezing a droplet

• Take an existing droplet, create a snapshot using the same name, then destroy the droplet.
• If a snapshot already exists with the same name, delete it (so freezing a previously thawed droplet replaces the previous snapshot).

Thawing a droplet

• As long as a droplet doesn’t already exist with the same name, create new droplet from a snapshot using the same name and start it up.

The goal:

$ # Assume we have a droplet foo
$ tugboat create foo
$ # Imagine you're done working with foo for now
$ freeze foo
$ # Foo is a snapshot & the droplet is destroyed.
$ # ... weeks pass and you have a hankering for foo ...
$ thaw foo
$ # Foo is back, Baby!

While it would be great to have freeze & thaw button on the DO website freeze & thaw parameters for Tugboat, I didn’t have the time to make a pull request for Tugboat… so here are the scripts:

~/bin/freeze

This script will snapshot a droplet, replacing any snapshot of the same name, and destroy the droplet.

#!/usr/bin/env groovy

class TugboatException {
	// Our very own little exception is born. It's a buoy!
}

def getImageInfo = {
	"tugboat images".execute().text.split("\n").find{
		it.startsWith(imageName+" ")
	}
}

def waitFor = { imageName, to='appear' -> /* to='appear' or 'disappear' */
	attempts = 0
	while (true) {
		attempts++
		imageInfo = "tugboat images".execute().text.split("\n").find{
			it.startsWith(imageName+" ")
		}
		if ((imageInfo && to=='appear') || (!imageInfo && to=='disappear')) break
		if (imageInfo || attempts > 20) {
			throw new TugboatException("$imageName did not $to within 3 min. Gave up.")
		}
		sleep(10000) // wait 10 seconds between checks
	}
}

def cmd = { description, command ->
	print description
	response = command.execute().text
	println "done."
}

def cli = new CliBuilder(usage:'freeze DROPLET_NAME')
cli.q(longOpt:'quiet', '')
def options = cli.parse(args)

if (!options.arguments() || options.arguments().size != 1) {
	cli.usage()
	System.exit(0)
}

imageName = options.arguments()[0]

if (! "tugboat droplets".execute().text ==~ /(?ims).*^$imageName\s.*/) {
	println "$imageName droplet does not exist"
	System.exit(1)
}

imageInfo = getImageInfo()
if (imageInfo) {
	imageId = (imageInfo =~ /id:\s*(\d+)/)[0][1]
	if (imageId) {
		cmd("Destroying old $imageName image...", "tugboat destroy-image -c -i $imageId")
		waitFor(imageName, 'disappear')
	}
}

cmd('Telling droplet to halt...', "tugboat halt $imageName")

cmd('Waiting for droplet to shut down...', "tugboat wait $imageName -s off")

sleep(3000)

cmd('Taking snapshot of droplet...', "tugboat snapshot $imageName $imageName")

print "Waiting for image to complete..."
waitFor(imageName)
println "done."

cmd("Destroying $imageName droplet...", "tugboat destroy -c $imageName")

~/bin/thaw

This script will restore a frozen droplet and start it up for you.

#!/usr/bin/env groovy

def getImageInfo = {
	"tugboat images".execute().text.split("\n").find{
		it.startsWith(imageName+" ")
	}
}

def cmd = { description, command ->
	print description
	response = command.execute().text
	println "done."
}

def cli = new CliBuilder(usage:'thaw IMAGE_NAME')
cli.q(longOpt:'quiet', '')
def options = cli.parse(args)

if (!options.arguments() || options.arguments().size != 1) {
	cli.usage()
	System.exit(0)
}

imageName = options.arguments()[0]

if ("tugboat droplets".execute().text ==~ /(?ims).*^$imageName\s.*/) {
	println "$imageName droplet already exists"
	System.exit(1)
}

imageInfo = getImageInfo()
if (!imageInfo) {
	println "Image $imageName not found"
	System.exit(2)
}

imageId = (imageInfo =~ /id:\s*(\d+)/)[0][1]
if (!imageId) {
	println "Unable to parse $imageName image id"
	System.exit(3)
}

print "Thawing image..."
response = "tugboat create $imageName -i $imageId".execute().text
println "done."

print "Waiting for droplet to start..."
response = "tugboat wait $imageName".execute().text
println "done."

Hack in the Right Direction

For over a decade, I’ve been using the phrase “hacking in the right direction” and, while I’ve explained it several times over the years, I was surprised to discover that I never blogged about it. So, here’s an explanation…

As any experienced coder knows, there’s some code that they are proud of and other code that is just there to get the job done. The latter is often referred to as “a hack” or “hacky code.” While it’s fair to say that coders wish all of their code could be beautiful, the reality of implementation requirements, delivery timelines, and competing demands nearly always force some amount of compromise. This leads many developers to approach projects in one of two ways:

Conditions	Approach
Sufficient timeline Critical code Shared or peer-reviewed code	Write good code
Short timelines Not critical code Short term need Used by a single persion/group	Hack

While it’s not always this black & white, most developers are balancing these two approaches in their head and their code is an amalgam of these two approaches. In OpenMRS, we often contrast “Development” with “Implementation coding,” where the latter tends to involve more hacking to meet time constraints.

How can we leverage these two approaches? Do these approaches have to contradict each other or can we make them complimentary? This is where hacking in the right direction comes in:

The next time life intervenes and prevents you from taking the time to generate beautiful code, before you just hack something out, take a moment to consider how you could hack in the right direction. This approach takes your ultimate goal(s) into consideration, but allows for guilt-free compromises.

The fundamental requirement of hacking in the right direction is understanding where you want to be. While a journey of a thousand miles may begin with a single step, it helps to take that step toward the destination.

How does this work in real life? I’ll give you an example that came up not too long ago. Imagine that you are asked to make a patient selection widget for an application. It would be pretty natural to have a method like this:

/* Get the selected patient */
public Patient getSelection();

But what if you took 20-30 minutes to review how the patient selection widget will be used in the application and learn that we want to use it for all patient selections, including for report generation where more than one patient may be selected? Then you would probably change your method to something like this:

/* Get the selected patient(s) */
public Patient[] getSelection();

It’s a subtle change, but it could have a dramatic effect on how you write and use your code. In this example, it’s a simple change to the method signature, so worth the effort. The trick is finding these easy wins without over-designing your hack. This approach will allow you to meet tight time constraints while at least building toward a longer-term vision.

It’s not too hard and the investment is small, even when time constraints are tight. The next time you need to hack, take a moment, and try to hack in the right direction!

IRCCloud Last Read Bookmarklet

If you are like me and participate in several IRC channels but don’t get to be in them all the time and sometime are away from them for several days, then IRCCloud is the perfect solution (a web-based IRC client that rivals any other and awesome mobile clients).  While IRCCloud makes it very easy to load a chunk of history (simply scrolling up or clicking on a bar), loading the history of a busy IRC channel for several days or weeks that can span multiple chunks of history is cumbersome.  I’ve bugged James a few times to ask for a click-once-to-load-all-history-back-to-last-read-message link.  When I recently suggested it again in the #feedback channel, I got some javascript tips from James that allowed me to do the next best thing: make a bookmarklet. Simply drag this link to your bookmark bar:

When you are using IRCCloud and enter a channel that you haven’t visited in a while (your last read message is several hours or days ago), click the bookmarklet and it will load history until it reaches your last read message.  In the rare case that you’ve been away for weeks or months and it takes more than a minute to load all the history you’ve missed, the bookmarklet will stop after a minute; just click it again to keep loading.

OpenMRS SDK

I’m very excited about the OpenMRS SDK and what it can mean for the OpenMRS Developer Community.

Chris Niesel (h3llborn) brought this beautiful creature to life during Google Summer of Code 2013 under the mentorship of Rafał Korytkowski.  Strong work, guys!  Looking forward to watching the SDK grow into both the first thing a new OpenMRS Developer touches and an invaluable tool for daily development of our most experienced developers!

Auditing OpenMRS repos in GitHub

OpenMRS has some basic conventions for its repositories in GitHub (within the openmrs org).  Basically, we add four teams to every repo (owners, full committers, partial committers, and repo owners) and we disable wiki & issues on the repo (since we already have a place for wiki & issues).

It’s easy for these things to get overlooked as new repos are added to the org in GitHub, so I made a little Groovy script to manually audit the repos.  Nothing fancy and it doesn’t really warrant a repo of its own, but I want to get the code off my laptop… so I’m blogging it. 🙂

The output should look something like this:

$ groovy AuditOpenMRSRepos.groovy
OpenMRS org has 78 repos
Owners team has 78 repos, missing none
Full Committers team has 78 repos, missing none
Partial Committers team has 77 repos, missing none
Repo Owners team has 77 repos, missing none
wikis or issues (should be empty): []

Audit OpenMRS repos

#!/usr/bin/env groovy

import groovyx.net.http.RESTClient
import org.apache.http.*
import org.apache.http.protocol.*
import groovyx.net.http.*

@Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.7.1')

def scriptDir = new File(getClass().protectionDomain.codeSource.location.path).parent
def token = new File("$scriptDir/github.token").text.trim()

// initialze a new builder and give a default URL
def github = new RESTClient( 'https://api.github.com' ).with {
  client.addRequestInterceptor(
    [process: { HttpRequest request, HttpContext context ->
        // using httpbuilders auth mechanism doesn't work, do it manually
        request.setHeader("Authorization", "token $token")
    }] as HttpRequestInterceptor
  )
  client.params.setIntParameter('http.connection.timeout', 5000)
  client.params.setIntParameter('http.socket.timeout', 5000)
  delegate
}

def getNext(response) {
  def next = null
  for (link in response.getHeaders('Link')?.value[0]?.split(',')) {
    def matcher = link =~ /< (.*)\?page=(\d+)>; rel="next"/
    if (matcher.matches()) {
      next = [url:matcher[0][1], page:matcher[0][2]]
    }
  }
  next
}

def fromGithub = { path ->
  def resp = github.get(path: path, headers:['User-Agent':'Groovy'])
  def data = resp.data
  def next = getNext(resp)
  while (next) {
    resp = github.get(path: next.url, query:[page: next.page], headers:['User-Agent':'Groovy'])
    data += resp.data
    next = getNext(resp)
  }
  data
}

def addRepoToTeam = { repo, team ->
  def resp = github.put(path: "/teams/$team.id/repos/openmrs/$repo.name",
  	headers:['User-Agent':'Groovy'])
}

def editRepo = { repo ->
  def body = """{"name":"$repo.name", "has_issues":false, "has_wiki":false}"""
  def resp = github.patch(path: "/repos/openmrs/$repo.name", body: body, 
    requestContentType:ContentType.URLENC, headers:['User-Agent':'Groovy'])
}

repos = fromGithub('/orgs/openmrs/repos')
teams = fromGithub('/orgs/openmrs/teams').findAll{ it.name != 'Transfer Team' && it.name != 'Release-test' }
println "OpenMRS org has ${repos.size()} repos"
for (team in teams) {
  teamRepoNames = fromGithub("/teams/$team.id/repos").collect{ it.name }
  missing = repos.findAll{ !(it.name in teamRepoNames) && it.name != 'openmrs-core' }
  print "$team.name team has ${teamRepoNames.size()} repos, "
  println missing.size() > 0 ? "missing from these: ${missing.collect{it.name}}" : "missing none"
  for (repo in missing) {
    print "fixing..."
    addRepoToTeam(repo, team)
    println "done."
  }
}

wikisOrIssues = repos.findAll{ it.has_wiki || it.has_issues }
println "wikis or issues (should be empty): ${wikisOrIssues.collect{it.name}}"
for (repo in wikisOrIssues) {
  print "fixing..."
  editRepo(repo)
  println "done."
}

OpenMRS Licensing

Background

In the process of upgrading the software license for OpenMRS, it seems like a good time to review how we got here and why we’re changing our license.  Here’s a brief history of OpenMRS Licensing…

• 2004 – OpenMRS is born as an informal collaboration between Regenstrief & Partners In Health starting with a data model, some code, and loads of experience.  None of us can even spell “open source,” yet we’re academics and go out of our way to work openly.
• 2005 – The OpenMRS platform starts to come together.  As we work openly, more interested parties join.
• 2006 – OpenMRS starts getting deployed
• 2007 – All contributors to the code (just shy of a dozen by then) formally sign over rights to code to the community.  OpenMRS is trademarked.  The OpenMRS Public License 1.0 is born.

OpenMRS Public License 1.0

Ok.  Let’s stop here for a second.  At this point, in 2007, OpenMRS transforms from a shared, unlicensed, pile of code within a public Subversion repository into officially licensed open-source software.  So, what is this OpenMRS Public License and where did it come from?  With the help of local lawyers, we hired a lawyer with expertise in open source licensing and described our goals:

The lawyer took these goals, reviewed the available open source licenses at the time, and suggested the Mozilla Public License 1.1 with a few tweaks to fit our specific needs.  So, we created the OpenMRS Public License 1.0 as a slightly modified version of MPL 1.1 and applied it to all of our code.

Living with OPL 1.0

• 2008 – 2010 – Life is good, but we run into a couple license issues: conflicts with other libraries (e.g., ExtJS) and some missed opportunities here & there.  Nothing horrible, but as time progresses, we start to learn some of the gotchas in licensing.  More importantly, we yearn to be using an official OSI-approved license.
• 2011 – We decide that we want to either switch to an OSI-approved license or get the OpenMRS Public License itself OSI-approved.
• 2012 – Efforts begin in earnest to explore an OSI-approved alternative.

OpenMRS Public License 2.0 Mozilla Public License 2.0 + Disclaimer

In 2013, with some hard work by Paul and help from OSI, Luis Villa, and some other lawyers, we discover that the Mozilla Public License 2.0 with a disclaimer could meet all of our needs, so we proposed the idea to the community.  Overall, people in the community are pleased to see us adopting an OSI-approved license.

Okay, OpenMRS on an OSI-approved license… at last.  That’s cool.  But why Mozilla Public License 2.0 (MPL 2.0)?

• MPL 2.0 is a natural evolution from MPL 1.1, meeting all of our original requirements while eliminating drawbacks of MPL 1.1.
• MPL 2.0 allows for a disclaimer, letting us add our medico-legal language without altering the license and remaining OSI-approved.
• MPL 2.0 is not only OSI-approved, but endorsed by the Free Software Foundation.
• There are more permissive licenses (e.g., MIT, Apache, LGPL) which would allow someone to adapt the core platform and profit from those changes without sharing their code.  While we make it easy for people to adapt the platform to their needs through modules (which can be closed), we also want to encourage work on the core platform to be shared back to the community.  So, we’ve chosen a slightly “copy-left” approach for the core platform to promote open collaboration on the platform.
• The MPL 2.0 is refreshingly simple to read/understand
  

 

See our license at http://openmrs.org/license.