OpenMRS Data Model Browser
Ever since the beginnings of OpenMRS, we’ve used the data model as a reference and as a teaching tool. As the number of tables has grown, it has become harder to keep the data model diagram updated. I also wanted an easy way to search for tables, columns, or foreign keys. So, I created dbtohtml to generate an easily browsable, standalone, singe-page HTML view of the data model.
Freezing and Thawing Droplets in a DigitalOcean
DigitalOcean has been a game-changer for me. Why create another space-hungry VM locally when you can spin up a new machine in 60 seconds on DO? And it gets better: Tugboat. Now, I can manage my droplets from the command line. Since, I typically use (and reuse) droplets like I did local VMs, I often want to set a droplet aside for a while (maybe weeks or months) and return to it later. Fortunately, DO provides a way to put snapshots into cold storage and then retrieve them later. But freezing and thawing droplets wasn’t easy enough. I made a suggestion to DO, but I doubt it’ll be implemented anytime soon (if ever), so I used Tugboat and some Groovy scripts to roll my own.
Here is what I was looking for:
Freezing a droplet
- Take an existing droplet, create a snapshot using the same name, then destroy the droplet.
- If a snapshot already exists with the same name, delete it (so freezing a previously thawed droplet replaces the previous snapshot).
Thawing a droplet
- As long as a droplet doesn’t already exist with the same name, create new droplet from a snapshot using the same name and start it up.
The goal:
$ # Assume we have a droplet foo
$ tugboat create foo
$ # Imagine you're done working with foo for now
$ freeze foo
$ # Foo is a snapshot & the droplet is destroyed.
$ # ... weeks pass and you have a hankering for foo ...
$ thaw foo
$ # Foo is back, Baby!
While it would be great to have freeze & thaw button on the DO website freeze & thaw parameters for Tugboat, I didn’t have the time to make a pull request for Tugboat… so here are the scripts:
~/bin/freeze
This script will snapshot a droplet, replacing any snapshot of the same name, and destroy the droplet.
#!/usr/bin/env groovy
class TugboatException {
// Our very own little exception is born. It's a buoy!
}
def getImageInfo = {
"tugboat images".execute().text.split("\n").find{
it.startsWith(imageName+" ")
}
}
def waitFor = { imageName, to='appear' -> /* to='appear' or 'disappear' */
attempts = 0
while (true) {
attempts++
imageInfo = "tugboat images".execute().text.split("\n").find{
it.startsWith(imageName+" ")
}
if ((imageInfo && to=='appear') || (!imageInfo && to=='disappear')) break
if (imageInfo || attempts > 20) {
throw new TugboatException("$imageName did not $to within 3 min. Gave up.")
}
sleep(10000) // wait 10 seconds between checks
}
}
def cmd = { description, command ->
print description
response = command.execute().text
println "done."
}
def cli = new CliBuilder(usage:'freeze DROPLET_NAME')
cli.q(longOpt:'quiet', '')
def options = cli.parse(args)
if (!options.arguments() || options.arguments().size != 1) {
cli.usage()
System.exit(0)
}
imageName = options.arguments()[0]
if (! "tugboat droplets".execute().text ==~ /(?ims).*^$imageName\s.*/) {
println "$imageName droplet does not exist"
System.exit(1)
}
imageInfo = getImageInfo()
if (imageInfo) {
imageId = (imageInfo =~ /id:\s*(\d+)/)[0][1]
if (imageId) {
cmd("Destroying old $imageName image...", "tugboat destroy-image -c -i $imageId")
waitFor(imageName, 'disappear')
}
}
cmd('Telling droplet to halt...', "tugboat halt $imageName")
cmd('Waiting for droplet to shut down...', "tugboat wait $imageName -s off")
sleep(3000)
cmd('Taking snapshot of droplet...', "tugboat snapshot $imageName $imageName")
print "Waiting for image to complete..."
waitFor(imageName)
println "done."
cmd("Destroying $imageName droplet...", "tugboat destroy -c $imageName")
~/bin/thaw
This script will restore a frozen droplet and start it up for you.
#!/usr/bin/env groovy
def getImageInfo = {
"tugboat images".execute().text.split("\n").find{
it.startsWith(imageName+" ")
}
}
def cmd = { description, command ->
print description
response = command.execute().text
println "done."
}
def cli = new CliBuilder(usage:'thaw IMAGE_NAME')
cli.q(longOpt:'quiet', '')
def options = cli.parse(args)
if (!options.arguments() || options.arguments().size != 1) {
cli.usage()
System.exit(0)
}
imageName = options.arguments()[0]
if ("tugboat droplets".execute().text ==~ /(?ims).*^$imageName\s.*/) {
println "$imageName droplet already exists"
System.exit(1)
}
imageInfo = getImageInfo()
if (!imageInfo) {
println "Image $imageName not found"
System.exit(2)
}
imageId = (imageInfo =~ /id:\s*(\d+)/)[0][1]
if (!imageId) {
println "Unable to parse $imageName image id"
System.exit(3)
}
print "Thawing image..."
response = "tugboat create $imageName -i $imageId".execute().text
println "done."
print "Waiting for droplet to start..."
response = "tugboat wait $imageName".execute().text
println "done."
Auditing OpenMRS repos in GitHub
OpenMRS has some basic conventions for its repositories in GitHub (within the openmrs org). Basically, we add four teams to every repo (owners, full committers, partial committers, and repo owners) and we disable wiki & issues on the repo (since we already have a place for wiki & issues).
It’s easy for these things to get overlooked as new repos are added to the org in GitHub, so I made a little Groovy script to manually audit the repos. Nothing fancy and it doesn’t really warrant a repo of its own, but I want to get the code off my laptop… so I’m blogging it. 🙂
The output should look something like this:
$ groovy AuditOpenMRSRepos.groovy
OpenMRS org has 78 repos
Owners team has 78 repos, missing none
Full Committers team has 78 repos, missing none
Partial Committers team has 77 repos, missing none
Repo Owners team has 77 repos, missing none
wikis or issues (should be empty): []
Audit OpenMRS repos
#!/usr/bin/env groovy
import groovyx.net.http.RESTClient
import org.apache.http.*
import org.apache.http.protocol.*
import groovyx.net.http.*
@Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.7.1')
def scriptDir = new File(getClass().protectionDomain.codeSource.location.path).parent
def token = new File("$scriptDir/github.token").text.trim()
// initialze a new builder and give a default URL
def github = new RESTClient( 'https://api.github.com' ).with {
client.addRequestInterceptor(
[process: { HttpRequest request, HttpContext context ->
// using httpbuilders auth mechanism doesn't work, do it manually
request.setHeader("Authorization", "token $token")
}] as HttpRequestInterceptor
)
client.params.setIntParameter('http.connection.timeout', 5000)
client.params.setIntParameter('http.socket.timeout', 5000)
delegate
}
def getNext(response) {
def next = null
for (link in response.getHeaders('Link')?.value[0]?.split(',')) {
def matcher = link =~ /< (.*)\?page=(\d+)>; rel="next"/
if (matcher.matches()) {
next = [url:matcher[0][1], page:matcher[0][2]]
}
}
next
}
def fromGithub = { path ->
def resp = github.get(path: path, headers:['User-Agent':'Groovy'])
def data = resp.data
def next = getNext(resp)
while (next) {
resp = github.get(path: next.url, query:[page: next.page], headers:['User-Agent':'Groovy'])
data += resp.data
next = getNext(resp)
}
data
}
def addRepoToTeam = { repo, team ->
def resp = github.put(path: "/teams/$team.id/repos/openmrs/$repo.name",
headers:['User-Agent':'Groovy'])
}
def editRepo = { repo ->
def body = """{"name":"$repo.name", "has_issues":false, "has_wiki":false}"""
def resp = github.patch(path: "/repos/openmrs/$repo.name", body: body,
requestContentType:ContentType.URLENC, headers:['User-Agent':'Groovy'])
}
repos = fromGithub('/orgs/openmrs/repos')
teams = fromGithub('/orgs/openmrs/teams').findAll{ it.name != 'Transfer Team' && it.name != 'Release-test' }
println "OpenMRS org has ${repos.size()} repos"
for (team in teams) {
teamRepoNames = fromGithub("/teams/$team.id/repos").collect{ it.name }
missing = repos.findAll{ !(it.name in teamRepoNames) && it.name != 'openmrs-core' }
print "$team.name team has ${teamRepoNames.size()} repos, "
println missing.size() > 0 ? "missing from these: ${missing.collect{it.name}}" : "missing none"
for (repo in missing) {
print "fixing..."
addRepoToTeam(repo, team)
println "done."
}
}
wikisOrIssues = repos.findAll{ it.has_wiki || it.has_issues }
println "wikis or issues (should be empty): ${wikisOrIssues.collect{it.name}}"
for (repo in wikisOrIssues) {
print "fixing..."
editRepo(repo)
println "done."
}
What are the first 10 consecutive digits of e that form a prime number?
After seeing some sample Google Interview Questions that included this question, I couldn’t resist the challenge…
def bigE() {
BigDecimal e = new BigDecimal(1.0G)
BigDecimal temp = new BigDecimal(1.0G)
for (int i in 1..100) {
temp *= i
e += new BigDecimal("1")
.divide(temp, new java.math.MathContext(10000))
}
return e.toString()
}
def isPrime(long n) {
if (n < 2) return false
if (n == 2 || n == 3) return true
if (n%2 == 0 || n%3 == 0) return false
long sqrtN = (long)Math.sqrt(n)+1
for (long i=6L; i < sqrtN; i += 6) {
if (n%(i-1) == 0 || n%(i+1) == 0) return false
}
return true
}
def firstPrimeBySize(n) {
String e = bigE()
for (int i in 2..e.size()-n) {
String chunk = e.substring(i,i+n)
if (isPrime(new BigDecimal(chunk).toLong())) {
return "$chunk at position $i"
}
}
}
assert firstPrimeBySize(10) == "7427466391 at position 100"
Displaying OpenMRS observations using Groovy
I wanted to list out some observations for a patient and used the following Groovy script:
def sql(s) {admin.executeSQL(s,true) }
patientIdentifier = "999-3" // a test patient, of course
sql("""
select
date(o.obs_datetime),
(select name from concept_name where concept_id=o.concept_id limit 1) as question,
case c.datatype_id
when 1 /* numeric */ then cast(o.value_numeric as char)
when 2 /* coded */ then (select min(name) from concept_name where concept_id=o.value_coded)
when 3 /* text */ then value_text
when 6 /* date */ then cast(date(o.value_datetime) as char)
when 7 /* time */ then cast(time(o.value_datetime) as char)
when 8 /* datetime */ then cast(o.value_datetime as char)
when 10 /* boolean */ then if(o.value_numeric=1,'TRUE','FALSE')
else '?'
end as answer
from
obs o
left outer join
concept c
on c.concept_id=o.concept_id
where
o.person_id = (select patient_id from patient_identifier where identifier = '$patientIdentifier' limit 1)
order by
o.obs_datetime desc
""").collect{ it.join(": ") }.join("n")
which generated output like this:
2011-02-18: PATIENT HAD SEX IN LAST 6MO: TRUE 2011-02-18: PlAN FOR METHOD OF FAMILY PLANNING, DETAILED: ? 2011-02-18: QUANTITY: 5 2011-02-18: FAMILY PLANNING METHOD PLAN: INITIATION 2011-02-18: METHOD OF FAMILY PLANNING: ECPS 2011-02-18: HIV DISCLOSURE TO ANYONE, SPECIFIC: OTHER HOUSEHOLD MEMBER 2011-02-18: REASON FOR REFUSAL - FAMILY PLANNING: TRYING TO CONCEIVE NOW 2011-02-18: PlAN FOR METHOD OF FAMILY PLANNING, DETAILED: ? 2011-02-18: METHOD OF FAMILY PLANNING: MALE CONDOMS 2011-02-18: QUANTITY: 5 2011-02-18: FAMILY PLANNING METHOD PLAN: INITIATION 2011-02-18: METHOD OF FAMILY PLANNING: BTL 2011-02-18: FREETEXT, GENERAL: POSITIVE 2011-02-18: REASON FOR REFUSAL - FAMILY PLANNING: ABSTINENCE 2011-02-18: FAMILY PLANNING: TRUE 2011-02-18: REVIEW OF MEDICAL HISTORY: ICTERUS 2011-02-18: PATIENT REPORTED PROBLEM: YES 2011-02-18: CURRENT MEDICATIONS: ALUVIA 2011-02-18: REVIEW OF MEDICAL HISTORY: DEPRESSION
Sometimes you need a little web server
Here’s a simple jetty server using a short Groovy script:
@Grab(group='org.mortbay.jetty', module='jetty-embedded', version='6.1.14')
import org.mortbay.jetty.*
import org.mortbay.jetty.servlet.*
import groovy.servlet.*
import javax.servlet.*
import javax.servlet.http.*
class MyServlet extends HttpServlet {
void doPost(HttpServletRequest request, HttpServletResponse response) {
println "--- Received ${new Date()}n" + request.getParameter('data')
}
}
def server = new Server(8888)
def root = new Context(server, "/", Context.SESSIONS)
root.setResourceBase(".")
root.addServlet(new ServletHolder(new MyServlet()), "/")
server.start()
Running the script with groovy AtlasServer.groovy will accept HTTP POSTs at the designated port (8888) and dump posted data to the screen. For example:
$ curl -X POST -d "data=Hello world!" http://localhost:8888/
should dump “Hello world!” to the screen. Press Ctrl+C to abort the server.
Groovy thought
During some noodling about OpenMRS design issues with Paul, we wanted to look at the most recent version of some of our tables. Did I open up MySQL? No sirree! I have 1.6 dev running on an appliance, so I just opened the Groovy module and ran this script:
/* Let's make a convenient function for executing SQL */
def sql = { s -> admin.executeSQL(s, false /* read-only */ ) }
sql("describe obs")
.collect{ """
${it[0]}
${it[1]}
""" }
.join("n") /* separate lines */
Which dumps out this:
obs_id int(11)
person_id int(11)
concept_id int(11)
encounter_id int(11)
order_id int(11)
obs_datetime datetime
location_id int(11)
obs_group_id int(11)
accession_number varchar(255)
value_group_id int(11)
value_boolean tinyint(1)
value_coded int(11)
value_coded_name_id int(11)
value_drug int(11)
value_datetime datetime
value_numeric double
value_modifier varchar(2)
value_text text
date_started datetime
date_stopped datetime
comments varchar(255)
creator int(11)
date_created datetime
voided smallint(6)
voided_by int(11)
date_voided datetime
void_reason varchar(255)
value_complex varchar(255)
uuid char(36)
Translating Listserv archive to mbox format
We wanted to import our old mailing list entries from the OpenMRS mailing list Listserv archives into Nabble. No problem. Finding the GET listname FILELIST and GET listname file1, GET listname file2, … commands was easy enough. A quick search of Nabble support made it clear that I needed to send them mbox files. So, I set out in search of a Listserv to mbox converter. I found a couple scripts: one in perl and another in PHP. But trying them out, made it clear that I was going to have to do some tweaking. After a few near misses, I thought: “I could do this easier in Groovy.” So, I ended up with this script. Basically, it came down to leaving the messages and their headers alone and just adding a From_ line in front. Otherwise, the only tricky part was getting the dates right (GMT time without timezone specified in the From_ line and a some reshuffling of the date format in the message header).
Both for future me and anyone else who might benefit, here’s the script I ended up with:
import java.text.SimpleDateFormat
delim = '=' * 73 // LISTSERV separates messages with a bar of equal signs
foundDelim = false // we skip all content until first delimiter
inHeader = true // true when processing header data
def header = "" // holds current header data
dfListserv = new SimpleDateFormat("E, d MMM yyyy HH:mm:ss z")
dfHeader = new SimpleDateFormat("E MMM dd HH:mm:ss yyyy z")
dfMbox = new SimpleDateFormat("E MMM dd HH:mm:ss yyyy")
dfMbox.timeZone = TimeZone.getTimeZone("GMT") // for mbox, convert to GMT and drop timezone reference
cal = Calendar.instance
// Process input line by line from stdin
System.in.eachLine() { line ->
if (!foundDelim)
foundDelim = (line == delim) // skip until we find first delim
else if (inHeader) {
// within header
if (line =~ /^s*$/) {
// empty line signals end of header
// fetch Date from header and reformat it for output
m1 = header =~ /(?ms)^Date:s+(.*?)s*$/
date = dfListserv.parse(m1[0][1])
cal.time = date
mboxDate = dfMbox.format(cal.time)
headerDate = dfHeader.format(cal.time)
// fetch From from header
m2 = header =~ /(?ms)^From:s+(.*?)s*$/
fromHeader = m2[0][1]
leftBracket = fromHeader.indexOf('<')
rightBracket = fromHeader.indexOf('>')
if (leftBracket > 0 && rightBracket > leftBracket)
from = fromHeader.substring(leftBracket+1, rightBracket)
else
from = fromHeader
// output header with mbox-required From_ line up front and reformatted date
header = "From $from $mboxDaten" + header.replaceAll(/(?m)^Date:s+.*$/, "Date: $headerDate")
println "$headern"
inHeader = false // no longer in header
header = "" // clear for next message
} else {
header += "$linen" // accumulate full header data
}
} else if (line == delim) {
// if we find a delim, begin processing next line as header
print "nn"
inHeader = true
} else {
// within a message, just send it through untouched
println line
}
}
Making a groovy module for OpenMRS
No, I’m not smoking anything and I’m not trying to do a Greg Brady impersonation. By “groovy” module…I mean a Groovy module. 😛
Groovy has the potential to bring the fun and rapid prototyping benefits of languages like Ruby to the Java scene. But a big advantage of Groovy is that it’s built in Java and designed to both work with Java and give Java programmers a fairly easy introduction to the fun of higher level scripting.
So, this Groovy module is a just testing the waters. Groovy is incredibly easy to embed into any Java application (they make it ridiculously simple…a single JAR library is all it takes). So, I made a simple little module that embeds Groovy into OpenMRS and gives a single web page with a textarea where you can play with Groovy scripting. The module adds a “Groovy Scripting” privilege to OpenMRS that you could assign to users…but I wouldn’t recommend it. Currently, this is for testing only and shouldn’t be used in production until we have some more experience with it.
So, fire up a copy of OpenMRS, install the new Groovy module and let’s take it for a spin.
Once the Groovy module is installed, you should see a link in the OpenMRS administration page under a new “Groovy Module” section. Clicking on that link should bring up a simple textarea dialog like the one shown above. Sorry…but that’s about as fancy as it gets for now.
There are several special variables automatically bound to the context for you: cohort, concept, encounter, form, locale, logic, obs, patient, person, and user. Most of these are bound to the corresponding service; locale is the locale of the current context, which is handy for some API calls.
So, type the following into the textarea and click the GO button:
p = patient.getPatient(2)
println p.givenName
That should display the given name of patient #2 from your database. Here are a few more examples of Groovy scripts you can copy and paste into the textarea:
Want to see a 100 concepts?
import groovy.xml.*
writer = new StringWriter()
b = new MarkupBuilder(writer)
b.table(border: 1) {
for (i in 1..100) {
c = concept.getConcept(i)
tr { td(c.name, style:"font-weight:bold"); td(c.name.description) }
}
}
println writer.toString()
How about some encounters?
p = patient.getPatient(2)
println "<h2>${p.givenName} ${p.familyName} Encounters</h2>"
for (e in encounter.getEncounters(p)) {
println "${String.format('%tF', e.encounterDatetime)} - ${e.creator}<br />"
println "<ul>"
for (o in e.getObs()) {
println "<li>${o.concept.name}</li>"
}
println "</ul>"
}
Or dump out some observations.
p = patient.getPatient(2)
println "<h2>${p.givenName} ${p.familyName}</h2>"
for (o in obs.getObservations(p, false)) {
println "${o.concept.name} → ${o.getValueAsString(locale)}<br />"
}
Any patients with “nga” in their name?
s = "nga"
for (p in patient.getPatientsByName(s)) {
n = "${p.givenName} ${p.familyName}".replaceAll("(?i)($s)", "<b>\$1</b>")
println "${n}<br />"
}
You can even manage modules…but don’t stop the Groovy module!
import org.openmrs.module.*
// Uncomment the next line to start the printing module
// ModuleFactory.startModule(ModuleFactory.getModuleById("printing"))
// list out your running modules
for (m in ModuleFactory.startedModules) {
println "${m.name} : ${m.started}<br />"
// uncomment the next line to stop the printing module
// if (m.name == "Printing") ModuleFactory.stopModule(m)
}
As you may gather from trying the examples above, I’m redirecting the stdout from the script into a <div> on the page as HTML, so you can play with formatted output. If you want to simply print some text within Groovy and preserve whitespace, then just put println “<pre>” and println “</pre>” around your println statements. Or use <textarea></textarea> instead.
Learning Groovy (especially for Java developers) is pretty easy. See the Groovy documentation for more about the language. Remember that you don’t need get and set — e.g., person.givenName will execute person.getGivenName() for you. And you can enjoy a break from the semicolons. 🙂
The real fun with Groovy is that you can throw in good ol’ import statements and directly reference java classes whenever you want.
What next?
While the ability to execute some scripts on the fly can be fun…maybe even handy for testing out code during development, my real hope is to evolve toward a Grails module that could facilitate rapid prototyping of web forms or quick & dirty report pages. The ultimate would be a Grails module that provides a quick and easy starting point, let’s you edit and extend it without any re-compile steps, and then provides a button to download a snapshot of the current state of the module as a new omod file to facilitate rapid prototyping of OpenMRS modules to target quick & dirty development needs and/or prototype new functionality.
Now that would be groovy!