Category: Open Data

So, what’s going on?

Nick Jackson / / CWD, Jerome, Linking You, Nucleus, Open Data, Service 2.0, Total ReCal, User Experience, Web 2.0

Good question. It’s been a while since I’ve blogged, so here’s a really quick overview of what I’m currently working on, pretending to work on, worked on but haven’t done anything with, or planning to work on.

  1. Linking You, our current JISC project on institutional identifiers. Finishing up next week, and currently causing Alex and myself epic amounts of beating our heads against the desks.
  2. Jerome, our other JISC project on making libraries slightly more awesome.
  3. Zendesk Phase 2, including bits and pieces of integration work to make it smoothly flow through everything else we’re doing.
  4. Nucleus (and assorted fluff), our epic store of everything, being brushed up, pinned down and fully documented.
  5. Authentication being made even cooler, and more reliable, along with support for more stuff like SAML.
  6. GAME, our application management environment, being made more awesome.
  7. Room Bookings will be coming over the summer, allowing people to find and book rooms faster than ever before.
  8. Lots of QR Code goodness all over the place, including on room labels (this hooks up to room bookings for added goodness).
  9. Possibly a bit of hardware hacking in the Library with RFID stuff.
  10. CWD updates to version 3. Faster, lighter, more accessible and generally good.
  11. Total ReCal rollout to replace our legacy Timetable system (we hope).
  12. Replacing the legacy phone book with the new one (we hope).
  13. Data, data, data.
  14. A bit of mucking around with telephony, just for kicks.
  15. Taking another look at our Student Communications project to try and address a few annoyances.

How Staff Directory Search Works

Nick Jackson / / Labs, Open Data, Projects

I’ve had a couple of people ask how my lunchtime project today actually works behind the scenes, so here’s the lowdown in easily-digestible speak. I should point out that I am relying heavily on two frameworks which we’ve already built at Lincoln. These are Nucleus – our heavy-lifting data platform – and the Common Web Design – our web design and application framework. These two gave me a massive head-start by already doing all of the hard work such as extracting data from our directory and making the whole thing look great. Now, on with technology.

Continue reading “How Staff Directory Search Works”

Who are you?

Nick Jackson / / Open Data

Today I’ve mostly been working on the magic of our user data collector for Nucleus, an awesome bit of technology which takes our slightly slow existing method of finding user information and replacing it with one blisteringly fast one based on our ever-favourite database Mongo.

What it does is – on a regular schedule – go through the entire directory letter by letter, collect all the users, and write their details to the database. How it does this, however, is a bit smarter than a bulk import in that it actually looks to see if the user has been updated or not, and records the changes. We can then use this data to do ‘push’ updates of user information – telling services which rely on user data that something has changed as soon as we can, rather than waiting for those services to have to look for changes themselves. We can also let those services do a ‘changes pull’, asking only for those records which have changed since a particular time. All of this combines to reduce network overhead and speed up processing by only sending changed details around, rather than a massive dump of all our data.

Coming soon to Nucleus will also be the first bit of cross-service collation as we begin to include data from students such as addresses and home email addresses. Where in the past this would require querying four different services, receiving a mix of data types and needing a lot of massaging to do anything useful we’ve done the hard bit for you. Even better, instead of giving insecure access to the data by providing direct database access, or blindly dumping the information, access will be controlled using the power of OAuth, giving us fine-grained control over exactly who can see what.

Let There Be Data

Nick Jackson / / Open Data, Total ReCal, Web 2.0

At the moment, Lincoln is standing on the edge of a huge change in the way things are done, at least as far as data is concerned. It’s been slowly pushed there by a small band of people (myself amongst them) who believe that one of the keys to making the world a better place is simply opening up data. Today it’s become clear to me that the availability of this data isn’t something that’s just wanted by an academic elite who want it ‘because it should be there’, but it’s something that’s wanted by people who just want to make things better.

Within hours of the University posting up a warning about severe weather, a Tweet dropped into my @mentions box from someone asking if there was the raw data for the warnings available. There’s already a student who’s wanting the not-yet-complete Total ReCal public timetable data for his own 3rd year project. Someone else was wanting to get hold of the GCW PC availability data. The list is endless.

So, what I’m going to try to do in some of my not-really-free time is to start the ball rolling for a website in the style of various other places around the world. Things like data.gov, data.gov.uk and data.open.ac.uk. I think data.lincoln.ac.uk should be fairly easy to rustle up. We could even use WordPress, and the whole thing is ready to go in under a day.

Who’s with me?

Cry Havoc and Let Loose the APIs of War

Nick Jackson / / Open Data

Those of you keeping up will know that my time at the moment is being mostly sliced up between Total ReCal, Nucleus, Jerome and Zendesk. Between them these four projects represent a monumental change in the way that a lot of things are done – that change is their reliance on APIs to achieve just about anything useful.

Today I’m going to focus on Zendesk’s integration with a few things, starting with hookups to a couple of new services we’re building to handle asset management and room information.

QR Code leading back to this site

Very simply, each asset which currently has a barcode on it will instead gain a QR Code containing a URL unique to the asset. This URL leads directly to a web based asset management system which a) gives us a way of updating assets stupidly quickly, and b) lets people report problems in three steps: scan the code, describe the problem, click ‘report’. The issue will be slotted straight into Zendesk over the API, with user details seamlessly completed from our Single Sign On service, and the asset number already completed.

On the other side, whenever someone opens a ticket on Zendesk with an asset number associated with it a cool bit of custom code will leap in and grab the asset details from our database so we can instantly see not only the details of the problem but also everything we know about the asset in question.

Rooms will similarly be gaining a QR Code which leads to the room’s entry on Nucleus Locations, from where people can view the room’s timetable, book a slot and tap a single button to tell us about anything which has gone wrong. A totally seamless experience for the end user, but which backs off to our full-blown helpdesk solution so we can track, manage and solve issues.

Continue reading “Cry Havoc and Let Loose the APIs of War”

It’s Just Awesome

Nick Jackson / / Open Data

Today and yesterday, between various other things, I’ve had a crack at solving Universal Search (mentioned previously), and I’ve currently got a bit of a mockup going at Labs. At the moment it’s only searching a very, very small set of test data but the entire supporting architecture is there; this is a working system, it just needs the content and a bit of polish on the front-end to handle some edge-case errors and IE6 (predictably). Obviously there’ll be some documentation on how to provide search sources and how to actually write some badass queries, but the majority of it is done.

It's the Universal Search!

The next step is to start plugging this in to external services. I’m going to start with the Library catalogue search from Jerome, making use of Sphinx’s distributed indexing model to avoid the need to double-up on data. Over time new services will be added, either sporting their own Sphinx search endpoints (the preferred way) or by outputting XML files for indexing by Universal Search itself.

Oh, and of course it comes with an API endpoint.

A Universal University Search

Nick Jackson / / Labs, Open Data, Web 2.0

During my time as a student I was faced with a great many challenges which involved some form of searching. Lots of it was academic, relying on the Library, Google Scholar and (gasp, horror, revoke his grades) Wikipedia. However, a lot of it was about more mundane stuff. Where exactly is AR1101? Who is John Smith, and what’s his phone number? What’s for lunch today?

The problem was that to find this information, you already had to know where to find it. Maps of the University are available on the Portal… if you know where to look. Phone numbers can be looked up… if you know the address for the service. The weekly menu is on the Portal… if you know where to look.

We’re left with a simply astonishing number of things which people may want to know about, but which is locked away as an image of a screenshot embedded in a Word document stored 14 levels down in Portal behind a page which nobody has access to, unless you happen to have asked for it. Rooms, events, books, journals, the Repository, blogs, people, news posts, lecture notes, the weekly menu and more are all available somewhere within the depths of a system. So, in traditional Nick fashion, I spent a few minutes in the shower this morning working out how to fix it whilst being refreshed by some particularly minty shower gel.

Continue reading “A Universal University Search”

Really Fast Rooms

Nick Jackson / / Labs, Open Data

Here’s yet another bit of Labs which we’re just kicking off – this time a big one (as in people who love data will love it). We’ve called it Nucleus, as in the centre of something, because it’s a little scheme of ours which aims to draw all the information which drives the heart of the University into one set of APIs.

The first very early look at our aim is the Locations API, or at least part of it. This is a RESTful interface (for those of you who know your types of API) to our database backend which runs on the frankly amazing MongoDB. The result is a really, really fast and flexible search of rooms.

To start with it’s been loaded with a small subset of the Spaces Database maintained by Estates, but in future this database will contain just about everything there is to know about a room. How many seats it has, which direction the windows face, if it’s air conditioned, if it has beanbags, when was it last carpeted and more could all (potentially) become available.

We’ll be adding application authentication to the Nucleus APIs soon (don’t worry, we aim to be pretty flexible about handing out keys) but for now the GET method to take a look at our rooms is open access. We know some bits of it are a bit flaky (floors) and some are just missing altogether (features, which worked with our 6-room test set but which we haven’t filled out data for the extended one) but we hope to improve those as soon as possible. Take a look at the documentation and see what you can make it do.

Continue reading “Really Fast Rooms”

Location Aware Lincoln

Nick Jackson / / Labs, Open Data, Web 2.0

One of the cooler things we’re working on at the moment is location awareness for our web applications. This lets us do useful things, like adjust page content based on where you are. The new Room Bookings system, for example, will detect your campus and automatically set your default location accordingly. Even things like the new Library service (“Jerome”) will be able to intelligently change things like library contact details and opening times based on where you are.

Give it a go. Visit http://location.ll.tn-uk.net/ from anywhere on campus (either wired or wirelessly) and it should tell you where you are. Sorry that it doesn’t look too pretty, but it’s basically a testing front-end to an API service we’re cooking up.