Category: Services & Servers

Post-Mortem: lncn.eu, 9 September 2012

Nick Jackson / / Services & Servers

What happened?

On Sunday 9th September our monitoring noticed that the Gateway site had become unavailable. Analysis provided by monitoring narrowed the problem down to the DNS not resolving correctly; our entire lncn.eu domain had been rendered inaccessible. The root cause of this was a DDoS attack against our DNS providers for the lncn.eu domain, PointHQ, causing their upstream provider to temporarily remove all their servers from the routing pool to mitigate damage.

What was affected?

PointHQ provides DNS services for the lncn.eu and lncd.org domains, meaning that any services using these domains were potentially unavailable. The most visible of these services are the lncn.eu address shortener, and the University’s Gateway service, although other services were affected.

As DNS caches expired any services using the affected domains were left unable to be resolved by end users, meaning that the services were inaccessible. Since gateway.lincoln.ac.uk implements a redirect to a lncn.eu subdomain, accessing Gateway using this domain was also affected.

What was done to fix the problem?

PointHQ were working to mitigate the problem throughout its duration, and DNS servers were restored later in the day. Essential records from the lncn.eu domain were also duplicated on the Rackspace Cloud DNS service, with the Rackspace DNS servers being added to the lncn.eu domain record to serve as a backup in the event of PointHQ becoming unavailable again later.

What is being done to stop this happening again?

  • The lncn.eu domain will retain at least one backup DNS server in its record, protecting essential services against a single failure.

Unlocking Gateway

Nick Jackson / / Projects, Services & Servers, User Experience

My most recent project (following on from Jerome, slotting in around the rest of the Summer’s “oh God, students are coming back, fix everything” mayhem has been to look at Gateway, more specifically to take Gateway and give it some extra awesome based around exploratory work we did with MPath. Here’s a quick breakdown of what you can expect when it’s turned on.

Very pretty. Very fast.

We’ve moved from a CWD 2.3 based design to our brand new CWD 3.0. This gives us a huge number of improvements in just about every area; layout, typography, readability, accessibility, compatibility, mobile readiness, new JavaScript frameworks, massively improved speed optimisations and more. We’ve compressed files and shaved off unnecessary bytes from almost the entire framework, making it load astonishingly quickly even over mobiles.

CWD 3.0 is also served over a blazingly fast content delivery network. Specifically we’re using Rackspace Cloud Files, who pipe their content to end users over the Akamai network. Put simply, this means that content such as the styling and images is delivered to your browser from a point much closer in internet space, regardless of where you are. If you want to access the Gateway from Bhutan then instead of serving all the content from a box in Lincoln some of it will come from whichever one of Akamai’s 84,000 servers happens to be closest. The result is a blisteringly fast experience, and since a lot of the Akamai servers are hooked straight into providers’ networks then it’ll still be quick even on mobile devices.

As mobile as your mobile.

We’ve made sure that Gateway optimises itself on the fly for most modern mobiles, and since it uses the CWD for its underlying design it’ll instantly take advantage of future improvements as we deploy them. We’ve sat down with our desks full of phones and tablets and tested things to make sure they’re easily read and are simple enough to use with just one finger.

Smarter.

Gateway now runs on a brand new system, meaning we can give it some extra smarts. If you visit Gateway and we’ve noticed there’s a problem with Blackboard then you’ll be told about it, meaning less clicking a link and waiting whilst it doesn’t load. It can tell you the local weather forecast, show you which trains are running late and even give you notices specific to your location, all in one place.

It’s all about you.

Sign in to the Gateway or use any of the services using Single Sign-In and it’ll gather all kinds of information you might find useful and display it for you. Your next lecture, assessment deadlines, how many library books you’ve got out and more is right at your fingertips.

Rock solid.

Gateway has moved from one very resilient platform to another even more resilient platform. Located off-campus on a world-class hosting platform Gateway can survive snow, flood and even builders cutting through power lines to provide you with updates even if everything else is going wrong.