Introduction to the Disruptor Slides

I’ve produced a very cut down version of the presentation I’ve been giving at a lot of conferences, giving a high level overview to the Disruptor.  This serves as a quick intro to the concepts behind it.

My slides are usually pretty useless without me (or someone else) talking over them, so for more context don’t forget there’s always my original blog posts (the Magic Ring Buffer, Reading from it, Writing to it, Wiring it up), which are now pretty dated, and the Java Magazine article I wrote at the start of the year.

Strangeloop Disruptor Workshop Materials

On Sunday I gave my very first workshop on the Disruptor.  The aim was to give people some hands-on coding experience using the syntax.  Because time was limited (you can’t get people to build an entire application architecture in 2.5 hours) the example is somewhat contrived, and needs a big leap to make it into a proper application context.  But the workshop should:

  1. Give an overview of the Disruptor
  2. Show how to create a simple one producer, one consumer example.
  3. Show how to wire up a parallel event handler
  4. Show how to how (and why) to create a diamond dependency graph.
  5. Extrapolate beyond these very simple examples to something closer to a real world architecture.
Requirements are:
  1. Basic Java skills
  2. Java 7 update 7
  3. (Optionally) IntelliJ.
The slides (not all that useful, I’ll grant you, without me talking) are available, and you can download the code.
If you want to work through the examples yourself, start with com.mechanitis.towerdefense.TowerDefence.  The starting point code for the workshop is in package com.mechanitis.towerdefense.  com.mechanitis.towerdefense1 shows a solution to exercise one, com.mechanitis.towerdefense2 to exercise 2 and so forth.  The final package, com.mechanitis.towerdefense4, has some different towers you can play with to see the effects.  They don’t really show anything special about the Disruptor, but they were fun to write and are interesting to play with.

QCon London 2012

I’m late with my write-up of QCon, and what’s worse, it will be partial - “sadly” I was in Lanzarote on a training week with the running club from the Thursday (8th) so I missed most of it.  A sacrifice I had to make for 7 days in the sunshine….

Firstly, me me me
I presented the talk I previewed at Skillsmatter the previous week, something I was calling the User’s Guide to the Disruptor, but actually turned out to be how-can-Trish-fill-95-slides-with-pictures-and-finish-in-under-40-minutes.
The audience was different to the Skillsmatter event, not surprisingly.  What was surprising is that I expected people at the conference to be less aware of the Disruptor, and those who came to the Disruptor-only LJC event to have had more exposure to it.  It was a (pleasant) surprise to see how many of the standing-room-only audience had not only heard of the Disruptor but had read stuff about it (I always love it when people have read my blog), played with it and were even using it in anger.
Because of that, I think if anything the talk did not go into enough detail, or enough new stuff, to please everyone.  Tough crowd!  But it was gratifying to hear the audience correct me in some of my answers, and answer other people’s questions - it’s always nice to know people are listening.
Of course, I will post a link to the presentation when it’s available.  For now only the slides are online.
I enjoyed QCon
QCon was noticeably different to the other conferences I’ve been to in the last six months.  For one, it’s not a Java conference - sure, I was hosted on the Java track, but QCon is wider-ranging than just one technology platform.  I’m not sure if it’s because of this, or because it was based in the notoriously impatient London, but I felt like there was a message of “Look, let’s just get stuff done, OK?".  Ultimately we get paid to deliver stuff for the business, and since my favourite question is always “but what are we trying to achieve?” I like to hear ideas around how to actually deliver.  Don’t get me wrong, I’ve loved the technology conferences - I like to hear about new stuff I had no idea about, and I really like the community vibe from them.  But it’s a nice change to be shaken up into thinking exactly why we do all this.
The Data Panorama
Firstly, Rebecca Parsons and Martin Fowler from my old employer ThoughtWorks put Big Data into perspective.  Previously I hadn’t really cared about it - we process lots of data at LMAX but we don’t really have to dig into it, so Big Data is not top of my “oooh I really worry about that” problems.  There were quite a few interesting points I took out of this:

 - In the past, it was easy (and possibly even correct) to model the whole application based on the data you were collecting or manipulating (and probably storing in a relational database).  These days it’s not just the data from your app you need to worry about (and that can get big enough), but also all the news, blogs, twitter, and Facebook stuff generated by you and about you.  In addition, your data might not even be located with your app - the cloud has made the physical question of location redundant.  All of this pushes you towards an architecture which has to separate data from the application, and forces you to ponder your design.  I heard good arguments for Domain Driven Design"" here, which is nice because we like that at LMAX.

 - Reporting and analytics on Big Data must be more fluid.  There’s so much data about you, your users, your application, out there that you don’t even know the questions to ask.  Instead you want to be able to spot patterns in data you didn’t even know you had,  I thought this was dead interesting - I studied Computer Science and Artificial Intelligence at university, and we were told data mining using AI was going to be fundamental to companies who wanted to be on the bleeding edge.  Only now is it looking like people realise it’s becoming that important.
 - Martin referred to Data Scientists and said he was suspicious of scientists.  I’d been reading The Black Swan"" on the train in that morning and couldn’t help but wonder if he’s read the same thing - that was talking about how you should treat someone with suspicion if they suggest you can apply science and logic to anything that is… well, actually to anything other than actual science (by which he meant physics I believe).  By saying it’s a science you suggest it’s predictable and follows rules.  And if it was predictable, it wouldn’t be hard.  Big Data is anything other than predictable - the data could be corrupt (it’s safest to assume some of it is); it’s generated by people (and we all know how fickle they are); and if you’re collecting lots of it practically randomly, then cause/effect/correlation are not guaranteed.  So Martin suggests the term Data Journalist.  There was a storm on Twitter which suggested a certain amount of disagreement with this term, but I like it.  But then, I like to pretend I’m a writer and not a programmer.
 - They gave some examples of using data to drive economic growth (e.g. Kenya) - what I thought was interesting about this is that it was a win-win situation - expose the data to grow technology skills, but get a lot of interesting/useful/socially responsible applications in return.
 - Something I liked was about the idea of embracing “bad” data, stuff that’s not trustworthy for whatever reason.  You can assume that the good data overwhelms the bad but you can’t be sure.  By looking for more fluffy patterns, vague correlations, in your data, you might expose something interesting even if the data’s not “correct”.  As humans, we can’t expect that we won’t make a mistake - it’s better assume we will and work out how to deal with it.
 - There was a call to not passively accept requirements, but to play an active role.  I like to think this ties in to my post about working with your customer.  But then I would.
 - I got a lot from this keynote, even if it was just a feeling of “I knew it!  I was right!". 
Highly available systems in Erlang
Joe Armstrong was a very interesting person to listen to, clearly someone who’s been there and done that. Even without the presentation, the slides are interesting in their own right as they contain a lot of information and guidelines.
The points I took away are:
 - If part of the system fails, it’s not up to that part to fix itself.  You need special help to deal with failures.  If you fell over with a heart attack, you wouldn’t try and heal yourself, you’d get a medic
 - Isolation between threads/programs will mean that those different things cannot interfere with each other (i.e. no shared state).
 - My favourite quote was “If you make things synchronous you’ll bugger things up”.  In theory, it’s so much easier for us humans (programmers) to think synchronously.  But whenever you design your system to by asynchronous, you find that your system actually becomes simpler and not more complicated.
JVM performance optimizations at Twitter’s scale
 - I had a terrible view in the fully packed room, so I was just picking up phrases. I heard Attila mention the Uncanny Valley, a phenomenon I was introduced to by my sister when she was studying her Cybernetics PhD (Google it, I found it fascinating).  
 - There was a lot of really useful information about how the Java GC works.  It seemed to back up the (rough) premise we work on - stuff that’s very short-lived is fine, and stuff that lasts “forever” is fine - it’s the stuff in between which  cripples your system when it keeps getting shoved around.
Decisions Decisions
Dan North was, as always, an excellent speaker.  He entertained us but got us all thinking.  Five years ago at QCon London (my very first conference ever, and the thing that motivated me to start a professional blog), I saw Dan speaking and I was inspired to think about my working practices.  He was talking about BDD at the time, which was a relatively new concept to me.  I came away from that QCon with clearer ideas of what awesome development practices should look like.  Never did I imagine that five years later, not only would I be working in an environment which follows a lot of those practices (and pioneers many more) but that I would actually be speaking at the same conference.

I’ve come a long way since then, and of course, Dan isn’t talking about the same things either.  I’ve been to a lot of conferences this year, and I heard nothing really preaching to the “meta-agile” LMAX (still have a post pending about our agile practices…).  When it comes to agile, there seem to be very few people who we can learn off - don’t get me wrong, there are lots of things we want to improve on, which is why we’re looking for people to learn off.  But most people are still preaching TDD and we want to know “what next?".

Well, Dan took everything we thought we knew, and ripped it to pieces.  Taught us to challenge everything we think we know.

It was irritating actually because he had no answers.  But he did tell us that the answers we think we already have might not be the right ones….

Well worth watching his talk when it comes online, I really can’t summarise it here.
Developers Have a Mental Disorder
I nearly missed this ending keynote.  I’m so glad I didn’t, Greg Young is awesome at ranting.  He said developers have a disease - we overcomplicate things, when we try to simplify things.  We want to abstract stuff, we love to look for patterns and reuse when actually sometimes we just need to solve the problem.

In my notes I have “People come to conferences for answers, when they should be remembering to use their brains”.  I assume that’s a quote from him, and not a comment I thought of at the time, but I wholeheartedly agree with it - if a shiny new technology solved your problems, you’d be out of a job.  Your job is to take the problem and figure out how to solve it.  Not to drag and drop an answer into place.

Another talk that I can’t do justice to, watch the video when it comes online.

…and finally….
At the end of the day, the Atlassian-sponsored community night was awesome.  I got to chat to (be prepared for gratuitous name dropping here) Martin FowlerSimon BrownLJC colleagues Martijn and JohnZero Turnaround guys; a couple of ex-colleagues from Evolution / Detica; a heap of LMAX and ex-LMAX guys; and, of course, bundles and bundles of new interesting people.
Summary (i.e the short version for those who can’t be bothered to read this whole post)

Maybe it’s wishful thinking, but the messages I took from QCon were:

  • If you want to solve the problem your business has, you might want to model your system around their world.  Funny, that sounds suspiciously like Domain Driven Design.
  • Synchronous is bad, mmmkay? 
  • Hardcore understanding of what the computer is really doing seems to be coming back into fashion.  Hmmm, I wonder who started that…? (tongue firmly in cheek, we can’t have been the only ones)
  • "You can’t tune something you don’t understand” - testing and monitoring is kinda important.
  • Our business is all about trade offs.  There is no perfect solution, they pay us because it’s very very hard to work out something that’s Good Enough.
  • Ultimately it feels like back to basics: understand the problem; model the domain, and have sympathy for the hardware that’s running your solution.
My Corporate Bit
QCon overall turned out to be a bit of an LMAX fest in the end, with Mike & Martin and Andy Stewart (our Chief Lord Business Analyst), all giving presentations there as well as me.  It’s nice to be on home turf, and it’s very cool to see that we have such a range of things to talk about that so many of us are invited to speak.
PS
I just found out there’s a QCon in New York.  My invitation seems to have got lost in the post.  Don’t suppose anyone wants me to speak there…?

Java Magazine: Intro to the Disruptor Part One

This month’s Java Magazine features an article by yours truly, which is yet another intro to the Disruptor.  It’s basically a summary of the stuff I’ve written in this blog, updated for version 2.7 - so the names of the classes should be up to date and the responsibilities follow the simplified pattern we use now.  If you were looking for an more recent version of my introduction blog posts, this article gives a reasonable overview.

This is intended as part one of a series, as it’s a basic and high-level view with no code examples.  In fact, it probably could be used to document the C# version as well as the Java version, although I haven’t taken a look at that for a while.  Next, I would like to give some more code examples of how you use it - as always, any suggestions welcome.

New Disruptor Presentation Unveiled to the LJC!

A few weeks ago, I presented my new “User’s Guide to the Disruptor” talk to the London Java Community.  Since it was very kindly hosted at Skillsmatter, there is a video of the presentation available, and the slides are below.

The presentation is a little different to the ones we’ve done before.  Previously we’ve gone into how it works and why it’s fast.  This time I wanted to step back a little from the internals and show how real developers might actually use it.  The example is somewhat contrived, but the idea is to give some hints on how to break your problem down into something that will work with a Disruptor at the heart of it.

I thought the event went really well.  It was a tiny bit completely intimidating, as there were no lightning talks and I was the only attraction.  Seeing over a hundred people turn up after work, before beer, to hear you talk is a humbling experience.  Fortunately, I think the audience was perfect for the presentation - they had heard of the Disruptor but hadn’t seen anything very detailed about it, so my walkthrough of how you might use it seemed to go down well.  I certainly got a lot of very sensible questions (which hopefully I’ve remembered to repeat for the benefit of the recording), and people had some good ideas about how and when to use it.

I ran the same presentation to a different audience at QCon London a week later, I’ll post a link to that if/when it becomes available.

Upcoming speaking events

In theory, I am busy writing material for my upcoming speaking events, rather than writing terribly illuminating posts on my blog (see what I did there?).  In actuality I am being lazy and have pretty much taken January off for a recharge.

In the spirit of doing something which ticks both the event-speaking and blogging boxes, this is a quick update on the conferences I’m confirmed for so far.  Put the following dates in your diary - these are my first international solo speaking events:

7th March - QCon London - Concurrent Programming Using The Disruptor (sadly I can’t stay for the whole conference as it clashes with the only holiday I had booked for 2012).
23rd May - GOTO Copenhagen - Concurrent Programming Using The Disruptor & War Stories.
25-26th May - GOTO Amsterdam - Concurrent Programming Using The Disruptor.

The presentation will be more of a user’s guide to the Disruptor than anything we’ve done before.  An hour isn’t a lot of time to cover all the functionality everyone might want to see, so I’m still trying to work out the balance between giving an introduction/overview for those who haven’t seen it before, and going into some of the cool features that have been added since I first started blogging about it.  If there’s anything you would particularly like to see covered, let me know - I’ll put the most frequently requested things in there.

Ideally I’d run a workshop session at some point, but that will require quite a lot more preparation, so I’ll only do that if there is interest in it (if someone wants to fly me somewhere interesting to do that so much the better!!).

Maybe I’ll see you at one of these events?

Video of our JAX London session

At JAX London Mike and I presented “Understanding the Disruptor - A Beginner’s Guide to Hardcore Concurrency”. This is the session we initially previewed to the London Java Community a few weeks earlier. The content is the same, but the feel of the presentation was quite different to us - the venue for the LJC event was more intimate, and it was easier to interact with the audience. At JAX, we were up on stage, which was pretty cool actually, but meant that it felt more like a lecture and it was less easy to connect with the audience.

We received some really great feedback on this presentation, and it was brilliant to see a lot of the speakers from JAX there watching us.