A look back: our company’s first 3 months and first 1000 user registrations

February 9, 2012 by Priit Potter

In the last days of January, the number of Plumbr users exceeded an important milestone – we now have more than 1,000 users! As we started offering Plumbr as a public beta in the beginning of November last year, we can say that „collecting“ the first 1,000 customers took us 3 months.

Let’s dive into the numbers (we are big fans of factual proof!), and elaborate on:

  • What did we do to get here?
  • What have we learned from the customers and where are we planning to head with product development?
  • What are we going to do to reach 10,000? 100,000? Bazillion?

The first users of Plumbr

The beginning was easy. The founders were working in a large IT service provider Webmedia and throughout their day to day Java development jobs experienced tens of different memory-related problems every year. For reasons still unknown the founders became respected specialists in the company and got bogged down by memory troubleshooting issues of different projects. As this got boring to handle manually, they went ahead and created a solution to automate it.

Hence, the first users where the tens (hundreds?) of Java projects developed within Webmedia. The memory patterns provided a very valuable pool to study, and to develop the first version of the leak detection and reporting algorithms. When the internal beta version was launched, we advertised the tool to the Java developers of the company and recruited the first 40 users.

Now to the sad part – those users did not have many problems with memory leaks at that moment. We received data of… well, 2 memory leaks. We didn’t feel comfortable with that knowledge base and started looking for ways to get more data.

Getting out to the wild

Where does one find out whether the problem they are solving is real? Apparently you can Google for it. Or – to be more precise – you can check the Google Keyword Tool. It told us that 135 000 searches for „java.lang.OutOfMemory“ are fired every month. Wow. We had hit the jackpot!

But since we are from Estonia, and by common belief Estonians are slow by nature, we didn’t dare to raise a huge pile of money and pour it to Google’s pockets in exchange of targeted Adwords ads. Instead we decided to grow at a rate where we could interact with the users and improve the product based on their feedback.

Still, it takes months to get the initial PageRank and start receiving organic traffic. How can we enforce this without paying for the users? By luck, one possible solution found us by itself. A friend posted a link to our web page in the Hackernews channel. We only found out about it because the Google Analytics Real Time window (requires login, but is an awesome feature!) suddenly started showing intense traffic. So we picked up on this and posted a link to reddit/r/programming ourselves. Voila – we got 2,000 unique visitors between November 8th and 10th, 2011. Those 2,000 first visitors converted into 194 registrations and 146 downloads.

From there on, until the end of January (when we hit the 1,000 registration mark) we’ve had 26,000 unique web site visitors. This gives us a 3,8% visitor -> registration conversion rate. Not the best conversion rate out there, but at least we had found some channels that we could use for building awareness of the product, and reached the growth rate that we had dreamt of in the beginning.

So, where did these 26,000 unique visitors came from?

The graph below is based on data gathered by Google Analytics. It is a bit skewed: for example, a Hackernews post is automatically tweeted by several bots, and top posts receive extra tweet coverage – and that is why some visitors who were directed to the site via Hackernews, are actually shown as Twitter (t.co) referrals. Nevertheless, I think that the distribution is a pretty good illustration:

channel distribution

It has not been only us advertising in those sites. Unproportional advertising of your own site would conflict the policy of several of them and we would probably be quickly banned as spammers. Fortunately, there have been other visitors who liked our content and shared it, so I think more than half of the blog posts have actually been featured in at least one of those link sharing sites.

If you know of any other communities to whom our content would be interesting, please let us know in the comments. Or even better – share the post yourself! (We will pick it up in the referral statistics, don’t worry)

How (much) do people actually use Plumbr?

Of course, getting a thousand registrations is great, and certainly marks a milestone. But a clever reader has wondered since the beginning – how much have those 1000 users actually used the product, and what might be the bottlenecks they face that inhibit larger usage? Let us look at the the numbers and elaborate on this next.

But first – for people who have not used Plumbr before – this is a quick overview of how your experience with Plumbr typically flows:

  1. You enter an e-mail address on our home page
  2. We automatically assemble a personal download package for you and immediately send out an e-mail with the download link
  3. You click on the link, download the zip file, and see installation instructions
  4. You unzip and attach Plumbr to your Java server or application
  5. You start your Java app and let Plumbr monitor it. With some luck, your application is leak-free and you can breathe relieved. Sometimes Plumbr discovers a leak or few, then it composes a report saying what exactly leaks in your code and gives all the info you need to fix the leak.
  6. If you like the product and trust it works well, you leave it attached, and let it take care of worrying about memory leaks

The above effectively describes our usage funnel. And our goal, naturally, is to direct as many users as possible into the later stages of the funnel.

We get feedback from the users in three ways. All of them provide us different data, and for reaching true insights, we need to combine all three.

  1. User-initiated feedback in the forums, Skype support channel, and support@ emails. This source gives us qualitative data on how people use Plumbr. Or rather – why they don’t use it. Most of the feedback reports problems with installation or usage.As of today, we have received 100 e-mails, there have been 13 questions in the forums, and around the same number of people have contacted us via Skype. We haven’t measured, but I estimate that we solve around 70-80% of the complaints, and the rest are either mysteries that dissolved themselves or represent unsupported environments that we have yet to extend Plumbr to.
  2. Feedback initiated by us. I have followed Rob Fitzpatrick’s advice on personally contacting the first users to get to know our customers. Out of the first 1,000 registrations I have personally e-mailed to more than 900 (some of the rest I have met in person, some contacted us first, and some I just overlooked in the list and missed to contact). I have got around 70 replies. (Anybody out there with more experience – is this good number?)Around one third of them say „indeed I downloaded Plumbr, have not tried yet, but hope to do it in the near future“. Another third praise Plumbr, and say that Plumbr didn’t find any leaks. And the rest like Plumbr and also found a memory leak (albeit some only tested it with the packaged leaking demo app).
  3. The third source is purely quantitative and connected to Plumbr usage. As Plumbr is software that you download and run on your own machine, it is a bit tricky, but it is still possible to get some feedback. At launch, Plumbr dials home to check for version upgrades. While the main idea is to alert people when a newer version is out (being in beta we might need to release fast bug fixes), we also get good data of which versions of Plumbr people run and how many people of the ones who downloaded have actually started to use it.We also get a note when people open a leak report (there is a message on the report explaining that since they use a free evaluation copy they need to notify us of receiving the report before they can see the contents).

Let’s visualize the data we have collected:

funnel1

It seems that out of 1,000 registered users 263 actually started the tool. Shocking, isn’t it?

There is one technical catch though. If we look at the report opening data, we see that 18% of the „open report“ events come from people who have never sent us the „run Plumbr“ event. The reason is that some of the users use Plumbr on servers that forbid outgoing traffic, and the release upgrade check never reaches us.

Assuming that people behind firewalls get (and open) leak reports with the same distribution as the ones who are not, let’s take that 18% into account when analysing our funnel:

funnel2

Still, shocking. What could be the possible reasons of the low 31% conversion rate?

From the qualitative data we estimate that a whole 30% of people intend to try Plumbr (they download it), but never do because something distracts them and they just add it to their task list.

A fair amount of users probably need Plumbr in an environment that is not supported yet. Many people have asked for Java 5 support (this is one of the top priorities in the list), but we have also received reports of Plumbr not working on Websphere and some Glassfish vesions, with Groovy and some environments.

There are definitely also users who just don’t know how to modify JVM start-up parameters and for whom the installation instruction  is too complex. Some of them contact us, but every participation (in)equality study (see e.g. the 90–9–1 principle) shows you that 90% don’t. Let’s use a best guess and accommodate these assumptions on our funnel graph as well:

funnel3

These are the biggest barriers we could think of, and that were supported by the qualitative feedback. If you think that we have missed something important, please let us know in the comments below!

What can we do to raise the conversion rates?

We have several ideas on what to do to raise the „new visitors->registrations“ conversion on our web page. We need to improve the landing pages (the home page and the blog post page, where more than 90% of the people arrive to plumbr.eu). We need to make a better and more clear offering, explain better what Plumbr is, and publish references and testimonials of current users. We are also planning to A/B test different value proposition and Call-to-actions (I will write a separate blog post on this). We follow closely Peep Laja’s blog  on website conversion and have more ideas than resources to try them out.

However, the web conversion rate is only the tip of the iceberg. What we really need to improve is the conversion to product usage. Without it, two-thirds of every effort made to increasing web site traffic or traffic->download conversion would be lost. And without a bigger user base we might take wrong product development decisions.

From the reasoning above we concluded that although we liked to think that our installation process is very smooth and easy, it is still too complex – complex enough that people who download, don’t get „hooked“ and give up using Plumbr before they get started.

So, we plan to investigate and implement:

  • An installer that would somehow detect the Java application servers that exist on the computer, and assist in modifying the start-up arguments.
  • An executable that would be able to connect to running Java apps instantly, and re-connect when those are restarted. Possibly also connect to an application running on a remote machine – like jconsole does.
  • Include more guidelines and integrate our support channels in the installation process
  • Since at the moment Plumbr is mostly tried out by developers, we are thinking of describing the installation for more popular IDEs and application servers separately.

What are the next actions?

We released Plumbr 1.0RC4 on Tuesday, and have dived into the development of the next release by now, implementing the aforementioned directions.

For our blog readers – my blog post pipeline includes topics on customer/user segmentation, channel based conversion analysis, our product licensing plans, and – of course – follow-ups to the above-mentioned topics. Follow our twitter feed to be notified when the posts are published!

And, of course, if you got interested in trying out Plumbr yourself, just tell us your e-mail address and you’ll be flown into our usage funnel instantly!

Can't figure out what causes your OutOfMemoryError? Read more

ADD COMMENT

COMMENTS

Nice story, interesting and well-written. Excellent “study material” for the start-up companies applying for funding with unrealistic business plans – already for their first public beta. You’ve got a good story, and show that – once again – it’s all about 10% inspiration and 90% transpiration. Good luck, I’m looking forward to read some more startup insights…

Nico

Hi Nico, thanks for the good words. Make sure to follow our RSS feed or @JavaPlumbr and spread the word in the Belgian Java world :)

Priit Potter

Impressive! Thanks for sharing your story in such detail. I’d recommend not to be too obsessed with your conversion rate. 3.6% is actually pretty good. As Avianash Kaushik, Google’s Analytics evangelist, states, 4% is the average (http://www.kaushik.net/avinash/stop-obsessing-about-conversion-rate/)nFocus on driving more qualified traffic (links, press releases, white papers, articles, reviews, tutorials, presentations at Java related forums like Meetups, etc.) and have a look at Avinash’s comments about good analytics and his concept of Macro and Micro conversions. But it looks like you are well on the way already.

Kaj Kandler

Thanks, Kaj, for the comment. I agree with you that striving to find and drive more good quality traffic to our site is also important. The critical view on the current conversion is mostly based on gut feeling – I have a feeling that (considering the source channels and the topics that drive people to our site) we could do better. That is why, among other things, we also try to identify and eliminate the possible barriers in the funnel that give people “excuses” for not moving to the next stage.nnThere is always a number of people among site visitors who would be glad to try the offering, but perceive their effort too big relative to the expected value they’d get. And I think we can still lower the number of people who drift away from our site without trying out the software, and do so with some relatively small changes to the web site content, layout, and wording. That is why we will do some A/B testing to find the more optimal configurations for bigger the conversion rates.nnBut I agree with you on that this is not most important issue for us at the moment. We will focus more on making the product easier to install and use.

Priit Potter

Can't figure out what causes your OutOfMemoryError? Read more

Latest
Recommended
You cannot predict the way you die
When debugging a situation where systems are failing due to the lack of resources, you can no longer count on anything. Seemingly unrelated changes can trigger completely different messages and control flows within the JVM. Read more
Tuning GC - it does not have to be that hard
Solving GC pauses is a complex task. If you do not believe our words, check out the recent LinkedIn experience in garbage collection optimization. It is a complex and tedious task, so we are glad to report we have a whole lot simpler solution in mind Read more
Building a nirvana
We have invested a lot into our continuous integration / delivery infrastructure. As of now we can say that the Jenkins-orchestrated gang consisting of Ansible, Vagrant, Gradle, LiveRebel and TestNG is something an engineer can call a nirvana. Read more
Creative way to handle OutOfMemoryError
Wish to spend a day troubleshooting? Or make enemies among sysops? Registering pkill java to OutOfMemoryError events is one darn good way to achieve those goals. Read more