The Explanation For Our Recent Downtime

By

July 19, 2016

Summary: I got a false alert that the issues were resolved. Foolishly, I didn’t look into it further and didn’t realize we had an ongoing outage. 

My deepest apologies for the two outages we’ve had in the past day. The first happened yesterday at around 5pm pacific time, the second today at around 6am.  Both were caused by the same simple, easy to resolve issue. In both cases, I did not resolve the issue causing the outage because I didn’t realize the outages were happening. Our monitoring system sent me alerts saying the issues had resolved themselves just moments after it pointed out the problems in the first place. I should have looked into the issues upon getting the alerts, regardless of their supposed resolution, and I didn’t. Thus, we had hours of downtime when we should have had minutes.

I am ashamed of the situation, and stressed by the potential repercussions on the trust of our users. Due to my actions, tens of thousands of people around the world were blocked from doing their work over the past day. My apologies to all of you.

Here is what I will do to prevent this happening in the future:

  1. I’ve added two new alerts to our monitoring which are direct indicators of our most common issues. These metrics will not incorrectly resolve themselves while an outage remains active.
  2. I will always look into outage alerts, even if I get an automated alert saying an issue is resolved.
  3. After an alert, I will look at our core metrics dashboards as opposed to my own app responsiveness to determine whether the app is healthy. Loading the home page is an incredibly dumb way to verify the health of our servers. I did it simply because I was away from my computer and have never seen a real outage falsely resolved  without additional alerts. That was a huge mistake.
  4. After an alert, I will monitor our user communication channels such as Twitter and email, to get a sense of how widespread an issue is.

Again, I am truly sorry for these completely unnecessary outages. We will do our best to maintain our previously wonderful reliability.

P.S. Please consider using our offline Chrome Desktop Application. This way you will have access to your WorkFlowy lists even if we have an outage, or if you’re away from the internet.

0 0 votes
Article Rating
Subscribe
Notify of
guest
92 Comments
Inline Feedbacks
View all comments
Richard Bird
5 years ago

Apology accepted 🙂

Rocky Kev (@RockyKev)
5 years ago
Reply to  Richard Bird

That looks great, and I appreciate the transparency. From the comments, many appreciate it as well.

I also appreciate the offline version, which I never knew about. Thank you!

If I can add to that, a number #5 — say something! Monitoring is user communication channels is nice, but a quick ‘we’re on it’ goes a long way.

Kaelri
5 years ago

This response is much appreciated. I use Workflowy personally and professionally—literally, all day, every day—and I have to admit that this episode made me reflect on the basket into which I’ve put a whole lot of my eggs. Hopefully this will just serve as a cautionary tale in the long run. 🙂 I’m as much a fan as ever, and deeply grateful.

Jeffrey P Wyman
5 years ago
Reply to  Kaelri

I’ve already learned my lesson a few months ago, luckily. I use the Chrome app and save daily backups.

Bob
Bob
5 years ago

Jesse, I amazed that there haven’t been more WF outages, glitches, and patches like I have experienced from other subscription applications. Your reflection on what happened and your assumption of responsibility, steps to address the issues, and your update is refreshing. Thanks

Paul Rider
5 years ago

Thanks for the explanation. Things really do go “bump” in the night, huh?

Pam McCormack
Pam McCormack
5 years ago

Nobody’s perfect. The best we can do is learn from our mistakes. Thanks for all you do!

Wizmann (@Wizmann)
5 years ago

workflowy is great. keep going that way.

John A. Taylor (@johnataylor)

Thanks for the update. This morning was rough for all of us, but we still love WorkFlowy, and appreciate your honest reflection.

YatharthROCK
5 years ago

As others have said, the transparency is appreciated! 😀

I would only suggest having a status page so I would know when to not freak out (“ahh… now they’re investigating… now they’re monitoring… it’ll take this long”) or know to freak out (THEY DON’T KNOW THERE’S AN OUTAGE AAH!!).

Jeffrey P Wyman
5 years ago
Reply to  YatharthROCK

Seconded 🙂

Sal Peña
Sal Peña
5 years ago

That is the most genuine customer-focused outage notification I’ve ever seen. Thank you for sparing the fluff. Bonuses points if you we can see an RCA. 🙂

SilverSideDown
5 years ago
Reply to  Sal Peña

Is RCA a credit of some sort?

Jesse Patel
Jesse Patel
5 years ago
Reply to  SilverSideDown

It means Root Cause Analysis, I believe. Asking “Why” Five times is a common way to do it. https://www.isixsigma.com/tools-templates/cause-effect/determine-root-cause-5-whys/

Martin Pugh
Martin Pugh
5 years ago

Thanks for the update Jesse. It’s great to see companies owning up to issues rather than sweeping them under the carpet. It would be great to understand the underlying cause (for us techie minded folk) and some feedback on current status should another issues occur would help manage the panic level a little.

leo424y
5 years ago

I love you workflowy

Bert
Bert
5 years ago

Incidents happen. The funny thing is I am a using Workflowy every day for a long time now and never experienced outage, not even yesterday.
Remarkable reliable if you ask me. Add the great open communication and the outstanding functionality, you get a perfect tool. Time to switch to Workflowy Pro!

Modestas
Modestas
5 years ago

Thank you. Appreciate your notification and thank you for the best and simplest tool – WorkFlowy!

Joshua Levin
Joshua Levin
5 years ago

I love Workflowy, and I appreciate your professionalism and good form in communicating with while working towards a better system! Thank you for making and maintaining this profoundly useful tool!

yarden
5 years ago

Forget it, thanks for this great product!

Simon M
Simon M
5 years ago

Well done for writing a proper explanation of the problem, in plain English, and explaining how it will be avoided in future. Very refreshing in an age where so many other online services (eg Evernote) spew useless psychobabble like “service availability is impacted”, which tells users nothing and just adds to their frustration.

Jenny Lank
5 years ago

What a refreshing response. I only noticed the outage in the last hour or so that it has been happening. I was mildy worried that I’d never get my list back and really appreciate all you do to make this a fantastic service! Thanks so much.

appdevelopersbiz
5 years ago

I love workflowy. I have just upgraded to show my support.

rsearjeant
5 years ago

If I were American, and younger, and more hip, I might write something like “you guys rock”. But I’m from the UK, and we don’t do that.

Workflowy is a truly great product and your honesty (and action) after something like this just makes your service more attractive.

Yehoshua Kahan
Yehoshua Kahan
5 years ago

Thanks for the explanation and apology, though I don’t really deserve to receive an apology since I use the free version and you’re the one doing me a service – I have no claims upon you, only great appreciation for such a simple yet powerful and useful tool! May you continue assisting many people and be blessed to make a good living from it!

Vlad Alive (@vladalive)

Jess, I’m very impressed with your attitude of taking these responsibilities so serious and personal. I wish to have more people like you around. Thank you for all outstanding job you do with WorkFlowy. You help us to organize ideas and projects in the most efficient and user friendly way. Please keep doing what you’re doing.

Martin
Martin
5 years ago

I use WF every day since I met it few month ago and I love it. Today I’ve subscribed to support you.
Keep going, you’re doing great.
Respect about your honest explanation !

Chris
Chris
5 years ago

Jesse, your willingness to take responsibility is admirable, not to mention taking the time to post and update users, it’s the attitude that will hopefully take Workflowy far. I’m still using it as much as ever, and love the service you guys provide, keep it up! 🙂

smarterdigitalltd
5 years ago

It happens… thanks for the honest explanation!

Dimitar Giulev
5 years ago

Thanks for the feedback! Problems do occur. For Europe, the timing was perfect… 😉

I still love Workflowy and will keep on telling everyone how great it is… (I can assure you, managing a software company myself, I haven’t recommended other software in years…)

Dimitar

Jibé
5 years ago

Thank you for this explanation; it is reassuring. All the best!

Jessi R.
5 years ago

I did encounter the outage, to no real impact but I comment just to commend you on what you have wrote here.

Ron Brn Gross
5 years ago

I love you guys too, agree with all of the above. Truly inspiring.

To err is human, to own your errors, divine.

Tim Woodward
5 years ago

Only effected me for a few seconds, just went and did something else more interesting instead! 😊

Heather
Heather
5 years ago

I’m surprised that nobody on the team is actively using Workflowy (or monitoring Twitter and Facebook at least daily), or they’d have known it was actually down.

Does the outage on the 15th (listed on Twitter as: “WorkFlowy is down, you may have noticed. We are working on it, should be back soon.”) fall into the same category? That one you did know about, but there was no update afterward.

I’ve been trusting WF for quite a while now as a paid user but these few days have definitely shaken my confidence, I’m afraid.

I do like your plan for going forward, with one tiny adjustment: I hope you move “monitor our communication channels” up a little higher in your process. I was genuinely concerned yesterday you’d simply pulled the plug on the app since there were no responses to anything.

Jesse Patel
Jesse Patel
5 years ago
Reply to  Heather

The outage on the 15th was a brief outage to move to a larger database, we just forgot to post once it was back up :/

We are a tiny team of two and one half of us was on vacation.

Wick Thomas
Wick Thomas
5 years ago

Thank you for your straightforwardness in explaining this issue. Workflowy is great.

John
5 years ago

Thanks Jesse for your being so transparent about the problem. I really appreciate Workflowy taking responsibility for the error, though in the end the actual fault certainly isn’t yours. Thank you for letting us all know what happened and what the Workflowy team is doing to avoid this issue in the future. I remain a fan of what the team is doing here.

Clint
Clint
5 years ago

“[I am] stressed by the potential repercussions on the trust of our users.”

Can’t speak for everyone, but you garnered more trust on my end from this post than if there was no outage at all.

Ken Guie
5 years ago

I agree, explanation much appreciated and apology accepted. Question, love that you told us about offline functionality through the app as well but if there’s an outage, won’t this app fail to update as well with the latest list? If so, we’d be in the same situation. I’ve installed it nonetheless and I appreciate that functionality. Thanks!

SilverSideDown
5 years ago

Thank you for the detailed explanation and mea culpa. It was definitely a hassle, but hopefully now the service will stay stable. As for the Chrome app, I have to say I did use it before, but recently switched back to Chrome because:

1) Having multiple tabs is just too convenient, which is not possible in the desktop version
2) Chrome extensions are not supported on desktop, which I’m now more reliant on, particularly color highlighting via Painter for Workflowy

KC
KC
5 years ago

It’s great to see people taking responsibility when things go wrong. Thank you for being honest and upfront.

I love workflowy and recommend it to everyone I know.

Thanks

Frederick S. Lane
5 years ago

Thank you so much for the thoughtful and obviously heartfelt explanation. WorkFlowy is an integral part of my professional life and I am glad to hear about the measure you have put in place to minimize further downtime. I am an enthusiastic promoter of WF and will continue to do so.

Randi
Randi
5 years ago

I’ve used Things, and Omnifocus, and Todoist, each one for at least a year, sometimes years, and now that I use WorkFlowy I don’t ever want to go back to any of those. In fact, I tried to do so yesterday, when my faith in the availability of WF was shaken, and … it just made me appreciate the beautiful design of WF all the more. So I am really heartened by this communication, and by the transparency of it, which I know (having done IT support) must be painful for you. Thank you! I am more than ever sold on WorkFlowy.

hodanli
hodanli
5 years ago

hi,

could you please add export and import action with encryption.

Annie
Annie
5 years ago

I can echo what everyone said. Thank you for your honesty and willingness to make things right. I was in the midst of having many tech difficulties yesterday and I am grateful that for yours, you took responsibility. You have encouraged my loyalty greatly with this post. Thank you.

davidjwilliamson
5 years ago

Thanks for the update. You have restored my trust in WorkFlowy by (a) explaining the cause which was really minor, and (b) telling us what you are going to do about it. WorkFlowy rocks!

Michele
Michele
5 years ago

I appreciate your honesty and explanation. I won’t lie it was frustrating as I work directly out of workflowy during meetings and reference it constantly for my information and notes. It is also a key program that my boss and I use to communicate and share information. All that said we will still continue using and I appreciate your list of ways you will work to prevent this from happening in the future! Thanks!!

John Stephan (@jmstephan)

Seconding what Clint said above. This is a tremendous statement of commitment to your users and I am very grateful. That this is the the only outage that I have experienced in over three years of using WorkFlowy is a testament to your reliability. I anticipate continuing to live in WorkFlowy for many years to come.

ninrode
ninrode
5 years ago

No problem jesse. Your site is great. Keep it up!

Constance Verhagen
Constance Verhagen
5 years ago

I echo all the above sentiments – especially like your movement forward strategy and your transparency
— need you workflowy

ziipppp
ziipppp
5 years ago

Yes this stressed me out. Yes it was inconvenient and alarming. But this note and accountability redoubles my faith and understanding in what you are doing. Thank you for this entry. It really means the world to us who have outsourced our brains to Workflowy. If I lost Workflowy I would LITERALLY lose my mind.

Susan Putnins
Susan Putnins
5 years ago

Thank you Jesse! As a new user this helps me feel more comfortable using it. If it’s helpful to trace the problem too, I experienced an outage at around 6am pacific time yesterday, but thought it might have just been my computer so waited until the end of a meeting, at which point it was fixed.

Susan Putnins, LCSW
Evaluation and Research Manager
The Home for Little Wanderers
617-927-0651
sputnins@thehome.org

[WeAreOrlando]

Anthony
Anthony
5 years ago

Don’t be too stressed. Mistakes happen, we appreciate your accountability. We’re still hooked on Workflowy!

Cezary D.
Cezary D.
5 years ago

Keep doing great work!

Vadim Vasilev
Vadim Vasilev
5 years ago

Thanks for the explanation, Jesse.
Your post is the example we all should behave in case of mistakes.
Thanks to daily Dropbox backup I had access to crucial records.

Share on twitter
Share on facebook
Share on linkedin
Share on email

Subscribe to the newsletter

We'll send you a weekly roundup of the latest posts

Overwhelmed?

Workflowy replaces your notebooks, stickies and bloated apps with a simple, smooth digital notebook that makes it easy to get organized and be productive.