Summary: I got a false alert that the issues were resolved. Foolishly, I didn’t look into it further and didn’t realize we had an ongoing outage.
My deepest apologies for the two outages we’ve had in the past day. The first happened yesterday at around 5pm pacific time, the second today at around 6am. Both were caused by the same simple, easy to resolve issue. In both cases, I did not resolve the issue causing the outage because I didn’t realize the outages were happening. Our monitoring system sent me alerts saying the issues had resolved themselves just moments after it pointed out the problems in the first place. I should have looked into the issues upon getting the alerts, regardless of their supposed resolution, and I didn’t. Thus, we had hours of downtime when we should have had minutes.
I am ashamed of the situation, and stressed by the potential repercussions on the trust of our users. Due to my actions, tens of thousands of people around the world were blocked from doing their work over the past day. My apologies to all of you.
Here is what I will do to prevent this happening in the future:
- I’ve added two new alerts to our monitoring which are direct indicators of our most common issues. These metrics will not incorrectly resolve themselves while an outage remains active.
- I will always look into outage alerts, even if I get an automated alert saying an issue is resolved.
- After an alert, I will look at our core metrics dashboards as opposed to my own app responsiveness to determine whether the app is healthy. Loading the home page is an incredibly dumb way to verify the health of our servers. I did it simply because I was away from my computer and have never seen a real outage falsely resolved without additional alerts. That was a huge mistake.
- After an alert, I will monitor our user communication channels such as Twitter and email, to get a sense of how widespread an issue is.
Again, I am truly sorry for these completely unnecessary outages. We will do our best to maintain our previously wonderful reliability.
P.S. Please consider using our offline Chrome Desktop Application. This way you will have access to your WorkFlowy lists even if we have an outage, or if you’re away from the internet.
Thanks for the explanation, Jesse.
Your post is the example we all should behave in case of mistakes.
Thanks to daily Dropbox backup I had access to crucial records.
Keep doing great work!
Don’t be too stressed. Mistakes happen, we appreciate your accountability. We’re still hooked on Workflowy!
Thank you Jesse! As a new user this helps me feel more comfortable using it. If itâs helpful to trace the problem too, I experienced an outage at around 6am pacific time yesterday, but thought it might have just been my computer so waited until the end of a meeting, at which point it was fixed.
Susan Putnins, LCSW
Evaluation and Research Manager
The Home for Little Wanderers
617-927-0651
sputnins@thehome.org
[WeAreOrlando]
Yes this stressed me out. Yes it was inconvenient and alarming. But this note and accountability redoubles my faith and understanding in what you are doing. Thank you for this entry. It really means the world to us who have outsourced our brains to Workflowy. If I lost Workflowy I would LITERALLY lose my mind.
I echo all the above sentiments – especially like your movement forward strategy and your transparency
— need you workflowy
No problem jesse. Your site is great. Keep it up!
Seconding what Clint said above. This is a tremendous statement of commitment to your users and I am very grateful. That this is the the only outage that I have experienced in over three years of using WorkFlowy is a testament to your reliability. I anticipate continuing to live in WorkFlowy for many years to come.
I appreciate your honesty and explanation. I won’t lie it was frustrating as I work directly out of workflowy during meetings and reference it constantly for my information and notes. It is also a key program that my boss and I use to communicate and share information. All that said we will still continue using and I appreciate your list of ways you will work to prevent this from happening in the future! Thanks!!
Thanks for the update. You have restored my trust in WorkFlowy by (a) explaining the cause which was really minor, and (b) telling us what you are going to do about it. WorkFlowy rocks!
I can echo what everyone said. Thank you for your honesty and willingness to make things right. I was in the midst of having many tech difficulties yesterday and I am grateful that for yours, you took responsibility. You have encouraged my loyalty greatly with this post. Thank you.
hi,
could you please add export and import action with encryption.
I’ve used Things, and Omnifocus, and Todoist, each one for at least a year, sometimes years, and now that I use WorkFlowy I don’t ever want to go back to any of those. In fact, I tried to do so yesterday, when my faith in the availability of WF was shaken, and … it just made me appreciate the beautiful design of WF all the more. So I am really heartened by this communication, and by the transparency of it, which I know (having done IT support) must be painful for you. Thank you! I am more than ever sold on WorkFlowy.
Thank you so much for the thoughtful and obviously heartfelt explanation. WorkFlowy is an integral part of my professional life and I am glad to hear about the measure you have put in place to minimize further downtime. I am an enthusiastic promoter of WF and will continue to do so.
It’s great to see people taking responsibility when things go wrong. Thank you for being honest and upfront.
I love workflowy and recommend it to everyone I know.
Thanks
Thank you for the detailed explanation and mea culpa. It was definitely a hassle, but hopefully now the service will stay stable. As for the Chrome app, I have to say I did use it before, but recently switched back to Chrome because:
1) Having multiple tabs is just too convenient, which is not possible in the desktop version
2) Chrome extensions are not supported on desktop, which I’m now more reliant on, particularly color highlighting via Painter for Workflowy
I agree, explanation much appreciated and apology accepted. Question, love that you told us about offline functionality through the app as well but if there’s an outage, won’t this app fail to update as well with the latest list? If so, we’d be in the same situation. I’ve installed it nonetheless and I appreciate that functionality. Thanks!
“[I am] stressed by the potential repercussions on the trust of our users.”
Can’t speak for everyone, but you garnered more trust on my end from this post than if there was no outage at all.
Thanks Jesse for your being so transparent about the problem. I really appreciate Workflowy taking responsibility for the error, though in the end the actual fault certainly isn’t yours. Thank you for letting us all know what happened and what the Workflowy team is doing to avoid this issue in the future. I remain a fan of what the team is doing here.
Thank you for your straightforwardness in explaining this issue. Workflowy is great.
I’m surprised that nobody on the team is actively using Workflowy (or monitoring Twitter and Facebook at least daily), or they’d have known it was actually down.
Does the outage on the 15th (listed on Twitter as: “WorkFlowy is down, you may have noticed. We are working on it, should be back soon.”) fall into the same category? That one you did know about, but there was no update afterward.
I’ve been trusting WF for quite a while now as a paid user but these few days have definitely shaken my confidence, I’m afraid.
I do like your plan for going forward, with one tiny adjustment: I hope you move “monitor our communication channels” up a little higher in your process. I was genuinely concerned yesterday you’d simply pulled the plug on the app since there were no responses to anything.
The outage on the 15th was a brief outage to move to a larger database, we just forgot to post once it was back up :/
We are a tiny team of two and one half of us was on vacation.
Only effected me for a few seconds, just went and did something else more interesting instead! 😊
I love you guys too, agree with all of the above. Truly inspiring.
To err is human, to own your errors, divine.
I did encounter the outage, to no real impact but I comment just to commend you on what you have wrote here.
Thank you for this explanation; it is reassuring. All the best!
Thanks for the feedback! Problems do occur. For Europe, the timing was perfect… 😉
I still love Workflowy and will keep on telling everyone how great it is… (I can assure you, managing a software company myself, I haven’t recommended other software in years…)
Dimitar
It happens… thanks for the honest explanation!
Jesse, your willingness to take responsibility is admirable, not to mention taking the time to post and update users, it’s the attitude that will hopefully take Workflowy far. I’m still using it as much as ever, and love the service you guys provide, keep it up! 🙂
I use WF every day since I met it few month ago and I love it. Today I’ve subscribed to support you.
Keep going, you’re doing great.
Respect about your honest explanation !
Jess, I’m very impressed with your attitude of taking these responsibilities so serious and personal. I wish to have more people like you around. Thank you for all outstanding job you do with WorkFlowy. You help us to organize ideas and projects in the most efficient and user friendly way. Please keep doing what you’re doing.
Thanks for the explanation and apology, though I don’t really deserve to receive an apology since I use the free version and you’re the one doing me a service – I have no claims upon you, only great appreciation for such a simple yet powerful and useful tool! May you continue assisting many people and be blessed to make a good living from it!
If I were American, and younger, and more hip, I might write something like “you guys rock”. But I’m from the UK, and we don’t do that.
Workflowy is a truly great product and your honesty (and action) after something like this just makes your service more attractive.
I love workflowy. I have just upgraded to show my support.
What a refreshing response. I only noticed the outage in the last hour or so that it has been happening. I was mildy worried that I’d never get my list back and really appreciate all you do to make this a fantastic service! Thanks so much.
Well done for writing a proper explanation of the problem, in plain English, and explaining how it will be avoided in future. Very refreshing in an age where so many other online services (eg Evernote) spew useless psychobabble like “service availability is impacted”, which tells users nothing and just adds to their frustration.
Forget it, thanks for this great product!
I love Workflowy, and I appreciate your professionalism and good form in communicating with while working towards a better system! Thank you for making and maintaining this profoundly useful tool!
Thank you. Appreciate your notification and thank you for the best and simplest tool – WorkFlowy!
Incidents happen. The funny thing is I am a using Workflowy every day for a long time now and never experienced outage, not even yesterday.
Remarkable reliable if you ask me. Add the great open communication and the outstanding functionality, you get a perfect tool. Time to switch to Workflowy Pro!
I love you workflowy
Thanks for the update Jesse. It’s great to see companies owning up to issues rather than sweeping them under the carpet. It would be great to understand the underlying cause (for us techie minded folk) and some feedback on current status should another issues occur would help manage the panic level a little.
That is the most genuine customer-focused outage notification I’ve ever seen. Thank you for sparing the fluff. Bonuses points if you we can see an RCA. 🙂
Is RCA a credit of some sort?
It means Root Cause Analysis, I believe. Asking “Why” Five times is a common way to do it. https://www.isixsigma.com/tools-templates/cause-effect/determine-root-cause-5-whys/
As others have said, the transparency is appreciated! 😀
I would only suggest having a status page so I would know when to not freak out (“ahh… now they’re investigating… now they’re monitoring… it’ll take this long”) or know to freak out (THEY DON’T KNOW THERE’S AN OUTAGE AAH!!).
Seconded 🙂
Thanks for the update. This morning was rough for all of us, but we still love WorkFlowy, and appreciate your honest reflection.
workflowy is great. keep going that way.
Nobody’s perfect. The best we can do is learn from our mistakes. Thanks for all you do!
Thanks for the explanation. Things really do go “bump” in the night, huh?
Jesse, I amazed that there haven’t been more WF outages, glitches, and patches like I have experienced from other subscription applications. Your reflection on what happened and your assumption of responsibility, steps to address the issues, and your update is refreshing. Thanks
This response is much appreciated. I use Workflowy personally and professionally—literally, all day, every day—and I have to admit that this episode made me reflect on the basket into which I’ve put a whole lot of my eggs. Hopefully this will just serve as a cautionary tale in the long run. 🙂 I’m as much a fan as ever, and deeply grateful.
I’ve already learned my lesson a few months ago, luckily. I use the Chrome app and save daily backups.
Apology accepted 🙂
That looks great, and I appreciate the transparency. From the comments, many appreciate it as well.
I also appreciate the offline version, which I never knew about. Thank you!
If I can add to that, a number #5 — say something! Monitoring is user communication channels is nice, but a quick ‘we’re on it’ goes a long way.