On May 17 2019, our Salesforce org was experiencing a severe breakdown. It was a Friday morning and I, as a Salesforce Administrator, was working on some system configuration at that moment. All of the sudden, a couple of my colleagues in the same office told me that they could not create new case in Salesforce.

At first, I was like, well, nothing biggie, let me take a look at it first. In fact, I was a little paranoid when they told me that they couldn't create new case as I was the only system administrator in the company, I knew my way around Salesforce ecosystem, and I knew for sure that I didn't modify anything regarding to the object permission. So, I came over to their desk, trying to refresh the webpage see if the problem goes away (works most of the time though 😏). So yeah, the 'Create New' button didn't show up, that's fine, I thought. Though I didn't quite believe what I saw right there, so I went back to my desk and checked on the case object permission. The next thing that I saw completely terrified me 😱😱😱...

What I saw was a completely misconfigured object permissions (something like the image below).

How could the custom profile had the 'Delete' and even 'Modify All' checked? That was so gonna cause hell lot of problems! When I tried to revert it back to the previous setting (Just 'Read', 'Create' and 'Edit') , the system didn't let me to do so, instead it showed whole bunch of error messages that I didn't find any relevance to what I was doing there.

The next thing I did was to check the debug log. I wondered who was trying to mess with the system although deep down I knew no one would be able to do so. I exported it out and all I could see was the modification that I had done. Nothing suspicious from unknown users at all.

I started getting panicked as I really thought it was something had to do with what I did in the morning. I never thought of the day when Salesforce would just breakdown like that. More and more users had approached me and told me that they couldn't login to Salesforce (I had a login flow which look-up Opportunity Object and since the Opportunity object wasn't accessible for them, the flow was caught in error, hence they couldn't continue with the process). It was really the first time that I exprienced such stressful situation since I became Salesforce Administrator. My first thought was to reach out Salesforce Support. When I typed in 'Salesforce Support' in search bar and clicked into the salesforce website and I saw this:

That was the first announcement message that I saw (it later on became red with different messages). I knew the impact might not hit just us but others as well. I did not what it was at that time but I knew it was some issues on Salesforce end. So, I quickly told everyone about it, rest assuring them we would be back to business in probably next couple of hours or so (actually, I thought it would be solved sooner though...). Ever since then, I was busy checking online, the Salesforce forums, searching other websites to follow up with the latest update (none from official site yet at that time). The trust.salesforce.com was still showing that all of the services were available and no ongoing incidents reported on the dashboard. People on the internet were all talking about it. Some were angry, some were shocked like me, basically everyone was helpless but to pray. Admins (mostly) were stressed out because of this. It was not an easy day for me though because there was literally nothing that I could do to remedy the situation.

The incident happened at around 9.56 am EST on May 17, 2019. It was right before when my colleagues told me about the issue of creating new case. Right after this, I saw the permission on the object slowly being unchecked, starting from 'Read' , then it was the 'Delete', 'View All' , and lastly 'Modify All' was unchecked as well. I witnessed the whole process on how Salesforce gradually removed the permissions on non-admin profiles. It took roughly 4 hours for all the CRUD permissions to be removed. I estimated the whole recovery process would probably take at most another one hour or two to be backed on and recovered to original state but I was completely wrong about it too. At around 2.00 pm EST ish, I came back to the office from lunch and I found out that my entire Salesforce org had completely down now. But thankfully, the whole thing didn't go down for long, it lasted about one hour and came back online again. The messed-up profiles were still messed-up and not restored yet. I really didn't think it took so long for them to recover though. Our company relied on Salesforce a lot - the whole Sales department, Service department and Accounting department were completely interfered by this incident. I had to 'delegate' their daily usuals at least to keep the whole business flows and operations running.

While everyone were so stressed about this breakdown, some people twitted meme to find pleasure amidst suffering.

It was literally like me on that day...
Salesforce decided to give no access to non-admin profiles (best answer).

I started receiving emails periodically from Salesforce support feedback team to join the webinar to follow up with this incident. It was a database script accidentally deployed to Salesforce that overriding all the custom profiles' permissions to full access. It affected all the customers who currently use Pardot or have used Pardot in the past. The Salesforce engineers were working hard to resolve this issue urgently.

Reply from Salesforce CTO, Parker Harris, after the outage.

Looking at the brightside, this incident happened at the end of the week, at least the impact weren't as huge as if it happened on the beginning of the week. The next day morning (Saturday), I checked again and this issue was still not fixed yet. They had posted a temporary solution to revert back the permissions from Sandbox.

https://success.salesforce.com/issues_view?Id=a1p3A000001SHDl

I tried to submit the changeset with the custom profiles from Sandbox to Production but it wasn't successful though unfortunately. When I read the workaround details again, I just found out that if the sandbox org was impacted, it was not a valid source for recovery. So I was pretty much screwed. This incident had already caused us a whole day of trouble and I didn't think it was optimal to wait for the fix any longer. As a result, I decided to manually create permission sets to give access to the users. Thankfully, our company's profiles were not that complicated. I just had to configure for both Sales and Service profiles making sure things could still be functioning the way it should be.

Up to today (May 19, 2019), the profile permissions are still not restored yet. I am not expecting anything at this point, honestly. I am sure it has been a bigger disaster for other affected companies. I am also sure Salesforce has learned its lesson and will be more careful next time.

Last time I checked (May 19, 2019 9:52 pm EST), the permissions on object have finally been restored. Now I can finally rest and sleep tight for the incoming week. See ya all!

Post was published on , last updated on .

Like the content? Support the author by paypal.me!