This is the part 2 of: My Nightmare with Magento. If you haven't read the Part 1, here is the link 😉!
I asked the support team to place some test orders to make sure things were in the right place. After some testing, the support team told me that there was an issue when they checkout the order. When the checkout button is clicked, all of the items on the cart will be cleared and showing big text "SHOPPING CART IS EMPTY".
I thought that was some "standard" issues and I told them to do more checking on their own see if that was something that they can handle themselves. They agreed to check that on their own. I was so tired and finally dozed off at 6 AM. The sun was up and the sky was getting brighter. I knew I would still have to take a look at it in the morning so I set alarm clock at 10 AM.
Sure enough, when I woke up and checked on my phones, they were so many unread messages from support team saying the website was not working and that the customers were having issues placing orders.
I really didn't want to believe it was true. I really hope it was something that could be fixed right away.
It wasn't. It was a nasty issue.
Anyway, I got up from bed and started logging into the server to check for the issues. 30 mins passed, 1 hour passed and the support team was started getting frustrated as they got more and more complaints from the customers.
The CEO called. "What was the issue with the system? Do you need help from XX (previous developer)?", the CEO asked and I said, "Yeah, I need help!". Within few minutes, the ex-developer reached out to me and asked me what happened. I explained to him the situation and he was actively helping me to check for issues. That was the moment when I saw my life saver. I was not all alone. It felt so much different when you had someone who you could count on to fight with you. Later in the afternoon, my manager and one other technical engineer joined the crew to help out as well.
We did so many things to troubleshoot why the cart became empty after checkout. We used the AMI Image that we created two days ago to launch a new server (the one before the mess up). We set up the database IP address on the app/etc/local.xml (though using endpoint was a preferred and recommended solution), cleared the cache and session files, and lastly, we rebooted the Nginx and Php-form just in case. We did not solve the issue.
We thought there could be a database change when we upgraded the instance size even though it was highly impossible, but we were just trying our luck here. We created another RDS using the backup created two days ago, then we repeated the process again. The result was still a big nope. The developer kept asking if I had modify any other thing in the server and I said no. "How would I edit something that I don't even know? Am I being stupid?", I thought.
Time passed so quickly and we were busy checking and debugging the issue. We also dug further into cookies and sessions in Magento as well as the HTTP-HTTPS redirection. None of those yielded a positive result. Debugging on the server environment was no fun either. The code structure was complicated and debugging efficiency was so low. We were pretty exhausted, at least that was how I felt after working over 24 hours straight. The ex-developer was also working very hard and we stayed up till 3 AM in the morning. We had done so much troubleshooting yet we still couldn't find the real culprit behind. It seemed like we would never be able to get anything else done by the end of the day. On the other hand, the company had begun rolling out a quick migration plan so that our customers could continue using our service with our newer version of the system rather than the old one which was totally stopped working at the moment.
The next day morning, I told the management team that we might need to hire 3rd party Magento developer/expert to help us with the issue because I honestly don't think we are able to pull it off on our own. The management team agreed to hire so I started drafting my job description and posted it on upwork.com which is a freelancer job posting platform. Within minutes after I posted my job request, a few freelancers reached out to me. One of the freelancers reached out to me at the perfect timing and said he could offer some professional help and asked if I could jump on a quick call. I checked out his profile and I think he is the perfect match for this job so I accepted his call and told him the issue that we are facing right now. After I showed him a quick walk-through to reproduce the issue, he said he could get it fix and asked me to hire him on the website. We communicated on Slack afterwards.
I felt a weight has been lifted off my shoulders after getting to know that someone who can really help (of course, we paid for the service). Because I was under huge pressure and I didn't even eat much in a day, I lost 2kg straight and it was my lowest weight in my adult's life.
They started logging into our server at around 12 PM-ish. Since the issue was not somewhat obvious, they actually took a couple of hours before they could identify it was a problem related to MySQL query when saving quote data. They still need some time debugging to find out the real cause of it. He requested to set up a local dev environment tomorrow and see if they can replicate the issue.
At 4:05 AM in the morning, he sent me a message saying that they have found the issue. It was an option the "Use Flat Catalog Product" that causes the reindex to fail due to huge number of products stored in the database, which turns out causing the quote data failed to save to database.
He disabled the feature and things start to get back to normal now.
Because it was a configuration setting that I have set earlier (probably a week or weeks ago), I have no idea that I have set it before which then turns out to be a huge disaster. I was too blindly following whatever solutions provided on the Internet without thinking deeper. The reason why I set this feature because it was claimed to improve the overall performance. Magento didn't tell me that setting this option was failing. It also didn't tell me the reindex was failing as well. If Magento could reflect the option sooner (without manually deleting the caches and sessions), I would have known the issue so much sooner. Instead, the issue was concealed until I manually cleared out the cache and session files for loading new database connection. There were so much more things that could have gone wrong at that point.
By the way, that click to disable the feature costs us over $1,000, but it is definitely worth it. Without their help, we will continue losing so much more and we probably won't be able to figure that out still.
I am so glad this incident is finally over. Now, I have the time to write the post and share with you all. Here are the lessons that I have learned:
-
Be extra cautious when dealing with production server
You never know what kind of s**t you will get into. Look at my case here, I am so sure that upgrading a database instance will not affect anything at all, but here I am. -
Don't simply apply any settings that you don't know
This is the most critical one and we should put the most attention to it. If you are not sure of something, just don't do it. If you want to try your luck, please try it in your dev environment first. Don't be a lazy a** and take shortcuts. Applying the configuration right on the production could cost you a lot more. Look at me. -
Ask for help when needed
Don't always keep it to yourself. Asking for help does not mean you are weak but you are more willing to share with others. It really feels different when you know you have someone on your side working together even if they are not really helping much. Trust me, it is so much different when you are fighting alone vs together in a team. -
Always have backups
Storage is cheap and setting up backup automation is easy. Don't be afraid to pay for any backup incurred fee because the cost of not having backup is much higher than that.
That's all about it! Hope you guys like it!
Post was published on , last updated on .
Like the content? Support the author by paypal.me!
