Thank you for all the continued support for CustomsForge it means the world to us.
I want to start off by apologizing for all the downtime recently. It was never intentional and we’ve been running into several issues over the last week. I’ll start at the beginning:
Around Saturday (Oct. 12, 2019) there were user and staff reports of 502 and other types of errors. I investigated and started adjusting the server configs and other files. Unfortunately, this led to more severe issues so we had to revert to our server backups. For our backups we have two servers - one for MySQL and one for data. The MySQL server was reverted back to 7 days due to the fact that the last two backups had problems in the database.
Additionally, we had to run a backup restore on our data server. Due to the mass amount of files on that server, it can take up to 16 hours to complete. Our backups failed multiple times. We had to have our host work on it and help fix issues with the recovery. While that was running (16 hours 5-6 times is a very big pain and super time consuming) we did have a backup that failed but mostly at OS level. I was able to recover a majority of the files and start a donation goal so we could launch a new server while our host tried to fix the backups. Thanks to Chainbrain and others, we were able to get that server going and I was able to bring CustomsForge back online.
From there even when the server was recovered we’ve been running into multiple problems with the servers including DNS, SSL, and email issues. I’ve been working around the clock trying to resolve any outstanding issues and we still have a lot of work ahead of us.
The Road Ahead:
Fix login issues on the website and improve performance
Rewrite some basic functions on Ignition & the forum backend to prep for MySQL migration
Migrate MySQL to our new improved web server
Prep the website for Forum & Ignition upgrade (please note: no ETA but our goal is within the next few months)
Thanks to everyone for the support and once again I apologize for the downtime, at the moment it’s just me working on everything.
If you’re experienced in Laravel, MySQL, PHP, and Server Administration we’re always looking for people to add to the team to help. You can apply here: https://goo.gl/forms/yWt0FX8GQGie3WGG2
"Just remember: when something breaks, kick it as hard as you fucking can"
"Just remember: when something breaks, kick it as hard as you fucking can"
-Trent Reznor
Support Me (Via Patreon)