-= Building a Guitar from Scratch with Absolutely No Woodworking Experience =-

As you are fully aware by now CustomsForge was completely down for over 24 hours due to a catastrophic backup failure. We worked with our host (Linode) to resolve this but unfortunately, it took 10 hours to just restore the files. We have a lot of upgrades for CustomsForge and we will be working on adding more fail safes to prevent problems like this happening in the future. As we grow into a bigger website we must evolve, and to do so we must upgrade our forums and ignition. During our upgrades, we ran into several problems, and unfortunately we learned about them days after. We were forced to roll back to a backup we took on March 5th. We still need to do these upgrades but we will be working closely with our board software support staff this time to fix the bug that caused these issues. If you have any questions or concerns feel free to ask us on our discord. We will be attempting these upgrades again, but in a different method which will reduce any future downtime if any problem was to occur again. (Example would be 30-minute downtime~ instead of 24 hours)

 

(p.s. if my wording sounds weird, I've been working on this non-stop for 24 hours trying to get this fixed for you guys, no sleep until I made sure our community is fixed! #exhausting) 

 

- Mark

I notice there's a gap in uploads to ignition, from 5th March?  Do people who uploaded while there were problems need to upload again?

I notice there's a gap in uploads to ignition, from 5th March?  Do people who uploaded while there were problems need to upload again?

That is correct, everything from March 5th-11th is gone unfortunately.

Thanks for the hard work.  Its great to see it working again.

As we grow into a bigger website we must evolve, and to do so we must upgrade our forums and ignition. During our upgrades, we ran into several problems, and unfortunately we learned about them days after. We were forced to roll back to a backup we took on March 5th.

 

Is there no development instance where you may test things before upgrading the productive system?

 

I have no discord - was the backup broken since 5th and you had to go back each day to find a working backup copy? This was then really bad luck, because when upgrading there's always a chance that something goes wrong and if you can't rely on the backup you're lost.

 

As we grow into a bigger website we must evolve, and to do so we must upgrade our forums and ignition. During our upgrades, we ran into several problems, and unfortunately we learned about them days after. We were forced to roll back to a backup we took on March 5th.

 

Is there no development instance where you may test things before upgrading the productive system?

 

I have no discord - was the backup broken since 5th and you had to go back each day to find a working backup copy? This was then really bad luck, because when upgrading there's always a chance that something goes wrong and if you can't rely on the backup you're lost.

 

There is a developer instance but it is currently being used for IgnitionFUSE / IPBoard 4.1 Testing. During the upgrades everything was going smooth and there was only a few reports that could of been fixed, but unfortunately we found some severe problems a few days later. We had to make the decision to either A) rebuild B) Revert. We went ahead and chose B. Then during the backup, our backups failed. We have multiple backups but on the 5th is when we started the modifications and I took a snapshot, we have two backups that day, both failed. Linode's backup administrator went in and fixed the backup and it took 13 hours+ to restore the backup, and it worked. There was a bug during the upgrade process that we are working on for IP Board (we need to upgrade nginx, php, and convert our database) that occured that wasn't supposed to happen. IP Board will be working with us next time we attempt this to fix that bug, and we will have a backup ready. And better yet, next time it will take about 5-10 minutes to restore it instead of 12 hours+.

 

We don't have any proper system admins and this is all on me. I am a support tech at a web hosting company so I do know how to do this stuff, but I'm not certified and there are people way more knowledgeable who could volunteer and take on the job and assist us. Unfortunately, at the moment I am managing it and trying to do my best. Our last server admin betrayed us and was a liar. The one before that completely disappeared off the face of the planet never to be heard again. But I am always up to the task and I will work on CustomsForge as hard as I can. I love this site, this community, this game. I am extremely passionate and I hope we can definitely push forward and upgrade these damn forums and fix ignition!

Thx for the hard work, Mark!

 

Up and rocking again!

 

sent from mobile

Thanks for the hard work Mark.

I just donated but it has gone onto the Months goal rather than the emergency one (only noticed after donating) please swap it over or use it as needed :)

We highly appreciate the hard work and efforts you put into this site, i've seen it grow quite significantly over the past year. I'm sure everyone here highly supports you and will continue to for the years to come

Unfortunately i tried to download the latest version 1301 but i cant because the source cant be found error...is it something on my hand or has it to do with the site....cheers

Unfortunately i tried to download the latest version 1301 but i cant because the source cant be found error...is it something on my hand or has it to do with the site....cheers

I believe you have replied to the wrong topic , 1301 you are refering to is csm (customsforge song manager) the link to which is still on the dedicated website and does still work as i have just checked it . the link can be found here clicky me

Ohh ok thx my thought was that it something has to do with this website and offline and stuff missing thing...anyway sorry for the trouble i will sort it out ..peace

 

As you are fully aware by now CustomsForge was completely down for over 24 hours due to a catastrophic backup failure. We worked with our host (Linode) to resolve this but unfortunately, it took 10 hours to just restore the files. We have a lot of upgrades for CustomsForge and we will be working on adding more fail safes to prevent problems like this happening in the future. As we grow into a bigger website we must evolve, and to do so we must upgrade our forums and ignition. During our upgrades, we ran into several problems, and unfortunately we learned about them days after. We were forced to roll back to a backup we took on March 5th. We still need to do these upgrades but we will be working closely with our board software support staff this time to fix the bug that caused these issues. If you have any questions or concerns feel free to ask us on our discord. We will be attempting these upgrades again, but in a different method which will reduce any future downtime if any problem was to occur again. (Example would be 30-minute downtime~ instead of 24 hours)

 

(p.s. if my wording sounds weird, I've been working on this non-stop for 24 hours trying to get this fixed for you guys, no sleep until I made sure our community is fixed! #exhausting) 

 

- Mark

 

Hate to burst yo bubble bro, but ignition is down again. Says there is no such thing.

Probably when I was working on it- it's up again.

Just wanted to say thanks for everything you do here. It's in no small part down to you guys and the contributors here that my playing has improved more over the last few months than it had over the previous 20 years.

If ain't broken... Don't fix it...

Your work and your dedication is greatly appreciated!

bit late but only just read this, thanks for your work Mark

Thank you all for your hard work on this site. Happy New Year

Not sure where else to put this and couldn't find a place to contact an admin directly but I've been trying to use the CDLC search for the last few days and when I am scrolling through the recent adds on the main CDLC page I get redirected to an unsecured website telling me my flash player is out of date. I have confirmed that my flash is definitely not out of date. It pretty much is making it so I can't use the CDLC search and I am constantly having to close/reopen the web page because the flash player message is clearly some kind of malware attempt. I am on a Mac if that makes any difference or helps. I just want to bring it to the admin's attention in case it is someone trying to attack Custom Forge users. I have screenshot of the website and message that is coming up. I can provide it to someone if they end up looking into it. Feel free to reach out to me.