Jump to content

Growing Pains


Recommended Posts

  • Administrator
As you are fully aware by now CustomsForge was completely down for over 24 hours due to a catastrophic backup failure. We worked with our host (Linode) to resolve this but unfortunately, it took 10 hours to just restore the files. We have a lot of upgrades for CustomsForge and we will be working on adding more fail safes to prevent problems like this happening in the future. As we grow into a bigger website we must evolve, and to do so we must upgrade our forums and ignition. During our upgrades, we ran into several problems, and unfortunately we learned about them days after. We were forced to roll back to a backup we took on March 5th. We still need to do these upgrades but we will be working closely with our board software support staff this time to fix the bug that caused these issues. If you have any questions or concerns feel free to ask us on our discord. We will be attempting these upgrades again, but in a different method which will reduce any future downtime if any problem was to occur again. (Example would be 30-minute downtime~ instead of 24 hours)

 

(p.s. if my wording sounds weird, I've been working on this non-stop for 24 hours trying to get this fixed for you guys, no sleep until I made sure our community is fixed! #exhausting) 

 

- Mark

  • Like 39

"Just remember: when something breaks, kick it as hard as you fucking can"

-Trent Reznor
Support Me (Via Patreon)

Link to comment
Share on other sites

  • Administrator

I notice there's a gap in uploads to ignition, from 5th March?  Do people who uploaded while there were problems need to upload again?

That is correct, everything from March 5th-11th is gone unfortunately.

"Just remember: when something breaks, kick it as hard as you fucking can"

-Trent Reznor
Support Me (Via Patreon)

Link to comment
Share on other sites

As we grow into a bigger website we must evolve, and to do so we must upgrade our forums and ignition. During our upgrades, we ran into several problems, and unfortunately we learned about them days after. We were forced to roll back to a backup we took on March 5th.

 

Is there no development instance where you may test things before upgrading the productive system?

 

I have no discord - was the backup broken since 5th and you had to go back each day to find a working backup copy? This was then really bad luck, because when upgrading there's always a chance that something goes wrong and if you can't rely on the backup you're lost.

Link to comment
Share on other sites

  • Administrator

 

As we grow into a bigger website we must evolve, and to do so we must upgrade our forums and ignition. During our upgrades, we ran into several problems, and unfortunately we learned about them days after. We were forced to roll back to a backup we took on March 5th.

 

Is there no development instance where you may test things before upgrading the productive system?

 

I have no discord - was the backup broken since 5th and you had to go back each day to find a working backup copy? This was then really bad luck, because when upgrading there's always a chance that something goes wrong and if you can't rely on the backup you're lost.

 

There is a developer instance but it is currently being used for IgnitionFUSE / IPBoard 4.1 Testing. During the upgrades everything was going smooth and there was only a few reports that could of been fixed, but unfortunately we found some severe problems a few days later. We had to make the decision to either A) rebuild B) Revert. We went ahead and chose B. Then during the backup, our backups failed. We have multiple backups but on the 5th is when we started the modifications and I took a snapshot, we have two backups that day, both failed. Linode's backup administrator went in and fixed the backup and it took 13 hours+ to restore the backup, and it worked. There was a bug during the upgrade process that we are working on for IP Board (we need to upgrade nginx, php, and convert our database) that occured that wasn't supposed to happen. IP Board will be working with us next time we attempt this to fix that bug, and we will have a backup ready. And better yet, next time it will take about 5-10 minutes to restore it instead of 12 hours+.

 

We don't have any proper system admins and this is all on me. I am a support tech at a web hosting company so I do know how to do this stuff, but I'm not certified and there are people way more knowledgeable who could volunteer and take on the job and assist us. Unfortunately, at the moment I am managing it and trying to do my best. Our last server admin betrayed us and was a liar. The one before that completely disappeared off the face of the planet never to be heard again. But I am always up to the task and I will work on CustomsForge as hard as I can. I love this site, this community, this game. I am extremely passionate and I hope we can definitely push forward and upgrade these damn forums and fix ignition!

  • Like 8

"Just remember: when something breaks, kick it as hard as you fucking can"

-Trent Reznor
Support Me (Via Patreon)

Link to comment
Share on other sites

  • Rocksmith Championship Organizer

Thx for the hard work, Mark!

 

Up and rocking again!

 

sent from mobile

  • Like 2

-= Building a Guitar from Scratch with Absolutely No Woodworking Experience =-
-= ROCK - Rodmans Course for Kids and Beginners =-

-=GET RID OF ON DISC SONGS IN THE SONG SELECTION LIST? , FULL SETLIST MANAGEMENT? - YOU NEED A ROADIE !!!=-
-= SPOTIFY Rocksmith CS Playlist =-
-= Use your DLC library on multiple PCs w. autoupdate =-
-= Build your own RS Toneswitch Fb =-

-= Join the Championship - cause the Championship is good for you =-
-= What´s my RANK again??? - check the 10-weeks Leaderboard and PLAYER Level =-
-= Put the Songs you want to be played into SONG SELECTION LIST =-

 

On 11/8/2016 at 8:32 PM, Vodka said:

I hated Royal Republic prev. time. I hate it this time too.

UPD: OK. I hate every song. I guess I need to stop.

 

Link to comment
Share on other sites

Thanks for the hard work Mark.

I just donated but it has gone onto the Months goal rather than the emergency one (only noticed after donating) please swap it over or use it as needed :)

Guitars: ESP LTD F-54 Bass, Tanglewood Curbow4 Bass, Stagg BC300-N Bass, Yamaha BBN5bk (5String), ESP LTD AX-360, Plus a cheapo no name acoustic

Link to comment
Share on other sites

Unfortunately i tried to download the latest version 1301 but i cant because the source cant be found error...is it something on my hand or has it to do with the site....cheers

I believe you have replied to the wrong topic , 1301 you are refering to is csm (customsforge song manager) the link to which is still on the dedicated website and does still work as i have just checked it . the link can be found here clicky me

Guitars: ESP LTD F-54 Bass, Tanglewood Curbow4 Bass, Stagg BC300-N Bass, Yamaha BBN5bk (5String), ESP LTD AX-360, Plus a cheapo no name acoustic

Link to comment
Share on other sites

 

As you are fully aware by now CustomsForge was completely down for over 24 hours due to a catastrophic backup failure. We worked with our host (Linode) to resolve this but unfortunately, it took 10 hours to just restore the files. We have a lot of upgrades for CustomsForge and we will be working on adding more fail safes to prevent problems like this happening in the future. As we grow into a bigger website we must evolve, and to do so we must upgrade our forums and ignition. During our upgrades, we ran into several problems, and unfortunately we learned about them days after. We were forced to roll back to a backup we took on March 5th. We still need to do these upgrades but we will be working closely with our board software support staff this time to fix the bug that caused these issues. If you have any questions or concerns feel free to ask us on our discord. We will be attempting these upgrades again, but in a different method which will reduce any future downtime if any problem was to occur again. (Example would be 30-minute downtime~ instead of 24 hours)
 
(p.s. if my wording sounds weird, I've been working on this non-stop for 24 hours trying to get this fixed for you guys, no sleep until I made sure our community is fixed! #exhausting) 
 
- Mark

 

Hate to burst yo bubble bro, but ignition is down again. Says there is no such thing.

Link to comment
Share on other sites

  • 6 months later...
  • 3 months later...
  • 1 year later...

Not sure where else to put this and couldn't find a place to contact an admin directly but I've been trying to use the CDLC search for the last few days and when I am scrolling through the recent adds on the main CDLC page I get redirected to an unsecured website telling me my flash player is out of date. I have confirmed that my flash is definitely not out of date. It pretty much is making it so I can't use the CDLC search and I am constantly having to close/reopen the web page because the flash player message is clearly some kind of malware attempt. I am on a Mac if that makes any difference or helps. I just want to bring it to the admin's attention in case it is someone trying to attack Custom Forge users. I have screenshot of the website and message that is coming up. I can provide it to someone if they end up looking into it. Feel free to reach out to me. 

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Unfortunately, your content contains terms that we do not allow. Please edit your content to remove the highlighted words below.
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...

Important Information

By using this site, you agree to our Guidelines. We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue. - Privacy Policy