Jump to content

Welcome to CustomsForge
Register now to gain access to all of our features. Once registered and logged in, you will be able to create topics, post replies to existing threads, give reputation to your fellow members, get your own private messenger, post status updates, manage your profile and so much more. This message will be removed once you have signed in.
Login to Account Create an Account

Learn & Play Rocksmith!

Want to play the songs you love? Registration and the use of this website is 100% free.
The only thing you need is a legal copy of Rocksmith 2014 w/ a cable. Click here to buy it.
 
Having trouble logging in or registering? Please click "Support" below.


Donate

Our end the month goal is up! $105 remaining. Thank you for donating.
We need to hit our goal by the 29th to keep the servers up.
Official Patreon - Facebook - Twitter

We're needing more volunteers for future projects. This includes:
- Laravel Developers (We're looking for a new Lead Developer *PAID*)
--- We need a developer to rewrite our auth code ASAP. Please apply immediately if you can help. ---
- Server Admin (*PAID*)
- Moderators
- C# Developers (& .Net Core)
Please apply here: https://goo.gl/forms/Y1irWCLoqODGKQVr1


Photo

Growing Pains


  • Please log in to reply
17 replies to this topic

#1
Online   Unleashed2k

Unleashed2k

    CustomsForge Founder

  • Administrator

  • PipPipPipPipPipPip
  • 738 posts
  • 457 thanks
  • LocationOrlando, FL
  • Rocksmith:Steam
  • XBOX:Unleashed2k
  • PSN:FWF_Unleashed2k
As you are fully aware by now CustomsForge was completely down for over 24 hours due to a catastrophic backup failure. We worked with our host (Linode) to resolve this but unfortunately, it took 10 hours to just restore the files. We have a lot of upgrades for CustomsForge and we will be working on adding more fail safes to prevent problems like this happening in the future. As we grow into a bigger website we must evolve, and to do so we must upgrade our forums and ignition. During our upgrades, we ran into several problems, and unfortunately we learned about them days after. We were forced to roll back to a backup we took on March 5th. We still need to do these upgrades but we will be working closely with our board software support staff this time to fix the bug that caused these issues. If you have any questions or concerns feel free to ask us on our discord. We will be attempting these upgrades again, but in a different method which will reduce any future downtime if any problem was to occur again. (Example would be 30-minute downtime~ instead of 24 hours)
 
(p.s. if my wording sounds weird, I've been working on this non-stop for 24 hours trying to get this fixed for you guys, no sleep until I made sure our community is fixed! #exhausting) 
 
- Mark

"Just remember: when something breaks, kick it as hard as you fucking can"

-Trent Reznor


#2
Online   missis sumner

missis sumner

    Not exactly an early adopter...

  • Donator
  • PipPipPipPipPipPipPip
  • 1,439 posts
  • 38 thanks
  • LocationUK
  • Rocksmith:Steam

I notice there's a gap in uploads to ignition, from 5th March?  Do people who uploaded while there were problems need to upload again?


All about the bass, no treble...

 

https://docs.google....WP8/edit#gid=37


#3
Online   Unleashed2k

Unleashed2k

    CustomsForge Founder

  • Administrator

  • PipPipPipPipPipPip
  • 738 posts
  • 457 thanks
  • LocationOrlando, FL
  • Rocksmith:Steam
  • XBOX:Unleashed2k
  • PSN:FWF_Unleashed2k

I notice there's a gap in uploads to ignition, from 5th March?  Do people who uploaded while there were problems need to upload again?

That is correct, everything from March 5th-11th is gone unfortunately.


"Just remember: when something breaks, kick it as hard as you fucking can"

-Trent Reznor


#4
Offline   MaZtoR

MaZtoR

    Great A'Tuin

  • Magna Charter
  • PipPipPipPipPipPipPip
  • 1,314 posts
  • 54 thanks
  • LocationSweden, Åhus
  • Rocksmith:Steam

User's Awards

              

Thanks for the hard work.  Its great to see it working again.



#5
Offline   papillon

papillon

    Master Guitarist

  • Member
  • PipPipPipPipPip
  • 369 posts
  • 30 thanks
  • Rocksmith:Steam

As we grow into a bigger website we must evolve, and to do so we must upgrade our forums and ignition. During our upgrades, we ran into several problems, and unfortunately we learned about them days after. We were forced to roll back to a backup we took on March 5th.

 

Is there no development instance where you may test things before upgrading the productive system?

 

I have no discord - was the backup broken since 5th and you had to go back each day to find a working backup copy? This was then really bad luck, because when upgrading there's always a chance that something goes wrong and if you can't rely on the backup you're lost.



#6
Online   Unleashed2k

Unleashed2k

    CustomsForge Founder

  • Administrator

  • PipPipPipPipPipPip
  • 738 posts
  • 457 thanks
  • LocationOrlando, FL
  • Rocksmith:Steam
  • XBOX:Unleashed2k
  • PSN:FWF_Unleashed2k

 

As we grow into a bigger website we must evolve, and to do so we must upgrade our forums and ignition. During our upgrades, we ran into several problems, and unfortunately we learned about them days after. We were forced to roll back to a backup we took on March 5th.

 

Is there no development instance where you may test things before upgrading the productive system?

 

I have no discord - was the backup broken since 5th and you had to go back each day to find a working backup copy? This was then really bad luck, because when upgrading there's always a chance that something goes wrong and if you can't rely on the backup you're lost.

 

There is a developer instance but it is currently being used for IgnitionFUSE / IPBoard 4.1 Testing. During the upgrades everything was going smooth and there was only a few reports that could of been fixed, but unfortunately we found some severe problems a few days later. We had to make the decision to either A) rebuild B) Revert. We went ahead and chose B. Then during the backup, our backups failed. We have multiple backups but on the 5th is when we started the modifications and I took a snapshot, we have two backups that day, both failed. Linode's backup administrator went in and fixed the backup and it took 13 hours+ to restore the backup, and it worked. There was a bug during the upgrade process that we are working on for IP Board (we need to upgrade nginx, php, and convert our database) that occured that wasn't supposed to happen. IP Board will be working with us next time we attempt this to fix that bug, and we will have a backup ready. And better yet, next time it will take about 5-10 minutes to restore it instead of 12 hours+.

We don't have any proper system admins and this is all on me. I am a support tech at a web hosting company so I do know how to do this stuff, but I'm not certified and there are people way more knowledgeable who could volunteer and take on the job and assist us. Unfortunately, at the moment I am managing it and trying to do my best. Our last server admin betrayed us and was a liar. The one before that completely disappeared off the face of the planet never to be heard again. But I am always up to the task and I will work on CustomsForge as hard as I can. I love this site, this community, this game. I am extremely passionate and I hope we can definitely push forward and upgrade these damn forums and fix ignition!


"Just remember: when something breaks, kick it as hard as you fucking can"

-Trent Reznor


Thanked by 2 Members:

#7
Online   Rodman

Rodman

    AAA

  • Rocksmith Championship Organizer

  • PipPipPipPipPipPipPipPip
  • 3,555 posts
  • 214 thanks
  • LocationKlagenfurt, Austria
  • Rocksmith:Steam

User's Awards

           
Thx for the hard work, Mark!

Up and rocking again!

sent from mobile

#8
Offline   Bluetac

Bluetac

    Touring Guitarist

  • Donator
  • PipPipPipPipPipPip
  • 551 posts
  • 21 thanks
  • LocationUK
  • Rocksmith:Steam

Thanks for the hard work Mark.

I just donated but it has gone onto the Months goal rather than the emergency one (only noticed after donating) please swap it over or use it as needed :)


76561197968639313.png

Guitars: ESP LTD F-54 Bass, Tanglewood Curbow4 Bass, Stagg BC300-N Bass, Yamaha BBN5bk (5String), ESP LTD AX-360, Plus a cheapo no name acoustic


#9
Offline   AxLShiv

AxLShiv

    Expert Guitarist

  • Magna Charter
  • PipPipPipPip
  • 151 posts
  • 3 thanks
  • LocationDurban
  • Rocksmith:Steam

We highly appreciate the hard work and efforts you put into this site, i've seen it grow quite significantly over the past year. I'm sure everyone here highly supports you and will continue to for the years to come



#10
Offline   Daslekker

Daslekker

    Beginner Guitarist

  • Member
  • Pip
  • 6 posts
  • 0 thanks
  • Rocksmith:Steam

Unfortunately i tried to download the latest version 1301 but i cant because the source cant be found error...is it something on my hand or has it to do with the site....cheers



#11
Offline   Bluetac

Bluetac

    Touring Guitarist

  • Donator
  • PipPipPipPipPipPip
  • 551 posts
  • 21 thanks
  • LocationUK
  • Rocksmith:Steam

Unfortunately i tried to download the latest version 1301 but i cant because the source cant be found error...is it something on my hand or has it to do with the site....cheers

I believe you have replied to the wrong topic , 1301 you are refering to is csm (customsforge song manager) the link to which is still on the dedicated website and does still work as i have just checked it . the link can be found here clicky me


76561197968639313.png

Guitars: ESP LTD F-54 Bass, Tanglewood Curbow4 Bass, Stagg BC300-N Bass, Yamaha BBN5bk (5String), ESP LTD AX-360, Plus a cheapo no name acoustic


#12
Offline   Daslekker

Daslekker

    Beginner Guitarist

  • Member
  • Pip
  • 6 posts
  • 0 thanks
  • Rocksmith:Steam

Ohh ok thx my thought was that it something has to do with this website and offline and stuff missing thing...anyway sorry for the trouble i will sort it out ..peace



#13
Offline   TheVanillaMiner

TheVanillaMiner

    Beginner Guitarist

  • Member
  • Pip
  • 37 posts
  • 0 thanks
  • Rocksmith:XBOX 360

 

As you are fully aware by now CustomsForge was completely down for over 24 hours due to a catastrophic backup failure. We worked with our host (Linode) to resolve this but unfortunately, it took 10 hours to just restore the files. We have a lot of upgrades for CustomsForge and we will be working on adding more fail safes to prevent problems like this happening in the future. As we grow into a bigger website we must evolve, and to do so we must upgrade our forums and ignition. During our upgrades, we ran into several problems, and unfortunately we learned about them days after. We were forced to roll back to a backup we took on March 5th. We still need to do these upgrades but we will be working closely with our board software support staff this time to fix the bug that caused these issues. If you have any questions or concerns feel free to ask us on our discord. We will be attempting these upgrades again, but in a different method which will reduce any future downtime if any problem was to occur again. (Example would be 30-minute downtime~ instead of 24 hours)
 
(p.s. if my wording sounds weird, I've been working on this non-stop for 24 hours trying to get this fixed for you guys, no sleep until I made sure our community is fixed! #exhausting) 
 
- Mark

 

Hate to burst yo bubble bro, but ignition is down again. Says there is no such thing.



#14
Online   Unleashed2k

Unleashed2k

    CustomsForge Founder

  • Administrator

  • PipPipPipPipPipPip
  • 738 posts
  • 457 thanks
  • LocationOrlando, FL
  • Rocksmith:Steam
  • XBOX:Unleashed2k
  • PSN:FWF_Unleashed2k
Probably when I was working on it- it's up again.

"Just remember: when something breaks, kick it as hard as you fucking can"

-Trent Reznor


#15
Offline   RubberDave

RubberDave

    Medium Guitarist

  • Donator
  • PipPip
  • 57 posts
  • 0 thanks
  • Rocksmith:Steam
Just wanted to say thanks for everything you do here. It's in no small part down to you guys and the contributors here that my playing has improved more over the last few months than it had over the previous 20 years.

#16
Offline   El_Noodle

El_Noodle

    Beginner Guitarist

  • Member
  • Pip
  • 18 posts
  • 0 thanks

If ain't broken... Don't fix it...



#17
Offline   deansmayhem

deansmayhem

    Hardcore Guitarist

  • Ex-Staff Member
  • PipPipPip
  • 83 posts
  • 1 thanks
  • LocationMpls. MN
  • Rocksmith:Steam

Donator

Your work and your dedication is greatly appreciated!



#18
Offline   bLiNx32

bLiNx32

    Medium Guitarist

  • Donator
  • PipPip
  • 62 posts
  • 0 thanks
  • LocationCrewe, Cheshire, UK
  • Rocksmith:Steam
  • XBOX:bLiNx45
  • PSN:bLiNx32

bit late but only just read this, thanks for your work Mark