Jump to content

Welcome to CustomsForge Forums
Register now to gain access to all of our features. Once registered and logged in, you will be able to create topics, post replies to existing threads, give reputation to your fellow members, get your own private messenger, post status updates, manage your profile and so much more. This message will be removed once you have signed in.
Login to Account Create an Account

Learn & Play Rocksmith!

Want to play the songs you love? Registration and the use of this website is 100% free.
The only thing you need is a legal copy of Rocksmith 2014 w/ a cable. Click here to buy it.
 
Having trouble logging in or registering? Please click "Support" below.


Announcements:

October donation goal is up! click here to donate.

Join us on Discord | Facebook | Twitter | Patreon | Merch | Volunteer | Newsletter Issue 11 (Sept) | Interview with Rodman (Championship)
Known issue with records not loading, please download from popup modal.


Photo

Non English Lyrics/Extended ASCII

- - - - -

  • Please log in to reply
25 replies to this topic

#1
Offline   PC Plum

PC Plum

    Guitar God

  • Member
  • PipPipPipPipPipPipPipPip
  • 2,012 posts
  • Location:Edinburgh

Well I got around to making a song with some non English characters.  Kaijin kindly obliged by producing a set of synced Portugese lyrics for a little acoustic ditty from Youtube.

 

I am not sure if the problem is EoF or the Toolkit, and I decided to post here.

 

For reference I am using EoF v1.8RC12 (4-17-2017) and Toolkit 2.8.2.0.  Also, I have Abridged XMLs turned on.

 

And it is a software problem because the game has no problem displaying all characters in song/singer names

 

 

https://drive.google...iew?usp=sharing

 

https://drive.google...iew?usp=sharing

 

 

It just has issues displaying them in the lyric lines.

 

 

I wanted to post here because there is a difference I can reproduce at EoF level.  When I use the PART VOCALS_RS2_EXT to build my .psarc I get quite different results from using the normal PART VOCALS_RS2.  Both files were spit out by EoF during the same save.

 

Using the _EXT I get

 

https://drive.google...iew?usp=sharing

 

Broken line breaks and all of the accented characters are removed.

 

 

When I use the "normal" Vocals XML I get

 

https://drive.google...iew?usp=sharing

 

https://drive.google...iew?usp=sharing

 

Working line breaks and only the "a" with the squigly line removed (sorry I don't know what its called).

 

Clearly the game can display the "a" as its in the name of the artist.

 

 

 

A question I have is, even though I am using the latest EoF it warns me when I save that I have non-ascii charaters.  Are non-ascii characters different from extended ascii characters?

 

Any ideas/thoughts on the evidence above?

 

I can make a pretty working version for release just now, it's just going to miss out the squigly "a" from the lyrics.  Everything else works technically correct; the song plays, finishes and scores as expected.

 

I suppose I could upload the files for inspection

 

https://drive.google...iew?usp=sharing

 

 

 

 

Thanks :D



#2
Offline   cozy1

cozy1

    Master Peon

  • Developer

  • PipPipPipPipPipPipPip
  • 1,170 posts

@PC Plum What specific github version of toolkit are you using?  Need more than just 2.8.2.0. 


Are you tired of AV False Positives???  Now accepting donations on my website (Click Here)  Your donation will be used towards buying a code signing certificate.   CGT is now compatible and safe to use with Rocksmith® 2014 Remastered ... 

 

Latest Build of Custom Game Toolkit (CGT) w/ Game Save Gigbox       Latest Build of Rocksmith Toolkit       Latest Build of Customs Forge Song Manager (CFSM)

 

All bug reports and help requests please include your: OS, CPU, AV, .NET Framework versions along with a description of the issue (include screenshots of error if possible).  It should go without having to say ... make sure you are using the latest build before submitting bug reports or asking for help.

 

*  Remember to use your magic words (please and thank you) if you would like a response.  Don't use phrases like 'thanks anyhow' as it is demeaning.


#3
Offline   PC Plum

PC Plum

    Guitar God

  • Member
  • PipPipPipPipPipPipPipPip
  • 2,012 posts
  • Location:Edinburgh

I downloaded it today, so it looks like

 

Latest test build - 2.8.2.0 (rev 3ff0b529)

 

 

 

 

Thanks :D



#4
Offline   cozy1

cozy1

    Master Peon

  • Developer

  • PipPipPipPipPipPipPip
  • 1,170 posts

Looking at the two vocals xml ... the 'PART VOCALS_RS2.XML' contains the extended characters which seem to show up in the game based on your screen capture.

 

The ''PART VOCALS_RS2_EXT.XML' do not contain extended characters.

 

So it appears the two different vocals are working as expected.    Your question was 'Are non-ascii characters different from extended ascii characters?'  Have a look here which should explain it.  iminashi explains it here.  raynebc explains it here.  I use the terms  'extended characters' and 'special characters' interchangeably  ...  for which I could be ridiculed.  ;)


Are you tired of AV False Positives???  Now accepting donations on my website (Click Here)  Your donation will be used towards buying a code signing certificate.   CGT is now compatible and safe to use with Rocksmith® 2014 Remastered ... 

 

Latest Build of Custom Game Toolkit (CGT) w/ Game Save Gigbox       Latest Build of Rocksmith Toolkit       Latest Build of Customs Forge Song Manager (CFSM)

 

All bug reports and help requests please include your: OS, CPU, AV, .NET Framework versions along with a description of the issue (include screenshots of error if possible).  It should go without having to say ... make sure you are using the latest build before submitting bug reports or asking for help.

 

*  Remember to use your magic words (please and thank you) if you would like a response.  Don't use phrases like 'thanks anyhow' as it is demeaning.


#5
Offline   raynebc

raynebc

    Guitar God

  • Rocksmith Custom Developer

  • PipPipPipPipPipPipPipPip
  • 2,885 posts
When I re-save the project, EOF writes a UTF lyric XML file with the ã character, so my guess it's something needed on the toolkit's side of things. Of course, let me know if you find otherwise.

#6
Offline   PC Plum

PC Plum

    Guitar God

  • Member
  • PipPipPipPipPipPipPipPip
  • 2,012 posts
  • Location:Edinburgh

the 'PART VOCALS_RS2.XML' contains the extended characters which seem to show up in the game

 

"a" squigly does not show up in game, so its not in the Vocals_XML?

 

I haven't checked, there's no point you guys know a whole lot more anyway :D

 

 

 

Whats a UTF file, is it something normal EoF users can get?

 

 

 

I'll have a read at your links soon cozy, got the tabs to finish up first :D



#7
Offline   iminashi

iminashi

    Master Guitarist

  • Member
  • PipPipPipPipPip
  • 388 posts
  • Location:Finland

Whats a UTF file, is it something normal EoF users can get?

 
By using the PART VOCALS_RS2.XML file you are already using a UTF file.  :)
The vocals XML file is encoded in UTF-8 (Unicode) whereas the _EXT file is encoded in Windows-1252. You don't need to worry about that. Just know that the _EXT file does not work properly with the Toolkit.
 
ã is not in the default lyrics font so it will not work. Since it's written into the UTF lyrics file, it would seem that EOF's character filtering is not working correctly.
 
For the _EXT file the filtering does work properly; ã is filtered out.
 
The + characters for line breaks are not being written into the _EXT file.


#8
Offline   raynebc

raynebc

    Guitar God

  • Rocksmith Custom Developer

  • PipPipPipPipPipPipPipPip
  • 2,885 posts

I've fixed the filtering.  The understanding seems to be that even though Rocksmith can display certain characters like ã for artist/song name, it's not supported for lyrics.  If that's not the case, let me know.



#9
Offline   PC Plum

PC Plum

    Guitar God

  • Member
  • PipPipPipPipPipPipPipPip
  • 2,012 posts
  • Location:Edinburgh

How do we actually get ã into the game to test?

 

I tried manually adding them to the normal Vocals_XML but they were still filtered out in game.  On checking, Notepad++ does save the character, it must be filtered out when I add it to the toolkit (and the toolkit edits the XML file)?

 

It's also filtered out when I simply load up the toolkit save and re-generate.

 

 

 

If the game doesn't like it fair enough, I can move on.  I'm in the mood for testing this out just now though :D



#10
Offline   raynebc

raynebc

    Guitar God

  • Rocksmith Custom Developer

  • PipPipPipPipPipPipPipPip
  • 2,885 posts

@Chlipouni had tested all of the extended ASCII characters to find which ones would work in-game:

http://customsforge....394#entry250394



#11
Offline   PC Plum

PC Plum

    Guitar God

  • Member
  • PipPipPipPipPipPipPipPip
  • 2,012 posts
  • Location:Edinburgh

I see that I am a bit behind and that this has already been tested quite extensively.  I just presumed when I read some of the updates coming to EoF that "extended ascii" would mean that we could have any lyrics we wanted really, and ã seems to be handled no problem by web browers and every other program except the lyric lines of Rocksmith :D

 

I have asked Kaijin for some input from a language perspective.  Maybe a working character can be used as a substitute for ã while making linguistic sense.

 

 

 

Thanks all :D



#12
Offline   cozy1

cozy1

    Master Peon

  • Developer

  • PipPipPipPipPipPipPip
  • 1,170 posts

 

Whats a UTF file, is it something normal EoF users can get?

 
By using the PART VOCALS_RS2.XML file you are already using a UTF file.  :)
The vocals XML file is encoded in UTF-8 (Unicode) whereas the _EXT file is encoded in Windows-1252. You don't need to worry about that. Just know that the _EXT file does not work properly with the Toolkit.
 
ã is not in the default lyrics font so it will not work. Since it's written into the UTF lyrics file, it would seem that EOF's character filtering is not working correctly.
 
For the _EXT file the filtering does work properly; ã is filtered out.
 
The + characters for line breaks are not being written into the _EXT file.

 

So @raynebc seems EOF and toolkit are strictly correct because as @iminashi has pointed out, ã is not an ODLC vocal character list.  We could add it to the vocals validation/filtering in EOF and toolkit and see if it works in the game.

 

EDIT:  I have added ã to the latest toolkit beta list of approved lyric characters for testing.


Are you tired of AV False Positives???  Now accepting donations on my website (Click Here)  Your donation will be used towards buying a code signing certificate.   CGT is now compatible and safe to use with Rocksmith® 2014 Remastered ... 

 

Latest Build of Custom Game Toolkit (CGT) w/ Game Save Gigbox       Latest Build of Rocksmith Toolkit       Latest Build of Customs Forge Song Manager (CFSM)

 

All bug reports and help requests please include your: OS, CPU, AV, .NET Framework versions along with a description of the issue (include screenshots of error if possible).  It should go without having to say ... make sure you are using the latest build before submitting bug reports or asking for help.

 

*  Remember to use your magic words (please and thank you) if you would like a response.  Don't use phrases like 'thanks anyhow' as it is demeaning.


#13
Offline   PC Plum

PC Plum

    Guitar God

  • Member
  • PipPipPipPipPipPipPipPip
  • 2,012 posts
  • Location:Edinburgh

Thanks cozy1, I'll give it a go later :D
 

Speaking to Kaijin, if all else pails I can just use a normal English "a" as a substitue.  Apparently, Portugese people are well used to their language being abused :lol:

 

 

On a serious note though, it's clear from the screenies in my op that the _EXT is filtering out at least one working character.

 

https://drive.google...URSZk5ObUU/view

 

"são só meus" becomes "so s meus", where the ó works fine using the normal Vocal_XML

 

https://drive.google...mFlVVZJa28/view

 

Incidentally, ó is listed as working in Chillipouni's list too.



#14
Offline   PC Plum

PC Plum

    Guitar God

  • Member
  • PipPipPipPipPipPipPipPip
  • 2,012 posts
  • Location:Edinburgh

Double post ftw :D
 

 

After some testing, it does appear that the character ã cannot be displayed in the lyric lines of Rocksmith.

 

 

Time to move on :D



#15
Offline   cozy1

cozy1

    Master Peon

  • Developer

  • PipPipPipPipPipPipPip
  • 1,170 posts

How do we actually get ã into the game to test?

 

I tried manually adding them to the normal Vocals_XML but they were still filtered out in game.  On checking, Notepad++ does save the character, it must be filtered out when I add it to the toolkit (and the toolkit edits the XML file)?

 

It's also filtered out when I simply load up the toolkit save and re-generate.

 

 

 

If the game doesn't like it fair enough, I can move on.  I'm in the mood for testing this out just now though :D

Regenerate the vocals with EOF so that the ã is put back and then retest with the toolkit per my PM instructions.  Your should also be able to manual add the ã per PM instructions.

 

Here is PM instructions excerpted in case other readers want to test:

 

I added a new vocals editor feature to the toolkit awhile back.  Now it is the time to use/test it.

 

1.  Load up the toolkit Creator GUI like usual with your EOF project files. 

2.  Highlight the vocals arrangement.

3.  Select Edit.  Make sure the Arrangement Editor is showing that you have PART VOCALS_RS.xml loaded.

4.  In the Arrangement Editor, press the 'Type: Vocal' button.

5.  In the Edit Vocals popup select a vocal that you would like to add the special character 'ã'.  Go ahead and add the ã

6.  Click the 'Ok' button to save the edited lyric.

7.  Generate the CDLC and play in game.  Look and see if ã has been added.


Are you tired of AV False Positives???  Now accepting donations on my website (Click Here)  Your donation will be used towards buying a code signing certificate.   CGT is now compatible and safe to use with Rocksmith® 2014 Remastered ... 

 

Latest Build of Custom Game Toolkit (CGT) w/ Game Save Gigbox       Latest Build of Rocksmith Toolkit       Latest Build of Customs Forge Song Manager (CFSM)

 

All bug reports and help requests please include your: OS, CPU, AV, .NET Framework versions along with a description of the issue (include screenshots of error if possible).  It should go without having to say ... make sure you are using the latest build before submitting bug reports or asking for help.

 

*  Remember to use your magic words (please and thank you) if you would like a response.  Don't use phrases like 'thanks anyhow' as it is demeaning.


#16
Offline   PC Plum

PC Plum

    Guitar God

  • Member
  • PipPipPipPipPipPipPipPip
  • 2,012 posts
  • Location:Edinburgh

I followed your instructions and made sure that the character I wanted to check (ã) was in the Vocals_XML.  It does not appear in the lyric lines of Rocksmith, however.

 

 

I must say thanks though, that's another great Toolkit feature that I have only found out about :clap:



#17
Offline   raynebc

raynebc

    Guitar God

  • Rocksmith Custom Developer

  • PipPipPipPipPipPipPipPip
  • 2,885 posts

If it's desired, I could try to have EOF substitute non-accented vowels for the unsupported characters where applicable (ie. write "a" instead of "ã") in the exported lyrics.



#18
Offline   PC Plum

PC Plum

    Guitar God

  • Member
  • PipPipPipPipPipPipPipPip
  • 2,012 posts
  • Location:Edinburgh

When I was testing it was apparent to me that I would likely never use non supported characters again, so it's not something I would personally desire.

 

I was just a little confused why ã would appear in the artist name and not the lyrics.  cozy1 has since explained it to me :D



#19
Offline   raynebc

raynebc

    Guitar God

  • Rocksmith Custom Developer

  • PipPipPipPipPipPipPipPip
  • 2,885 posts

Is that the only common accented character that isn't supported in Rocksmith lyrics?



#20
Offline   PC Plum

PC Plum

    Guitar God

  • Member
  • PipPipPipPipPipPipPipPip
  • 2,012 posts
  • Location:Edinburgh

I am presuming that Chilipouni's list is pretty definitive.  I am not versed enough in languages other than English to know what accented characters would be considered common.

 

iminashi also made a comprehensive list of allowable characters (link supplied by cozy1 above)

 

http://customsforge....374#entry250374

 

 

I see you replied to that post, but the technical language is lost on me I'm afraid.