Jump to content

Non English Lyrics/Extended ASCII


PC Plum

Recommended Posts

Well I got around to making a song with some non English characters.  Kaijin kindly obliged by producing a set of synced Portugese lyrics for a little acoustic ditty from Youtube.

 

I am not sure if the problem is EoF or the Toolkit, and I decided to post here.

 

For reference I am using EoF v1.8RC12 (4-17-2017) and Toolkit 2.8.2.0.  Also, I have Abridged XMLs turned on.

 

And it is a software problem because the game has no problem displaying all characters in song/singer names

 

 

https://drive.google.com/file/d/0B2oZs0ta32KyWGIzRWpwNlNuN1U/view?usp=sharing

 

https://drive.google.com/file/d/0B2oZs0ta32KyNTZSanhCZWVEUDg/view?usp=sharing

 

 

It just has issues displaying them in the lyric lines.

 

 

I wanted to post here because there is a difference I can reproduce at EoF level.  When I use the PART VOCALS_RS2_EXT to build my .psarc I get quite different results from using the normal PART VOCALS_RS2.  Both files were spit out by EoF during the same save.

 

Using the _EXT I get

 

https://drive.google.com/file/d/0B2oZs0ta32KyYXpWLURSZk5ObUU/view?usp=sharing

 

Broken line breaks and all of the accented characters are removed.

 

 

When I use the "normal" Vocals XML I get

 

https://drive.google.com/file/d/0B2oZs0ta32KyUUxaTmFlVVZJa28/view?usp=sharing

 

https://drive.google.com/file/d/0B2oZs0ta32KyTDdLeUF1SHdhZFE/view?usp=sharing

 

Working line breaks and only the "a" with the squigly line removed (sorry I don't know what its called).

 

Clearly the game can display the "a" as its in the name of the artist.

 

 

 

A question I have is, even though I am using the latest EoF it warns me when I save that I have non-ascii charaters.  Are non-ascii characters different from extended ascii characters?

 

Any ideas/thoughts on the evidence above?

 

I can make a pretty working version for release just now, it's just going to miss out the squigly "a" from the lyrics.  Everything else works technically correct; the song plays, finishes and scores as expected.

 

I suppose I could upload the files for inspection

 

https://drive.google.com/file/d/0B2oZs0ta32KySWYwYm81TUU5VU0/view?usp=sharing

 

 

 

 

Thanks :D

Link to comment
Share on other sites

  • Developer

@PC Plum What specific github version of toolkit are you using?  Need more than just 2.8.2.0. 

Are you tired of AV False Positives???  Now accepting donations on my website (Click Here)  Your donation will be used towards buying a code signing certificate.   CGT is now compatible and safe to use with Rocksmith® 2014 Remastered ... 

 

Latest Build of Custom Game Toolkit (CGT) w/ Game Save Gigbox       Latest Build of Rocksmith Toolkit       Latest Build of Customs Forge Song Manager (CFSM)

 

All bug reports and help requests please include your: OS, CPU, AV, .NET Framework versions along with a description of the issue (include screenshots of error if possible).  It should go without having to say ... make sure you are using the latest build before submitting bug reports or asking for help.

 

*  Remember to use your magic words (please and thank you) if you would like a response.  Don't use phrases like 'thanks anyhow' as it is demeaning.

Link to comment
Share on other sites

  • Developer

Looking at the two vocals xml ... the 'PART VOCALS_RS2.XML' contains the extended characters which seem to show up in the game based on your screen capture.

 

The ''PART VOCALS_RS2_EXT.XML' do not contain extended characters.

 

So it appears the two different vocals are working as expected.    Your question was 'Are non-ascii characters different from extended ascii characters?'  Have a look here which should explain it.  iminashi explains it here.  raynebc explains it here.  I use the terms  'extended characters' and 'special characters' interchangeably  ...  for which I could be ridiculed.  ;)

Are you tired of AV False Positives???  Now accepting donations on my website (Click Here)  Your donation will be used towards buying a code signing certificate.   CGT is now compatible and safe to use with Rocksmith® 2014 Remastered ... 

 

Latest Build of Custom Game Toolkit (CGT) w/ Game Save Gigbox       Latest Build of Rocksmith Toolkit       Latest Build of Customs Forge Song Manager (CFSM)

 

All bug reports and help requests please include your: OS, CPU, AV, .NET Framework versions along with a description of the issue (include screenshots of error if possible).  It should go without having to say ... make sure you are using the latest build before submitting bug reports or asking for help.

 

*  Remember to use your magic words (please and thank you) if you would like a response.  Don't use phrases like 'thanks anyhow' as it is demeaning.

Link to comment
Share on other sites

the 'PART VOCALS_RS2.XML' contains the extended characters which seem to show up in the game

 

"a" squigly does not show up in game, so its not in the Vocals_XML?

 

I haven't checked, there's no point you guys know a whole lot more anyway :D

 

 

 

Whats a UTF file, is it something normal EoF users can get?

 

 

 

I'll have a read at your links soon cozy, got the tabs to finish up first :D

Link to comment
Share on other sites

  • Developer

Whats a UTF file, is it something normal EoF users can get?

 
By using the PART VOCALS_RS2.XML file you are already using a UTF file.  :)
The vocals XML file is encoded in UTF-8 (Unicode) whereas the _EXT file is encoded in Windows-1252. You don't need to worry about that. Just know that the _EXT file does not work properly with the Toolkit.
 
ã is not in the default lyrics font so it will not work. Since it's written into the UTF lyrics file, it would seem that EOF's character filtering is not working correctly.
 
For the _EXT file the filtering does work properly; ã is filtered out.
 
The + characters for line breaks are not being written into the _EXT file.
  • Like 1
Link to comment
Share on other sites

How do we actually get ã into the game to test?

 

I tried manually adding them to the normal Vocals_XML but they were still filtered out in game.  On checking, Notepad++ does save the character, it must be filtered out when I add it to the toolkit (and the toolkit edits the XML file)?

 

It's also filtered out when I simply load up the toolkit save and re-generate.

 

 

 

If the game doesn't like it fair enough, I can move on.  I'm in the mood for testing this out just now though :D

Link to comment
Share on other sites

I see that I am a bit behind and that this has already been tested quite extensively.  I just presumed when I read some of the updates coming to EoF that "extended ascii" would mean that we could have any lyrics we wanted really, and ã seems to be handled no problem by web browers and every other program except the lyric lines of Rocksmith :D

 

I have asked Kaijin for some input from a language perspective.  Maybe a working character can be used as a substitute for ã while making linguistic sense.

 

 

 

Thanks all :D

Link to comment
Share on other sites

  • Developer

 

Whats a UTF file, is it something normal EoF users can get?

 
By using the PART VOCALS_RS2.XML file you are already using a UTF file.  :)
The vocals XML file is encoded in UTF-8 (Unicode) whereas the _EXT file is encoded in Windows-1252. You don't need to worry about that. Just know that the _EXT file does not work properly with the Toolkit.
 
ã is not in the default lyrics font so it will not work. Since it's written into the UTF lyrics file, it would seem that EOF's character filtering is not working correctly.
 
For the _EXT file the filtering does work properly; ã is filtered out.
 
The + characters for line breaks are not being written into the _EXT file.

 

So @@raynebc seems EOF and toolkit are strictly correct because as @@iminashi has pointed out, ã is not an ODLC vocal character list.  We could add it to the vocals validation/filtering in EOF and toolkit and see if it works in the game.

 

EDIT:  I have added ã to the latest toolkit beta list of approved lyric characters for testing.

  • Like 1

Are you tired of AV False Positives???  Now accepting donations on my website (Click Here)  Your donation will be used towards buying a code signing certificate.   CGT is now compatible and safe to use with Rocksmith® 2014 Remastered ... 

 

Latest Build of Custom Game Toolkit (CGT) w/ Game Save Gigbox       Latest Build of Rocksmith Toolkit       Latest Build of Customs Forge Song Manager (CFSM)

 

All bug reports and help requests please include your: OS, CPU, AV, .NET Framework versions along with a description of the issue (include screenshots of error if possible).  It should go without having to say ... make sure you are using the latest build before submitting bug reports or asking for help.

 

*  Remember to use your magic words (please and thank you) if you would like a response.  Don't use phrases like 'thanks anyhow' as it is demeaning.

Link to comment
Share on other sites

Thanks cozy1, I'll give it a go later :D
 

Speaking to Kaijin, if all else pails I can just use a normal English "a" as a substitue.  Apparently, Portugese people are well used to their language being abused :lol:

 

 

On a serious note though, it's clear from the screenies in my op that the _EXT is filtering out at least one working character.

 

https://drive.google.com/file/d/0B2oZs0ta32KyYXpWLURSZk5ObUU/view

 

"são só meus" becomes "so s meus", where the ó works fine using the normal Vocal_XML

 

https://drive.google.com/file/d/0B2oZs0ta32KyUUxaTmFlVVZJa28/view

 

Incidentally, ó is listed as working in Chillipouni's list too.

Link to comment
Share on other sites

  • Developer

How do we actually get ã into the game to test?

 

I tried manually adding them to the normal Vocals_XML but they were still filtered out in game.  On checking, Notepad++ does save the character, it must be filtered out when I add it to the toolkit (and the toolkit edits the XML file)?

 

It's also filtered out when I simply load up the toolkit save and re-generate.

 

 

 

If the game doesn't like it fair enough, I can move on.  I'm in the mood for testing this out just now though :D

Regenerate the vocals with EOF so that the ã is put back and then retest with the toolkit per my PM instructions.  Your should also be able to manual add the ã per PM instructions.

 

Here is PM instructions excerpted in case other readers want to test:

 

I added a new vocals editor feature to the toolkit awhile back.  Now it is the time to use/test it.

 

1.  Load up the toolkit Creator GUI like usual with your EOF project files. 

2.  Highlight the vocals arrangement.

3.  Select Edit.  Make sure the Arrangement Editor is showing that you have PART VOCALS_RS.xml loaded.

4.  In the Arrangement Editor, press the 'Type: Vocal' button.

5.  In the Edit Vocals popup select a vocal that you would like to add the special character 'ã'.  Go ahead and add the ã

6.  Click the 'Ok' button to save the edited lyric.

7.  Generate the CDLC and play in game.  Look and see if ã has been added.

Are you tired of AV False Positives???  Now accepting donations on my website (Click Here)  Your donation will be used towards buying a code signing certificate.   CGT is now compatible and safe to use with Rocksmith® 2014 Remastered ... 

 

Latest Build of Custom Game Toolkit (CGT) w/ Game Save Gigbox       Latest Build of Rocksmith Toolkit       Latest Build of Customs Forge Song Manager (CFSM)

 

All bug reports and help requests please include your: OS, CPU, AV, .NET Framework versions along with a description of the issue (include screenshots of error if possible).  It should go without having to say ... make sure you are using the latest build before submitting bug reports or asking for help.

 

*  Remember to use your magic words (please and thank you) if you would like a response.  Don't use phrases like 'thanks anyhow' as it is demeaning.

Link to comment
Share on other sites

I followed your instructions and made sure that the character I wanted to check (ã) was in the Vocals_XML.  It does not appear in the lyric lines of Rocksmith, however.

 

 

I must say thanks though, that's another great Toolkit feature that I have only found out about :clap:

Link to comment
Share on other sites

When I was testing it was apparent to me that I would likely never use non supported characters again, so it's not something I would personally desire.

 

I was just a little confused why ã would appear in the artist name and not the lyrics.  cozy1 has since explained it to me :D

Link to comment
Share on other sites

I am presuming that Chilipouni's list is pretty definitive.  I am not versed enough in languages other than English to know what accented characters would be considered common.

 

iminashi also made a comprehensive list of allowable characters (link supplied by cozy1 above)

 

http://customsforge.com/topic/1529-latest-eof-releases-4-17-2017/page-103?p=250374&do=findComment&comment=250374

 

 

I see you replied to that post, but the technical language is lost on me I'm afraid.

Link to comment
Share on other sites

Latin characters with accents are ones that I could substitute easily.  Looking through the list, this looks like all of the ones the game doesn't support:

 

Z with caron:

Ž (character 142)

(the game already supports the lowercase version of this character)

 

Y with umlaut

Ÿ (character 159)

ÿ (character 255)

 

A with tilde:

à (character 195)

ã (character 227)

 

I with acute:

Í (character 205)

(the game already supports the lowercase version of this character)

 

O with tilde:

Õ (character 213)

õ (character 245)

 

Y with acute:

Ý (character 221)

ý (character 253)

 

I expect most authors would rather these affected characters to be automatically replaced so that they lyric is readable instead of having the characters omitted.  Should EOF display a warning notice if it ends up substituting any characters during RS export?

 

If the user didn't enable extended ASCII export, I could likewise have the other accented latin characters (which the game does support) substituted for non-accented versions to keep the lyrics readable.

Link to comment
Share on other sites

  • Developer

@iminashi  If raynebc hard codes these character substitutions, would this prevent users from being able to use custom lyric font DDS files with the toolkit? 

 

Seems like users may want the option in EOF to not use any character substitutions in case the user wants to create and use a custom lyric font DDS file.

Are you tired of AV False Positives???  Now accepting donations on my website (Click Here)  Your donation will be used towards buying a code signing certificate.   CGT is now compatible and safe to use with Rocksmith® 2014 Remastered ... 

 

Latest Build of Custom Game Toolkit (CGT) w/ Game Save Gigbox       Latest Build of Rocksmith Toolkit       Latest Build of Customs Forge Song Manager (CFSM)

 

All bug reports and help requests please include your: OS, CPU, AV, .NET Framework versions along with a description of the issue (include screenshots of error if possible).  It should go without having to say ... make sure you are using the latest build before submitting bug reports or asking for help.

 

*  Remember to use your magic words (please and thank you) if you would like a response.  Don't use phrases like 'thanks anyhow' as it is demeaning.

Link to comment
Share on other sites

Does anybody currently do that?  If nothing else, the substitutions would probably be a good idea if the author hasn't enabled the preference to export extended ASCII characters at all.

 

I'm wondering if there's a benefit to having EOF export the Win-1252 encoded lyric file if some of the characters are omitted.  Perhaps that export should include all characters, so that if it ends up being imported later on, it actually has all the content.

Link to comment
Share on other sites

  • Developer

@iminashi  If raynebc hard codes these character substitutions, would this prevent users from being able to use custom lyric font DDS files with the toolkit? 

 

Seems like users may want the option in EOF to not use any character substitutions in case the user wants to create and use a custom lyric font DDS file.

 

I don't think anybody uses custom fonts since there are no tools out that would ease the process.

 

Substituting the characters that can't be used is a good idea I think. Having a non-accented character is better than no character at all.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Unfortunately, your content contains terms that we do not allow. Please edit your content to remove the highlighted words below.
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...

Important Information

By using this site, you agree to our Guidelines. We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue. - Privacy Policy