Well I got around to making a song with some non English characters.  Kaijin kindly obliged by producing a set of synced Portugese lyrics for a little acoustic ditty from Youtube.

 

I am not sure if the problem is EoF or the Toolkit, and I decided to post here.

 

For reference I am using EoF v1.8RC12 (4-17-2017) and Toolkit 2.8.2.0.  Also, I have Abridged XMLs turned on.

 

And it is a software problem because the game has no problem displaying all characters in song/singer names

 

 

https://drive.google.com/file/d/0B2oZs0ta32KyWGIzRWpwNlNuN1U/view?usp=sharing

 

https://drive.google.com/file/d/0B2oZs0ta32KyNTZSanhCZWVEUDg/view?usp=sharing

 

 

It just has issues displaying them in the lyric lines.

 

 

I wanted to post here because there is a difference I can reproduce at EoF level.  When I use the PART VOCALS_RS2_EXT to build my .psarc I get quite different results from using the normal PART VOCALS_RS2.  Both files were spit out by EoF during the same save.

 

Using the _EXT I get

 

https://drive.google.com/file/d/0B2oZs0ta32KyYXpWLURSZk5ObUU/view?usp=sharing

 

Broken line breaks and all of the accented characters are removed.

 

 

When I use the "normal" Vocals XML I get

 

https://drive.google.com/file/d/0B2oZs0ta32KyUUxaTmFlVVZJa28/view?usp=sharing

 

https://drive.google.com/file/d/0B2oZs0ta32KyTDdLeUF1SHdhZFE/view?usp=sharing

 

Working line breaks and only the "a" with the squigly line removed (sorry I don't know what its called).

 

Clearly the game can display the "a" as its in the name of the artist.

 

 

 

A question I have is, even though I am using the latest EoF it warns me when I save that I have non-ascii charaters.  Are non-ascii characters different from extended ascii characters?

 

Any ideas/thoughts on the evidence above?

 

I can make a pretty working version for release just now, it's just going to miss out the squigly "a" from the lyrics.  Everything else works technically correct; the song plays, finishes and scores as expected.

 

I suppose I could upload the files for inspection

 

https://drive.google.com/file/d/0B2oZs0ta32KySWYwYm81TUU5VU0/view?usp=sharing

 

 

 

 

Thanks :D

@PC Plum What specific github version of toolkit are you using?  Need more than just 2.8.2.0. 

I downloaded it today, so it looks like

 

Latest test build - 2.8.2.0 (rev 3ff0b529)

 

 

 

 

Thanks :D

Looking at the two vocals xml ... the 'PART VOCALS_RS2.XML' contains the extended characters which seem to show up in the game based on your screen capture.

 

The ''PART VOCALS_RS2_EXT.XML' do not contain extended characters.

 

So it appears the two different vocals are working as expected.    Your question was 'Are non-ascii characters different from extended ascii characters?'  Have a look here which should explain it.  iminashi explains it here.  raynebc explains it here.  I use the terms  'extended characters' and 'special characters' interchangeably  ...  for which I could be ridiculed.  ;)

When I re-save the project, EOF writes a UTF lyric XML file with the ã character, so my guess it's something needed on the toolkit's side of things. Of course, let me know if you find otherwise.

the 'PART VOCALS_RS2.XML' contains the extended characters which seem to show up in the game

 

"a" squigly does not show up in game, so its not in the Vocals_XML?

 

I haven't checked, there's no point you guys know a whole lot more anyway :D

 

 

 

Whats a UTF file, is it something normal EoF users can get?

 

 

 

I'll have a read at your links soon cozy, got the tabs to finish up first :D

Whats a UTF file, is it something normal EoF users can get?

 

By using the PART VOCALS_RS2.XML file you are already using a UTF file.  :)

The vocals XML file is encoded in UTF-8 (Unicode) whereas the _EXT file is encoded in Windows-1252. You don't need to worry about that. Just know that the _EXT file does not work properly with the Toolkit.

 

ã is not in the default lyrics font so it will not work. Since it's written into the UTF lyrics file, it would seem that EOF's character filtering is not working correctly.

 

For the _EXT file the filtering does work properly; ã is filtered out.

 

The + characters for line breaks are not being written into the _EXT file.

I've fixed the filtering.  The understanding seems to be that even though Rocksmith can display certain characters like ã for artist/song name, it's not supported for lyrics.  If that's not the case, let me know.

How do we actually get ã into the game to test?

 

I tried manually adding them to the normal Vocals_XML but they were still filtered out in game.  On checking, Notepad++ does save the character, it must be filtered out when I add it to the toolkit (and the toolkit edits the XML file)?

 

It's also filtered out when I simply load up the toolkit save and re-generate.

 

 

 

If the game doesn't like it fair enough, I can move on.  I'm in the mood for testing this out just now though :D

@@Chlipouni had tested all of the extended ASCII characters to find which ones would work in-game:

http://customsforge.com/topic/1529-latest-eof-releases-4-17-2017/page-103?p=250394&do=findComment&comment=250394

I see that I am a bit behind and that this has already been tested quite extensively.  I just presumed when I read some of the updates coming to EoF that "extended ascii" would mean that we could have any lyrics we wanted really, and ã seems to be handled no problem by web browers and every other program except the lyric lines of Rocksmith :D

 

I have asked Kaijin for some input from a language perspective.  Maybe a working character can be used as a substitute for ã while making linguistic sense.

 

 

 

Thanks all :D

 

Whats a UTF file, is it something normal EoF users can get?

 

By using the PART VOCALS_RS2.XML file you are already using a UTF file.  :)

The vocals XML file is encoded in UTF-8 (Unicode) whereas the _EXT file is encoded in Windows-1252. You don't need to worry about that. Just know that the _EXT file does not work properly with the Toolkit.

 

ã is not in the default lyrics font so it will not work. Since it's written into the UTF lyrics file, it would seem that EOF's character filtering is not working correctly.

 

For the _EXT file the filtering does work properly; ã is filtered out.

 

The + characters for line breaks are not being written into the _EXT file.

 

So @@raynebc seems EOF and toolkit are strictly correct because as @@iminashi has pointed out, ã is not an ODLC vocal character list.  We could add it to the vocals validation/filtering in EOF and toolkit and see if it works in the game.

 

EDIT:  I have added ã to the latest toolkit beta list of approved lyric characters for testing.

Thanks cozy1, I'll give it a go later :D
 

Speaking to Kaijin, if all else pails I can just use a normal English "a" as a substitue.  Apparently, Portugese people are well used to their language being abused :lol:

 

 

On a serious note though, it's clear from the screenies in my op that the _EXT is filtering out at least one working character.

 

https://drive.google.com/file/d/0B2oZs0ta32KyYXpWLURSZk5ObUU/view

 

"são só meus" becomes "so s meus", where the ó works fine using the normal Vocal_XML

 

https://drive.google.com/file/d/0B2oZs0ta32KyUUxaTmFlVVZJa28/view

 

Incidentally, ó is listed as working in Chillipouni's list too.

Double post ftw :D
 

 

After some testing, it does appear that the character ã cannot be displayed in the lyric lines of Rocksmith.

 

 

Time to move on :D

How do we actually get ã into the game to test?

 

I tried manually adding them to the normal Vocals_XML but they were still filtered out in game.  On checking, Notepad++ does save the character, it must be filtered out when I add it to the toolkit (and the toolkit edits the XML file)?

 

It's also filtered out when I simply load up the toolkit save and re-generate.

 

 

 

If the game doesn't like it fair enough, I can move on.  I'm in the mood for testing this out just now though :D

Regenerate the vocals with EOF so that the ã is put back and then retest with the toolkit per my PM instructions.  Your should also be able to manual add the ã per PM instructions.

 

Here is PM instructions excerpted in case other readers want to test:

 

I added a new vocals editor feature to the toolkit awhile back.  Now it is the time to use/test it.

 

1.  Load up the toolkit Creator GUI like usual with your EOF project files. 

2.  Highlight the vocals arrangement.

3.  Select Edit.  Make sure the Arrangement Editor is showing that you have PART VOCALS_RS.xml loaded.

4.  In the Arrangement Editor, press the 'Type: Vocal' button.

5.  In the Edit Vocals popup select a vocal that you would like to add the special character 'ã'.  Go ahead and add the ã

6.  Click the 'Ok' button to save the edited lyric.

7.  Generate the CDLC and play in game.  Look and see if ã has been added.

I followed your instructions and made sure that the character I wanted to check (ã) was in the Vocals_XML.  It does not appear in the lyric lines of Rocksmith, however.

 

 

I must say thanks though, that's another great Toolkit feature that I have only found out about :clap:

If it's desired, I could try to have EOF substitute non-accented vowels for the unsupported characters where applicable (ie. write "a" instead of "ã") in the exported lyrics.

When I was testing it was apparent to me that I would likely never use non supported characters again, so it's not something I would personally desire.

 

I was just a little confused why ã would appear in the artist name and not the lyrics.  cozy1 has since explained it to me :D

Is that the only common accented character that isn't supported in Rocksmith lyrics?

I am presuming that Chilipouni's list is pretty definitive.  I am not versed enough in languages other than English to know what accented characters would be considered common.

 

iminashi also made a comprehensive list of allowable characters (link supplied by cozy1 above)

 

http://customsforge.com/topic/1529-latest-eof-releases-4-17-2017/page-103?p=250374&do=findComment&comment=250374

 

 

I see you replied to that post, but the technical language is lost on me I'm afraid.

Latin characters with accents are ones that I could substitute easily.  Looking through the list, this looks like all of the ones the game doesn't support:

 

Z with caron:

Ž (character 142)

(the game already supports the lowercase version of this character)

 

Y with umlaut

Ÿ (character 159)

ÿ (character 255)

 

A with tilde:

à (character 195)

ã (character 227)

 

I with acute:

Í (character 205)

(the game already supports the lowercase version of this character)

 

O with tilde:

Õ (character 213)

õ (character 245)

 

Y with acute:

Ý (character 221)

ý (character 253)

 

I expect most authors would rather these affected characters to be automatically replaced so that they lyric is readable instead of having the characters omitted.  Should EOF display a warning notice if it ends up substituting any characters during RS export?

 

If the user didn't enable extended ASCII export, I could likewise have the other accented latin characters (which the game does support) substituted for non-accented versions to keep the lyrics readable.

@iminashi  If raynebc hard codes these character substitutions, would this prevent users from being able to use custom lyric font DDS files with the toolkit? 

 

Seems like users may want the option in EOF to not use any character substitutions in case the user wants to create and use a custom lyric font DDS file.

Does anybody currently do that?  If nothing else, the substitutions would probably be a good idea if the author hasn't enabled the preference to export extended ASCII characters at all.

 

I'm wondering if there's a benefit to having EOF export the Win-1252 encoded lyric file if some of the characters are omitted.  Perhaps that export should include all characters, so that if it ends up being imported later on, it actually has all the content.

If the extended ascii export option is switched on then the output should be raw with no changes, and if the option is off then sub all non working characters for non accented versions.

 

That would be peachy I suppose :D

@iminashi  If raynebc hard codes these character substitutions, would this prevent users from being able to use custom lyric font DDS files with the toolkit? 

 

Seems like users may want the option in EOF to not use any character substitutions in case the user wants to create and use a custom lyric font DDS file.

 

I don't think anybody uses custom fonts since there are no tools out that would ease the process.

 

Substituting the characters that can't be used is a good idea I think. Having a non-accented character is better than no character at all.

Similar to Plum's point.  How about all xml arrangements be UTF-8 (this is what game supports).    Have the 'PART VOCALS_RS2.xml' file use only the ODLC lyric character set (per iminashi's and Chlipoini's findings), and any non-ODLC lyric characters will be substituted per raynebc's  proposal above.   This file would be used by most all users with the toolkit to produce the CDLC.  

 

The 'PART VOCALS_RS2_EXT.xml' would contain any special lyric characters (not found in the ODLC lyric character set), including those for which substitution occurs, so that the user later could create a custom lyric font DDS file if they desire and use it.  (The toolkit currently supports using custom lyric font DDS files in the lyric editor.  I don't know of anyone currently using this advanced feature except for possibly Japanese kanji, similar to what is found in ODLC girlgirlchop_p.psarc.)

 

This option allows for using custom lyric fonts should the user ever want it, and provides for compliance with the ODLC lyric character set.