repl: surrogate pair upsets repl / terminal

$ out/Release/iojs
> Buffer([0,0xD8,0,0xDC]).toString('utf16le')
'
# screen freezes

The REPL still accepts input but no longer echoes it. Typing in process.exit() cleanly shuts down the process but the terminal isn't reset, leaving it in an unusable state until you run reset(1).

Unable to replicate the behavior in io.js 1.0.4 on macosx 10.10.

Sorry, I should have mentioned that this is with v1.x HEAD and that I can reproduce it with the stock FC21 terminal emulator and with the OS X 10.8 terminal emulator when ssh'ing into an FC21 box but not when run locally - and only when TERM=screen, TERM=xterm works.

Ah... Just for completeness... did you double check the locale on the FC21 box?

It's en_US.UTF-8. Starting screen with e.g. LC_ALL=C doesn't make a difference. I'm not sure if screen is locale-aware, though.

Seems to work fine on mine...?

(I can't seem to copy that character to my clipboard though..)

but not when run locally - and only when TERM=screen

Oh, missed that.

Update to 4.0.0

@mathiasbynens

@mathiasbynens have you seen this?

While part of this is definatly in the V8 engine, I think probably the output is an issue from Node.
(Windows platform)

// heiroglyph.js
// actual character is 0x13063
// from http://unicode.org/charts/PDF/U13000.pdf
var character = "\ud80c\ud863"

var json = JSON.stringify( { msg : character });
console.log( json );
console.log( character );

console.log( "\ud80c\ud863".codePointAt(0).toString(16) );
console.log( "\ud80c\ud863".codePointAt(1).toString(16) );
console.log( "\u13302".charCodeAt(0).toString(16) );

node heiroglyph.js > output.txt

the unicode characters '\ud80c' is converted to utf8 '\xef\xf\bd'
and '\ud863' is converted to '\xef\xbf\xbd' which is entirely wrong...

output (loaded in console ascii editor)

{"msg":"ï¿½ï¿½"}
ï¿½ï¿½
d80c
d863
1306

or this... loaded in notepad++ with unicode which looks closer to correct...

{"msg":"��"}
��
d80c
d863
1306

expected output would be

{"msg":"𓁓"}
𓁓
d80c
d863
1306

or in utf8 bytes '\xf8\x9d\x81\x93'

(expected, given that charCodeAt doesn't function 'correctly' but that's not Node's fault)

@d3x0r

@d3x0r I don't think that's the bug - or a bug. \ud80c\ud863 is not a valid surrogate pair, the second half should be >= \udc00 (e.g. \ud80c\udc63 would be valid.)

\ud80c\ud863 is basically two invalid code points. Those get replaced with the replacement character, \ufffd, which is 0xEF 0xBF 0xBD when encoded as UTF-8.

Yes; I was reflecting on it and realized that. (and while re-reading found that the second byte is supposed to be DC.

var character = "\ud80c\udc63"

var json = JSON.stringify( { msg : character });
console.log( json );
console.log( character );
console.log( character.codePointAt(0).toString(16) );

output is now

{"msg":"ЁУБг"}
ЁУБг
13063

( F0 93 81 a3 ) which is right.... even if things aren't reading it correctly now :)

@bnoordhuis

I just attempted this in a screen session and everything worked. @bnoordhuis can you confirm you are still seeing this?

Yes, still happens. I don't mind closing this, though. I have looked at this issue a few times and I'm not sure if there is anything reasonable node.js can do here; it's quite possibly just an unfortunate interaction between ssh, screen and node.

@bnoordhuis

@bnoordhuis ... is it possible to put together a known-issue test for this?

Not sure that's possible... Let's just close this.

Update to 4.0.0

Navigation Menu

Repository navigation

repl: surrogate pair upsets repl / terminal #690

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

repl: surrogate pair upsets repl / terminal #690

Description

Activity

jasnell commented on Feb 2, 2015

bnoordhuis commented on Feb 2, 2015

jasnell commented on Feb 2, 2015

bnoordhuis commented on Feb 2, 2015

Fishrock123 commented on Apr 4, 2015

Fishrock123 commented on Apr 4, 2015

jdalton commented on Dec 10, 2015

d3x0r commented on Jan 30, 2016

bnoordhuis commented on Jan 30, 2016

d3x0r commented on Jan 30, 2016

MylesBorins commented on Feb 11, 2016

bnoordhuis commented on Feb 12, 2016

jasnell commented on Apr 2, 2016

Fishrock123 commented on Jul 19, 2016

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions