php-src/ext/mbstring/tests/uhc_encoding.phpt

--TEST--
Exhaustive test of verification and conversion of CP949 (UHC) text
--EXTENSIONS--
mbstring
--SKIPIF--
<?php
if (getenv("SKIP_SLOW_TESTS")) die("skip slow test");
?>
--FILE--
<?php
include('encoding_tests.inc');
testEncodingFromUTF16ConversionTable(__DIR__ . '/data/CP949.txt', 'UHC');

// Test "long" illegal character markers
mb_substitute_character("long");
convertInvalidString("\x80", "%", "UHC", "UTF-8");
convertInvalidString("\xA7\xF0", "%", "UHC", "UTF-8");
?>
--EXPECT--
Tested UHC -> UTF-16BE
Tested UTF-16BE -> UHC
Strict conversion of UHC text to Unicode Previously, mbstring would accept a lot of things which were not valid UHC text. No more. - Don't allow single-byte control characters to appear where the 2nd byte of a multi-byte character should be. - Validate that the 2nd byte of a multi-byte character is in the expected range. - Treat it as an error if a multi-byte character is truncated. Also add a test suite to confirm that UHC conversion (both to and from Unicode) works according to spec. 2021-05-08 20:49:22 +00:00			`--TEST--`
			`Exhaustive test of verification and conversion of CP949 (UHC) text`
			`--EXTENSIONS--`
			`mbstring`
			`--SKIPIF--`
			`<?php`
			`if (getenv("SKIP_SLOW_TESTS")) die("skip slow test");`
			`?>`
			`--FILE--`
			`<?php`
			`include('encoding_tests.inc');`
			`testEncodingFromUTF16ConversionTable(__DIR__ . '/data/CP949.txt', 'UHC');`
Test behavior of 'long' illegal character markers After mb_substitute_character("long"), mbstring will respond to erroneous input by inserting 'long' error markers into the output. Depending on the situation, these error markers will either look like BAD+XXXX (for general bad input), U+XXXX (when the input is OK, but it converts to Unicode codepoints which cannot be represented in the output encoding), or an encoding-specific marker like JISX+XXXX or W932+XXXX. We have almost no tests for this feature. Add a bunch of tests to ensure that all our legacy encoding handlers work in a reasonable way when 'long' error markers are enabled. 2021-07-27 11:21:48 +00:00
			`// Test "long" illegal character markers`
			`mb_substitute_character("long");`
mbstring no longer provides 'long' substitutions for erroneous input bytes Previously, mbstring had a special mode whereby it would convert erroneous input byte sequences to output like "BAD+XXXX", where "XXXX" would be the erroneous bytes expressed in hexadecimal. This mode could be enabled by calling `mb_substitute_character("long")`. However, accurately reproducing input byte sequences from the cached state of a conversion filter is often tricky, and this significantly complicates the implementation. Further, the means used for passing the erroneous bytes through to where the "BAD+XXXX" text is generated only allows for up to 3 bytes to be passed, meaning that some erroneous byte sequences are truncated anyways. More to the point, a search of publically available PHP code indicates that nobody is really using this feature anyways. Incidentally, this feature also provided error output like "JIS+XXXX" if the input 'should have' represented a JISX 0208 codepoint, but it decodes to a codepoint which does not exist in the JISX 0208 charset. Similarly, specific error output was provided for non-existent JISX 0212 codepoints, and likewise for JISX 0213, CP932, and a few other charsets. All of that is now consigned to the flames. However, "long" error markers also include a somewhat more useful "U+XXXX" marker for Unicode codepoints which were successfully decoded from the input text, but cannot be represented in the output encoding. Those are still supported. With this change, there is no need to use a variety of special values in the high bits of a wchar to represent different types of error values. We can (and will) just use a single error value. This will be equal to -1. One complicating factor: Text conversion functions return an integer to indicate whether the conversion operation should be immediately aborted, and the magic 'abort' marker is -1. Also, almost all of these functions would return the received byte/codepoint to indicate success. That doesn't work with the new error value; if an input filter detects an error and passes -1 to the output filter, and the output filter returns it back, that would be taken to mean 'abort'. Therefore, amend all these functions to return 0 for success. 2021-08-30 17:35:05 +00:00			`convertInvalidString("\x80", "%", "UHC", "UTF-8");`
			`convertInvalidString("\xA7\xF0", "%", "UHC", "UTF-8");`
Strict conversion of UHC text to Unicode Previously, mbstring would accept a lot of things which were not valid UHC text. No more. - Don't allow single-byte control characters to appear where the 2nd byte of a multi-byte character should be. - Validate that the 2nd byte of a multi-byte character is in the expected range. - Treat it as an error if a multi-byte character is truncated. Also add a test suite to confirm that UHC conversion (both to and from Unicode) works according to spec. 2021-05-08 20:49:22 +00:00			`?>`
			`--EXPECT--`
			`Tested UHC -> UTF-16BE`
			`Tested UTF-16BE -> UHC`