php-src/ext/mbstring/tests/casemapping.phpt

122 lines
3.0 KiB
Plaintext
Raw Normal View History

Implement full case mapping Implement full case mapping according to SpecialCasing.txt and also full case folding according to CaseFolding.txt (F). There are a number of caveats: * Only language-agnostic and unconditional full case mapping is implemented. The only language-agnostic conditional case mapping rule relates to Greek sigma in final position (Final_Sigma). Correctly handling this requires both arbitrary lookahead and lookbehind, which would require some larger changes to how the case mapping is implemented. This is a possible future extension. * The only language-specific handling that is implemented is for Turkish dotted/undotted Is, if the ISO-8859-9 encoding is used. This matches the previous behavior and makes sure that no codepoints not supported by the encoding are produced. A future extension would be to also handle the Turkish mappings specified by SpecialCasing.txt based on the mbfl internal language. * Full case folding is implemented, but case-insensitive mb_* operations continue to use simple case folding. The reason is that full case folding of the haystack string may change the position at which a match occurred. This would have to be mapped back into the position in the original string. * mb_convert_case() exposes both the full and the simple case mapping / folding, where full is the default. The constants are: * MB_CASE_LOWER (used by mb_strtolower) * MB_CASE_UPPER (used by mb_strtolower) * MB_CASE_TITLE * MB_CASE_FOLD * MB_CASE_LOWER_SIMPLE * MB_CASE_UPPER_SIMPLE * MB_CASE_TITLE_SIMPLE * MB_CASE_FOLD_SIMPLE (used by case-insensitive operations)
2017-07-27 20:48:00 +00:00
--TEST--
Unicode case mapping
2021-06-11 11:59:09 +00:00
--EXTENSIONS--
mbstring
Implement full case mapping Implement full case mapping according to SpecialCasing.txt and also full case folding according to CaseFolding.txt (F). There are a number of caveats: * Only language-agnostic and unconditional full case mapping is implemented. The only language-agnostic conditional case mapping rule relates to Greek sigma in final position (Final_Sigma). Correctly handling this requires both arbitrary lookahead and lookbehind, which would require some larger changes to how the case mapping is implemented. This is a possible future extension. * The only language-specific handling that is implemented is for Turkish dotted/undotted Is, if the ISO-8859-9 encoding is used. This matches the previous behavior and makes sure that no codepoints not supported by the encoding are produced. A future extension would be to also handle the Turkish mappings specified by SpecialCasing.txt based on the mbfl internal language. * Full case folding is implemented, but case-insensitive mb_* operations continue to use simple case folding. The reason is that full case folding of the haystack string may change the position at which a match occurred. This would have to be mapped back into the position in the original string. * mb_convert_case() exposes both the full and the simple case mapping / folding, where full is the default. The constants are: * MB_CASE_LOWER (used by mb_strtolower) * MB_CASE_UPPER (used by mb_strtolower) * MB_CASE_TITLE * MB_CASE_FOLD * MB_CASE_LOWER_SIMPLE * MB_CASE_UPPER_SIMPLE * MB_CASE_TITLE_SIMPLE * MB_CASE_FOLD_SIMPLE (used by case-insensitive operations)
2017-07-27 20:48:00 +00:00
--FILE--
<?php
function toCases($str) {
echo "String: $str\n";
echo "Lower: ", mb_convert_case($str, MB_CASE_LOWER), "\n";
echo "Lower Simple: ", mb_convert_case($str, MB_CASE_LOWER_SIMPLE), "\n";
echo "Upper: ", mb_convert_case($str, MB_CASE_UPPER), "\n";
echo "Upper Simple: ", mb_convert_case($str, MB_CASE_UPPER_SIMPLE), "\n";
echo "Title: ", mb_convert_case($str, MB_CASE_TITLE), "\n";
echo "Title Simple: ", mb_convert_case($str, MB_CASE_TITLE_SIMPLE), "\n";
echo "Fold: ", mb_convert_case($str, MB_CASE_FOLD), "\n";
echo "Fold Simple: ", mb_convert_case($str, MB_CASE_FOLD_SIMPLE), "\n";
echo "\n";
}
toCases("ß");
toCases("ff");
toCases("İ");
// Make sure that case-conversion in Turkish still works correctly.
// Using the language-agnostic Unicode case mappings would result in
Implement full case mapping Implement full case mapping according to SpecialCasing.txt and also full case folding according to CaseFolding.txt (F). There are a number of caveats: * Only language-agnostic and unconditional full case mapping is implemented. The only language-agnostic conditional case mapping rule relates to Greek sigma in final position (Final_Sigma). Correctly handling this requires both arbitrary lookahead and lookbehind, which would require some larger changes to how the case mapping is implemented. This is a possible future extension. * The only language-specific handling that is implemented is for Turkish dotted/undotted Is, if the ISO-8859-9 encoding is used. This matches the previous behavior and makes sure that no codepoints not supported by the encoding are produced. A future extension would be to also handle the Turkish mappings specified by SpecialCasing.txt based on the mbfl internal language. * Full case folding is implemented, but case-insensitive mb_* operations continue to use simple case folding. The reason is that full case folding of the haystack string may change the position at which a match occurred. This would have to be mapped back into the position in the original string. * mb_convert_case() exposes both the full and the simple case mapping / folding, where full is the default. The constants are: * MB_CASE_LOWER (used by mb_strtolower) * MB_CASE_UPPER (used by mb_strtolower) * MB_CASE_TITLE * MB_CASE_FOLD * MB_CASE_LOWER_SIMPLE * MB_CASE_UPPER_SIMPLE * MB_CASE_TITLE_SIMPLE * MB_CASE_FOLD_SIMPLE (used by case-insensitive operations)
2017-07-27 20:48:00 +00:00
// characters that are illegal under ISO-8859-9.
mb_internal_encoding('ISO-8859-9');
// Capital I with dot (U+0130)
$str = "\xdd";
echo bin2hex(mb_convert_case($str, MB_CASE_LOWER)), "\n";
echo bin2hex(mb_convert_case($str, MB_CASE_LOWER_SIMPLE)), "\n";
echo bin2hex(mb_convert_case($str, MB_CASE_FOLD)), "\n";
echo bin2hex(mb_convert_case($str, MB_CASE_FOLD_SIMPLE)), "\n";
echo "\n";
// Lower i without dot (U+0131)
$str = "\xfd";
echo bin2hex(mb_convert_case($str, MB_CASE_UPPER)), "\n";
echo bin2hex(mb_convert_case($str, MB_CASE_UPPER_SIMPLE)), "\n";
echo bin2hex(mb_convert_case($str, MB_CASE_FOLD)), "\n";
echo bin2hex(mb_convert_case($str, MB_CASE_FOLD_SIMPLE)), "\n";
echo "\n";
// Capital I without dot (U+0049)
$str = "\x49";
echo bin2hex(mb_convert_case($str, MB_CASE_LOWER)), "\n";
echo bin2hex(mb_convert_case($str, MB_CASE_LOWER_SIMPLE)), "\n";
echo bin2hex(mb_convert_case($str, MB_CASE_FOLD)), "\n";
echo bin2hex(mb_convert_case($str, MB_CASE_FOLD_SIMPLE)), "\n";
echo "\n";
// Lower i with dot (U+0069)
$str = "\x69";
echo bin2hex(mb_convert_case($str, MB_CASE_UPPER)), "\n";
echo bin2hex(mb_convert_case($str, MB_CASE_UPPER_SIMPLE)), "\n";
echo bin2hex(mb_convert_case($str, MB_CASE_FOLD)), "\n";
echo bin2hex(mb_convert_case($str, MB_CASE_FOLD_SIMPLE)), "\n";
Implement Unicode conditional casing rules for Greek letter sigma The capital Greek letter sigma (Σ) should be lowercased as σ except when it appears at the end of a word; in that case, it should be lowercased as the special form ς. This rule is included in the Unicode data file SpecialCasing.txt. The condition for applying the rule is called "Final_Sigma" and is defined in Unicode technical report 21. The rule is: • For the special casing form to apply, the capital letter sigma must be preceded by 0 or more "case-ignorable" characters, preceded by at least 1 "cased" character. • Further, capital sigma must NOT be followed by 0 or more case-ignorable characters and then at least 1 cased character. "Case-ignorable" characters include certain punctuation marks, like the apostrophe, as well as various accent marks. There are actually close to 500 different case-ignorable characters, including accent marks from Cyrillic, Hebrew, Armenian, Arabic, Syriac, Bengali, Gujarati, Telugu, Tibetan, and many other alphabets. This category also includes zero-width spaces, codepoints which indicate RTL/LTR text direction, certain musical symbols, etc. Since the rule involves scanning over "0 or more" of such case-ignorable characters, it may be necessary to scan arbitrarily far to the left and right of capital sigma to determine whether the special lowercase form should be used or not. However, since we are trying to be both memory-efficient and CPU-efficient, this implementation limits how far to the left we will scan. Generally, we scan up to 63 characters to the left looking for a "cased" character, but not more. When scanning to the right, we go up to the end of the string if necessary, even if it means scanning over thousands of characters. Anyways, it is almost impossible to imagine that natural text will include "words" with more than 63 successive apostrophes (for example) followed by a capital sigma. Closes GH-8096.
2023-01-07 18:27:59 +00:00
// Check handling of Greek letter capital sigma
echo mb_convert_case("ΚΑΛΗΣΠΕΡΑ ΣΑΣ", MB_CASE_TITLE, "UTF-8"), "\n";
echo mb_convert_case("ΚΑΛΗΣΠΕΡΑ ΣΑΣ", MB_CASE_TITLE_SIMPLE, "UTF-8"), "\n";
echo mb_convert_case("ΚΑΛΗΣΠΕΡΑ ΣΑΣ", MB_CASE_LOWER, "UTF-8"), "\n";
echo mb_convert_case("ΚΑΛΗΣΠΕΡΑ ΣΑΣ", MB_CASE_LOWER_SIMPLE, "UTF-8"), "\n";
Implement full case mapping Implement full case mapping according to SpecialCasing.txt and also full case folding according to CaseFolding.txt (F). There are a number of caveats: * Only language-agnostic and unconditional full case mapping is implemented. The only language-agnostic conditional case mapping rule relates to Greek sigma in final position (Final_Sigma). Correctly handling this requires both arbitrary lookahead and lookbehind, which would require some larger changes to how the case mapping is implemented. This is a possible future extension. * The only language-specific handling that is implemented is for Turkish dotted/undotted Is, if the ISO-8859-9 encoding is used. This matches the previous behavior and makes sure that no codepoints not supported by the encoding are produced. A future extension would be to also handle the Turkish mappings specified by SpecialCasing.txt based on the mbfl internal language. * Full case folding is implemented, but case-insensitive mb_* operations continue to use simple case folding. The reason is that full case folding of the haystack string may change the position at which a match occurred. This would have to be mapped back into the position in the original string. * mb_convert_case() exposes both the full and the simple case mapping / folding, where full is the default. The constants are: * MB_CASE_LOWER (used by mb_strtolower) * MB_CASE_UPPER (used by mb_strtolower) * MB_CASE_TITLE * MB_CASE_FOLD * MB_CASE_LOWER_SIMPLE * MB_CASE_UPPER_SIMPLE * MB_CASE_TITLE_SIMPLE * MB_CASE_FOLD_SIMPLE (used by case-insensitive operations)
2017-07-27 20:48:00 +00:00
?>
--EXPECT--
String: ß
Lower: ß
Lower Simple: ß
Upper: SS
Upper Simple: ß
Title: Ss
Title Simple: ß
Fold: ss
Fold Simple: ß
String: ff
Lower: ff
Lower Simple: ff
Upper: FF
Upper Simple: ff
Title: Ff
Title Simple: ff
Fold: ff
Fold Simple: ff
String: İ
Lower: i̇
Lower Simple: i
Upper: İ
Upper Simple: İ
Title: İ
Title Simple: İ
Fold: i̇
Fold Simple: İ
69
69
69
69
49
49
fd
fd
fd
fd
fd
fd
dd
dd
69
69
Καλησπερα Σας
Implement Unicode conditional casing rules for Greek letter sigma The capital Greek letter sigma (Σ) should be lowercased as σ except when it appears at the end of a word; in that case, it should be lowercased as the special form ς. This rule is included in the Unicode data file SpecialCasing.txt. The condition for applying the rule is called "Final_Sigma" and is defined in Unicode technical report 21. The rule is: • For the special casing form to apply, the capital letter sigma must be preceded by 0 or more "case-ignorable" characters, preceded by at least 1 "cased" character. • Further, capital sigma must NOT be followed by 0 or more case-ignorable characters and then at least 1 cased character. "Case-ignorable" characters include certain punctuation marks, like the apostrophe, as well as various accent marks. There are actually close to 500 different case-ignorable characters, including accent marks from Cyrillic, Hebrew, Armenian, Arabic, Syriac, Bengali, Gujarati, Telugu, Tibetan, and many other alphabets. This category also includes zero-width spaces, codepoints which indicate RTL/LTR text direction, certain musical symbols, etc. Since the rule involves scanning over "0 or more" of such case-ignorable characters, it may be necessary to scan arbitrarily far to the left and right of capital sigma to determine whether the special lowercase form should be used or not. However, since we are trying to be both memory-efficient and CPU-efficient, this implementation limits how far to the left we will scan. Generally, we scan up to 63 characters to the left looking for a "cased" character, but not more. When scanning to the right, we go up to the end of the string if necessary, even if it means scanning over thousands of characters. Anyways, it is almost impossible to imagine that natural text will include "words" with more than 63 successive apostrophes (for example) followed by a capital sigma. Closes GH-8096.
2023-01-07 18:27:59 +00:00
Καλησπερα Σασ
καλησπερα σας
καλησπερα σασ