* Remove ZEND_DVAL_TO_LVAL_CAST_OK
As far as I can see, this operation should always use the _slow method, and the results seem to be wrong when ZEND_DVAL_TO_LVAL_CAST_OK is enabled.
* update NEWS
This doesn't have an effect really, but humans and IDEs can struggle to see through the macro soup when they first interact with PHP's source code.
Moreover, this reduces some of the macro expansion hell when they appear in compiler warnings.
RAND_RANGE_BADSCALING() invokes undefined behavior when (max - min) >
ZEND_LONG_MAX, because the intermediate `double` might not fit into
`zend_long`.
Fix this by inlining a fixed version of the macro into Mt19937's range()
function. Fixing the macro itself cannot be done in the general case, because
the types of the inputs are not known. Instead of replacing one possibly broken
version with another possibly broken version, the macro is simply left as is
and should be removed in a future version.
The fix itself is simple: Instead of storing the "offset" in a `zend_long`, we
use a `zend_ulong` which is capable of storing the resulting double by
construction. With this fix the implementation of this broken scaling is
effectively identical to the implementation of php_random_range from a data
type perspective, making it easy to verify the correctness.
It was further empirically verified that the broken macro and the fix return
the same results for all possible values of `r` for several distinct pairs of
(min, max).
Fixes GH-9190
Fixes GH-9191
* Add Random\Random{Error,Exception} and Random\BrokenRandomEngineError
* Throw BrokenRandomEngineError
* Throw RandomException on seeding failure
* Throw RandomException when CSPRNG fails
* Remove unused include from ext/random/engine_combinedlcg.c
* Remove unused include from ext/random/engine_secure.c
* Remove unused include from ext/random/random.c
* [ci skip] Add ext/random Exception hierarchy to NEWS
* [ci skip] Add the change of Exception for random_(int|bytes) to UPGRADING
This removes the inconsistent and incorrectly working open basedir check
on pattern in glob. It means that an empty array will be returned even
if the whole pattern is outside the open basedir restriction.
* Fix PcgOneseq128XslRr64::__construct() definition in random.stub.php
The second parameter does not actually exist for a Oneseq PCG. It was removed
from the RFC before it went into voting.
* [ci skip] Add PcgOneseq128XslRr64 stub fix to NEWS
* Verify that the engine doesn't change in construct_twice.phpt
* Clean up the implementation of Randomizer::__construct()
Instead of manually checking whether the constructor was already called, we
rely on the `readonly` modifier of the `$engine` property.
Additionally use `object_init_ex()` instead of manually calling
`->create_object()`.
* Remove exception in Randomizer::shuffleBytes()
The only way that `php_binary_string_shuffle` fails is when the engine itself
fails. With the currently available list of engines we have:
- Mt19937 : Infallible.
- PcgOneseq128XslRr64: Infallible.
- Xoshiro256StarStar : Infallible.
- Secure : Practically infallible on modern systems.
Exception messages were cleaned up in GH-9169.
- User : Error when returning an empty string.
Error when seriously biased (range() fails).
And whatever Throwable the userland developer decides to use.
So the existing engines are either infallible or throw an Exception/Error with
a high quality message themselves, making this exception not a value-add and
possibly confusing.
* Remove exception in Randomizer::shuffleArray()
Same reasoning as in the previous commit applies.
* Remove exception in Randomizer::getInt()
Same reasoning as in the previous commit applies.
* Remove exception in Randomizer::nextInt()
Same reasoning as in the previous commit applies, except that it won't throw on
a seriously biased user engine, as `range()` is not used.
* Remove exception in Randomizer::getBytes()
Same reasoning as in the previous commit applies.
* Remove exception in Mt19937::generate()
This implementation is shared across all native engines. Thus the same
reasoning as the previous commits applies, except that the User engine does not
use this method. Thus is only applicable to the Secure engine, which is the
only fallible native engine.
* [ci skip] Add cleanup of Randomizer exceptions to NEWS
The old implementation did this. It also did the same to the
trim marker, if the trim marker was invalid in the specified
encoding, but I have not imitated that behavior (for performance).
If a downstream filter returns -1 (error), the CK macro
will make the UCS-4 conversion filter also immediately
return. This means that any necessary updates to the filter
state have to be done *before* using CK, or it will be left
in an invalid state and will not behave correctly when
flushed.
In my recent commit which replaced the implementation of
mb_convert_kana, the commit message noted that mb_convert_kana
previously had a bug whereby null bytes would be 'swallowed'
and not passed to the output.
This was actually the reason.
This new implementation of mb_strimwidth uses the new text
encoding conversion filters. Changes from the previous
implementation:
• mb_strimwidth allows a negative 'from' argument, which
should count backwards from the end of the string. However,
the implementation of this feature was buggy (starting right
from when it was first implemented).
It used the following code:
if ((from < 0) || (width < 0)) {
swidth = mbfl_strwidth(&string);
}
if (from < 0) {
from += swidth;
}
Do you see the bug? 'from' is a count of CODEPOINTS, but
'swidth' is a count of TERMINAL COLUMNS. Adding those two
together does not make sense. If there were no fullwidth
characters in the input string, then the two counts coincide
and the feature would work correctly. However, each
fullwidth character would throw the result off by one,
causing more characters to be skipped than was requested.
• mb_strimwidth also allows a negative 'width' argument,
which again counts backwards from the end of the string;
in this case, it is not determining the START of the portion
which we want to extract, but rather, the END of that portion.
Perhaps unsurprisingly, this feature was also buggy.
Code:
if (width < 0) {
width = swidth + width - from;
}
'swidth + width' is fine here; the problem is '- from'.
Again, that is subtracting a count of CODEPOINTS from a
count of TERMINAL COLUMNS. In this case, we really need
to count the terminal width of the string prefix skipped
over by 'from', and subtract that rather than the number
of codepoints which are being skipped.
As a result, if a 'from' count was passed along with a
negative 'width', for every fullwidth character in the
skipped prefix, the result of mb_strimwidth was one
terminal column wider than requested.
Since these situations were covered by unit tests, you
might wonder why the bugs were not caught. Well, as far as
I can see, it looks like the author of the 'tests' just
captured the actual output of mb_strimwidth and defined it
as 'correct'. The tests were written in such a way that it
was difficult to examine them and see whether they made
sense or not; but a careful examination of the inputs and
outputs clearly shows that the legacy tests did not conform
to the documented contract of mb_strimwidth.
• The old implementation would always pass the input string
through decoding/encoding filters before returning it to
the caller, even if it fit within the specified width. This
means that invalid byte sequences would be converted to
error markers. For performance, the new implementation
returns the very same string which was passed in if it
does not exceed the specified width. This means that
erroneous byte sequences are not converted to error markers
unless it is necessary to trim the string.
• The same applies to the 'trim marker' string.
• The old implementation was buggy in the (unusual)
case that the trim marker is wider than the requested
maximum width of the result. It did an unsigned subtraction
of the requested width and the width of the trim marker. If the
width of the trim marker was greater, that subtraction would
underflow and yield a huge number. As a result, mb_strimwidth
would then pass the input string through, even if it was
far wider than the requested maximum width.
In that case, since the input string is wider than the
requested width, and NONE of it will fit together with the
trim marker, the new implementation returns just the trim
marker. This is the one case where the output can be wider
than the requested width: when BOTH the input string and
also the trim marker are too wide.
• Since it passed the input string and trim marker through
decoding/encoding filters, when using "Quoted-Printable" as
the encoding, newlines could be inserted into the trim marker
to maintain the maximum line length for QP.
This is an extremely bizarre use case and I don't think there
is any point in worrying about it. QP will be removed from
mbstring in time, anyways.
PERFORMANCE:
• From micro-benchmarking with various input string lengths and
text encodings, it appears that the new implementation is 2-3x
faster for UTF-8 and UTF-16. For legacy Japanese text encodings
like ISO-2022-JP or SJIS, the new implementation is perhaps 25%
faster.
• Note that correctly implementing negative 'from' and 'width'
arguments imposes a small performance burden in such cases; one
which the old implementation did not pay. This slightly skews
benchmarking results in favor of the old implementation. However,
even so, the new implementation is faster in all cases which I
tested.
We are removing the entry in 8.0.23, as the fix was included
in the 8.0.22 release. Given that an entry already exists, we
are just deleting extra lines.
* Unify ext/random unserialize errors with ext/date
- Use `Error` instead of `Exception`.
- Adjust wording.
* Make `zend_read_property` silent in `Randomizer::__unserialize()`
Having:
> Error: Typed property Random\Randomizer::$engine must not be accessed before
> initialization
is not a value-add in this case.
* Insert the actual class name in the unserialization error of Engines
* Revert unserialization failure back to Exception from Error
see https://news-web.php.net/php.internals/118311
Since argument overloading is not safe for reflection, the method needed
to be split appropriately.
Co-authored-by: Tim Düsterhus <timwolla@googlemail.com>
Closes GH-9057.
