Mentioned in https://github.com/php/php-src/pull/5965#discussion_r467621123
This PR proposes 3 lines of context so the impact can be seen in tests.
Other `diff` programs show around 3 lines of context.
(This helps indicate exactly which position a test should be updated
to add a new expected line at)
Use the mapping for choosing order to display diffs
Properly include context in cases where the expected output had more lines than
the actual output, e.g.
```
--FILE--
A
A1
A
C
NEARBY
--EXPECTF--
A
B
A1
B
A
B
A
B
NEARBY
```
Closes GH-5968
And add a --color option for run-tests.php
See https://no-color.org/
> an informal standard is hereby proposed:
>
> All command-line software which outputs text with ANSI color added should check
> for the presence of a `NO_COLOR` environment variable that, when present
> (regardless of its value), prevents the addition of ANSI color.
This uses green for lines with `+` and red for lines with `-`.
Colors(Red and Green) would make the failure causes stand out visually when scrolling through errors.
Closes GH-5965
The run-tests.php file already requires php 7 syntax.
Done automatically with phpcbf with no other changes done/planned.
This file is already using short arrays in other places.
run-tests has already been substantially refactored in PHP 8.0
compared to PHP 7.4.
This makes merge conflicts more likely, but changes to run-tests.php on older
versions are less frequent bug fixes.
Add a `zend.exception_string_param_max_len` ini setting.
(same suffix as `log_errors_max_len`)
Allow values between 0 and 1000000 bytes.
For example, with zend.exception_string_param_max_len=0,
"" would represent the empty string, and "..." would represent something
longer than the empty string.
Previously, this was hardcoded as exactly 15 bytes.
Discussion: https://externals.io/message/110717
Closes GH-5769
As discussed on GH-5632, the HTML functionality does not appear
to be in active use. For HTML rendering of test results, it is
suggested to instead use the JUnit integration, in combination
with your favorite JUnit viewer.
Closes GH-5705.
Don't connect to each worker individually: First spawn all processes
and then establish connections in any order.
This avoids a ~1s wait when running on many cores.
The test runner currently defaults to running the entire test suite if
no selected tests can be found. This can be unexpected.
For example the ext/mysqlnd/ directory has no tests, if you specify that
directory when testing the entire test suite will be run.
run-tests.php [options] ext/mysqlnd/
Closes GH-5605.
--file-cache-prime populates the file cache,
--file-cache-use uses the file cache.
And fix a number of tests to run under file cache or disabled
timestamp validation.
I used php-cs-fixer to do the cs fixes. The configuration I used is
posted below. The reason I disabled some of the rules is because they
create too much noise and would make it difficult to review. But please
feel free to close this PR and run the php-cs-fixer yourself.
<?php
$config = PhpCsFixer\Config::create();
$config->setRiskyAllowed(false);
$config->setRules([
'@PSR2' => true,
'@Symfony' => true,
'array_syntax' => false,
'binary_operator_spaces' => false,
'blank_line_before_statement' => false,
'concat_space' => false,
'increment_style' => false,
'phpdoc_align' => false,
'single_quote' => false,
'trailing_comma_in_multiline_array' => false,
'unary_operator_spaces' => false,
'yoda_style' => false,
]);
$finder = PhpCsFixer\Finder::create();
$finder->in(getcwd());
$finder->exclude('Zend');
$finder->exclude('build');
$finder->exclude('ext');
$finder->exclude('pear');
$finder->exclude('sapi');
$finder->exclude('scripts');
$finder->exclude('win32');
$config->setFinder($finder);
return $config;
Closes GH-5557.
The FILE_BINARY (and FILE_TEXT) constants are not really valid or useful
constants. It looks like they were added in 5.2.7 and have "no effect,
and are only available for forward compatibility."
See: https://www.php.net/manual/en/filesystem.constants.php
The default value of the file_put_contents() flags parameter is 0 and
FILE_BINARY is set to 0, so removing it doesn't change functionality.
P.S. Maybe those constants should be deprecated or removed in 8.0.
Closes GH-5556.
On Windows, reusing/sharing of OPcache instances with different
configuration is not necessarily supported, so we have to make that it
does not happen for the clean scripts, by using `$orig_ini_settings`
instead of `$clean_params`.
var_dump() is debugging functionality, so it should print
floating-point numbers accurately. We do this by switching
to serialize_precision, which (by default) will print with
as much precision as necessary to preserve the exact value
of the float.
This also affects debug_zval_dump().
Closes GH-5172.
The primary motivation to have each test worker running its own console
is to allow the windows_mb_path tests to run in parallel. A nice side
effect is that this also prevents changing the code page of the
tester's console window (which can even cause its font to be changed).
To be able to do so, we introduce the `create_new_console` option for
`proc_open()`, which might occasionally be useful for other purposes
than testing.
- When valgrind is used, communication overhead is relatively small,
so just use a batch size of 1.
- If this is running a small enough number of tests,
reduce the batch size to give batches to more workers.
(Previously, if there were 90 tests and -j8,
only 3 of 8 workers would get a batch of size 32 or less.
After this change, the batch size is 12 or less)
Closes GH-5098
If there's only 2 files to test, then only start 2 workers instead of N.
If there's only 1 file, then avoid parallelism entirely.
A separate option such as `--force-parallel` could be added
if this turns out to be something developers would want to do
when debugging test failures.
Currently mail related tests are split for *nix and Windows (if there
are even Windows versions). The basic difference is that the *nix
variants set the INI directive sendmail_path to just write the email to
disk, while the Windows tests use ext/imap. The latter tests are way
more verbose, and such duplicated tests are generally a pain point.
Furthermore, the Windows tests are much slower, and could not be run
without ext/imap being available.
We therefore introduce a small fakemail application, which basically
works like `tee <path> >/dev/null`, and which will be shipped with the
Windows tests packs. fakemail.exe would also need to be added to the
PHP binary SDK, so these tests could be run during developments.
To cater to the remaining differences, we also introduce support for
`{MAIL:<path>}` placeholders in the INI sections to run-tests.php. How
to use this can be seen in mail_basic.phpt, which is currently the only
modified test case, because these tests are yet supposed to fail on
Windows, due to the missing fakemail.exe in the PHP SDK.
Now that we're displaying errors in skipif, suppress this particular
category. Otherwise we get warnings in the SKIPIF of tests with
EXTENSION where the extension does not exist, and there's no way
to suppress them.
Otherwise we would try to access an array element of `false`, which
issues a notice as of PHP 7.4.0. This would happen, for instance, for
bug63447_001.phpt if CGI is not available.
- when $status is boolean, E_NOTICE appears and tests fail
- opcache is never enabled on these tests anyway.
- Add opcache.revalidate_freq=0 to testing script
Instead of checking this in the tests files it can be used directly in
the main testing script. Also this fixes failed tests.
- Use opcache_invalidate() instead of sleep() delay in some tests. Some
delays are still used since the opcache_invalidate seems to be buggy
and not working as expected.
Closes GH-4392
The classic Windows console only supports the Unicode BMP, so it can't
handle the LIGHTNING MOOD character (U+1F5F2). It also seems that
common fonts do not support the LIGHTNING (U+2607) character, so a
general removal is appropriate.
However, Windows is already "ready for the future"[1], but PHP for
Windows is not (yet).
[1] <https://devblogs.microsoft.com/commandline/windows-command-line-unicode-and-utf-8-output-text-buffer/>
Formerly, there was at most a single OPcache instance per user and the
so called system ID (which is determined from the PHP version).
Sometimes multiple OPcaches might be desired, though, particularly for
unrelated CLI scripts, which may even be necessary (e.g. for our test
suite in parallel mode).
We therefore introduce a new INI directive `opcache.cache_id` which
allows to configure independent OPcache instances for the same user.
We also use `GetUserNameW()` instead of `php_win32_get_username()`,
because the latter retrieves the user name encoded in the
`default_charset`, which can obviously yield different results for
different charsets, leading to OPcache "incompatibilities". Slightly
worse, some characters may not even be encodeable in the
`default_charset` and would be replaced by question marks, which could
result in different users sharing the same OPcache.
We also refactor, and re-use existing APIs to avoid duplicated code.
Also adds an --asan flag to run-tests.php to setup all the necessary
environment variables. Some tests are marked as skipped because they
are incompatible with asan or too slow.
I'm basing this on the DEBUG_ZTS build, which seems to give us the
most mileage.
If we're testing ini settings that cause startup failures, we'll
never get to that SKIPIF block...
Also change settings2param to return a value instead of modifying
(to a different type no less!) in place.
Also add PHP_TEST_EXTRA_ARGS environment variable, which allows
to pass on -c, -d etc flags provided by run-tests.php. Otherwise
we won't get the built-in server to run with opcache.
Add XLEAK section, this is made necessary by the support
for valgrind tools other than memcheck.
XLEAK is to a leaking test what XFAIL is to a failing test.
Tidy up summary a little to exclude Expected Fail and Expected Leak
when there are no expected fail/leaks
Several tests use `/tmp` in the `--INI--` section, but this is not
portable. We therefore introduce the `{TMP}` placeholder which
evaluates to the system's temporary directory using
`sys_get_temp_dir()`.
We also remove the doubtful `strpos()` optimization.
Motivation:
As an extension author, I want to speed up running tests in php <=7.3,
both locally and in CI (e.g. with valgrind).
This can be done by manually copying php 7.4's run-tests.php script
to replace the one generated by `phpize`
- list() doesn't work in php 7.0
- negative string offset doesn't work in php 7.2
If run-tests.php can be copied from php-src without any manual patches,
that would be the easiest.
Related to #2822 - I didn't see any discussion for/against compatibility
with older php versions
If a test conflicts with "all", then no other tests may be run in
parallel. This is needed for windows_mb_path tests, which rely on
the console codepage, which is shared across all parallel workers.
Also add support for comments in the CONFLICTS section/file.
Inherit the main stdin/stdout/stderr streams into the worker processes.
We don't talk to workers via these pipes, so there shouldn't be a
problem doing this.
This patch removes the so called local variables defined per
file basis for certain editors to properly show tab width, and
similar settings. These are mainly used by Vim and Emacs editors
yet with recent changes the once working definitions don't work
anymore in Vim without custom plugins or additional configuration.
Neither are these settings synced across the PHP code base.
A simpler and better approach is EditorConfig and fixing code
using some code style fixing tools in the future instead.
This patch also removes the so called modelines for Vim. Modelines
allow Vim editor specifically to set some editor configuration such as
syntax highlighting, indentation style and tab width to be set in the
first line or the last 5 lines per file basis. Since the php test
files have syntax highlighting already set in most editors properly and
EditorConfig takes care of the indentation settings, this patch removes
these as well for the Vim 6.0 and newer versions.
With the removal of local variables for certain editors such as
Emacs and Vim, the footer is also probably not needed anymore when
creating extensions using ext_skel.php script.
Additionally, Vim modelines for setting php syntax and some editor
settings has been removed from some *.phpt files. All these are
mostly not relevant for phpt files neither work properly in the
middle of the file.
sapi/tests/ has been introduced as a generic means to test different
SAPIs[1]. run-tests2.php has later be renamed to server-tests.php and
recently been dropped[2]. However, the sapi/tests/test00?.php test
cases remained, even though they make no sense for run-tests.php,
since they use an unsupported format for the `--ENV--` section and the
completely unsupported `--HEADERS--` section, respectively. While
these tests ran successfully under run-tests.php, that was only by
accident, and they did not really test something useful. Therefore, we
remove these tests altogether.
sapi/tests/bug69487.phpt is actually a CGI test (CGI is enforced due to
the `--POST--` section), so we move it to sapi/cgi/tests, which leaves
sapi/tests/ empty. Thus, we also remove the sapi/ directory from
run-tests.php.
Finally, we remove the `--HEADERS--` and `--REQUEST--` sections from
the list of allowed run-tests.php sections.
[1] <http://git.php.net/?p=php-src.git;a=commit;h=b671380b6b5b6e1f4f235e810afa4199e989d2ba>
[2] <http://git.php.net/?p=php-src.git;a=commit;h=4f36acb9e65935aa657f1f22e2320a401bdbdad3>
Certain sections, such as --EXTENSIONS--, --SKIPIF--, and --CLEAN--
require to be executed using only PHP CLI or PHP CGI SAPIs and cannot be
processed with the phpdbg.
This patch makes several scripts and PHP development tools files
executable and adds more proper shebangs to the PHP scripts.
The `#!/usr/bin/env php` shebang provides running the script via
`./script.php` and uses env to find PHP script location on the system.
At the same time it still provides running the script with a user
defined PHP location using `php script.php`.
* substr instead of preg_match
* cast instead of settype
* remove $borked and check if $bork_info isn't null to report borked
tests
* remove some error control operators
The junit XML format is purely documented, some existings spec like
http://llg.cubic.org/docs/junit/ also provide an XSD. The testsuite
tag included into itself doesn't seems to be correct, instead only a
flat list is included into "testsuites" tag.
- testcase classname attribute is closest to put the filename
- if tests run from php checkout, don't include the full path
- remove filename from the testcase description, as it's included into
classname
* Commands are not properly escaped for windows
* Specifying "-n" to check loaded modules causes "Module already loaded"
warning
* Extensions to be loaded need the "php_" prefix on Windows
Bug: https://bugs.php.net/bug.php?id=75042
Add back -n flag to fetch loaded extensions in run-tests.php
Add test for phpt EXTENSIONS directive
Add a second test for bug 75042
Add test to test loading of nonexistent shared module
with the EXTENSIONS phpt block
Pass ini settings when checking loaded extensions
Fix skipifs
* Commands are not properly escaped for windows
* Specifying "-n" to check loaded modules causes "Module already loaded"
warning
* Extensions to be loaded need the "php_" prefix on Windows
Bug: https://bugs.php.net/bug.php?id=75042
Add back -n flag to fetch loaded extensions in run-tests.php
Add test for phpt EXTENSIONS directive
Add a second test for bug 75042
Add test to test loading of nonexistent shared module
with the EXTENSIONS phpt block
Pass ini settings when checking loaded extensions
HEADSUP! With PCRE 8.39 the JIT related code was changed in the way,
that additional valgrind options became almost unavoidable. Valgrind
had it already sometimes hard with JIT, now there are seem to be more
cases requiring special valgrind options.
For this reason, the new configure option --with-pcre-valgrind was
introduced. The option is development/debugging only and turns on the
Valgrind related pieces in PCRE, so then false positives are avoided
to the big part. In addition, run-tests.php was added a new valgrind
option, when the leak check is enabled and the test filepath contains
pcre.
Thus, to debug the code related to PCRE with JIT enabled, two things
would likely make sense
- configure --with-pcre-valgrind
- valgrind option --smc-check=all if run-tests.php is not used
The checks so far reveal no new issues.
This reverts commit c1d5718697.
The commit seems to be inappropriate as it contains parts from master,
which have nothing to do with the variables order issue. Obviously patch
applied to wrong branch or alike.
If SKIPIF output starts with "xfail" instead of "skip", the test
will be marked as XFAIL with the following message. Example:
--TEST--
Test xfailif feature
--SKIPIF--
<?php
if (some_cond()) die('xfail Expected to fail!');
?>
--FILE--
...
