Support new PHP 8.4 syntaxes #734

PHP 8.4 introduces new syntaxes for which it should be verified if the Tokenizer needs updates and/or whether any sniffs need updating.

Action list

Notes:

• While Property Hooks is a single RFC, it introduces multiple new syntaxes. With that in mind, I've broken the RFC up into separately actionable parts.
• Action items in italics are tentative. Investigation and testing is needed to determine whether any changes are needed and if so, which.
• For action items not in italics, I'm fairly certain those will be needed.
• This list is not set in stone and is subject to change once investigation for supporting a syntax starts.

Action list

• Asymmetric Visibility v2
  • Proposal needed on how to handle this for the tokenizer - @DanielEScherzer PR 8.4 | Asymmetric Visibility support #851
  • Tokenizer changes needed - property declarations - @DanielEScherzer PR PHP 8.4 | Add tokenization of asymmetric visibility #871
  • Tokenizer changes needed - constructor property promotion - @DanielEScherzer PR PHP 8.4 | Add tokenization of asymmetric visibility #871
  • Updates needed to the predefined token collections in the Tokens class - @DanielEScherzer PR PHP 8.4 | Add tokenization of asymmetric visibility #871
  • Updates needed to utility functions - getMemberProperties(), getMethodParameters() (for constructor property promotion) - @jrfnl PR PHP 8.4 | File::getMemberProperties() + File::getMethodParameters(): add support for asymmetric visibility #1116
    These updates should include special handling for the abbreviated form.
  • Updates needed to abstract sniffs ?
    This needs investigation, but I imagine the AbstractScopeSniff and the AbstractVariableSniff might need updates.
  • Sniff updates needed - @jrfnl PRs Generic/LowerCaseKeyword: add support for asym visibility keywords #1117, PEAR/ValidVariableName: update for properties with asymmetric visibility #1118, PSR2/PropertyDeclaration: update for properties with asymmetric visibility #1119, Squiz/Commenting sniffs: update for properties with asymmetric visibility  #1120, Squiz/FunctionDeclarationArgumentSpacing: handle asym modifiers for constructor property promotion #1121, Squiz/MemberVarScope: update for properties with asymmetric visibility #1122, Squiz/MemberVarSpacing: update for properties with asymmetric visibility #1123, Squiz/ScopeKeywordSpacing: update for properties with asymmetric visibility #1124, Various sniffs: add tests for properties with asymmetric visibility #1125
• new MyClass()->method() without parentheses
  • Tokenizer changes needed - @jrfnl PR PHP 8.4 | Tokenizer/PHP: allow for anon class dereferencing #1160
  • Updates needed to utility functions ?
  • Sniff updates needed ? (or possibly only sniff test updates?)
• Transform exit() from a language construct into a standard function
  • Proposal needed on how to handle this for the tokenizer - @jrfnl PR Proposal to fix Tokenizer issue with FQN exit/die/true/false/null #1201
  • Tokenizer changes needed - @jrfnl PR Tokenizer/PHP: improved tokenization of fully qualified exit/die/true/false/null #1206
  • Updates needed to utility functions ?
  • Sniff updates needed - @jrfnl PR Tokenizer/PHP: improved tokenization of fully qualified exit/die/true/false/null #1206
• Property hooks - final properties
  • Tokenizer changes needed - @jrfnl - PR PHP 8.4 | Tokenizer/PHP: fix union/intersection/DNF type tokenization with final properties #907
  • Updates needed to utility functions - getMemberProperties(), getMethodParameters() (for constructor property promotion) - @DanielEScherzer, getMemberProperties() PR PHP 8.4 | Report final properties in File::getMemberProperties() #834
  • Sniff updates needed - @jrfnl PRs PHP 8.4 | Generic/[Lower|Upper]CaseConstant: add support for final properties #948, PHP 8.4 | Squiz/VariableComment: add support for final properties #949, PHP 8.4 | PSR2/PropertyDeclaration: add support for final properties #950, Squiz/DocCommentAlignment: add support for final classes and final properties #951, PHP 8.4 | Various sniffs: add tests with final properties #952
• Property hooks - abstract properties
  • Tokenizer changes needed - @jrfnl PR PHP 8.4 | Tokenizer/PHP: fix union/intersection/DNF type tokenization with abstract properties #1183
  • Updates needed to utility functions - getMemberProperties() - @jrfnl PR PHP 8.4 | File::getMemberProperties(): add support for abstract properties #1184
  • Sniff updates needed - @jrfnl PRs PHP 8.4 | Generic/[Lower|Upper]CaseConstant: add support for abstract properties #1185, PHP 8.4 | Squiz/VariableComment: add support for abstract properties #1186, PHP 8.4 | Squiz/MemberVarSpacing: add support for abstract properties #1187, PHP 8.4 | PSR2/PropertyDeclaration: add support for abstract properties  #1188, PHP 8.4 | Various sniffs: add tests with abstract properties #1189
• Property hooks - properties on interfaces
  • Tokenizer changes needed ?
  • Updates needed to utility functions - getMemberProperties(), getMethodParameters() (for constructor property promotion) - PR File::getMemberProperties(): removed parse error warning  #991 for getMemberProperties()
  • Sniff updates needed
• Property hooks - hook syntax, long form
  • Proposal needed on how to handle this for the tokenizer
  • Tokenizer changes needed
  • Updates needed to utility functions - getMemberProperties()
  • Sniff updates needed
• Property hooks - hook syntax. short (arrow) form
  • Proposal needed on how to handle this for the tokenizer
  • Tokenizer changes needed
  • Updates needed to utility functions - getMemberProperties()
  • Sniff updates needed
• Property hooks - hook syntax. other short form variations
  • Proposal needed on how to handle this for the tokenizer
  • Tokenizer changes needed
  • Updates needed to utility functions - getMemberProperties()
  • Sniff updates needed
• Property hooks - hook syntax for abstract properties/interface properties (no body)
  • Proposal needed on how to handle this for the tokenizer
  • Tokenizer changes needed
  • Updates needed to utility functions ?
  • Sniff updates needed
• Property hooks - hook syntax in constructor property promotion, long form
  • Tokenizer changes needed
  • Updates needed to utility functions - getMethodParameters()
  • Sniff updates needed
• Property hooks - hook syntax in constructor property promotion, short form
  • Tokenizer changes needed
  • Updates needed to utility functions - getMethodParameters()
  • Sniff updates needed
• Property hooks - availability of "magic" variable within hook functions
  • Updates needed to utility functions ? - getMethodParameters() ?
  • Sniff updates needed ?

Contribution Process

If anyone familiar with the PHP and PHPCS tokenizer wants to start work on any of these, please leave a comment in this ticket to claim an action item.

If you are not familiar with either tokenizer, please spare yourself the pain 😉

New custom PHPCS tokens may be needed (like for supporting property hooks). If that's the case, please open a separate issue with a detailed proposal on how to handle the syntax, what alternatives have been considered, what the consequences would be and why the proposed option should be accepted. Please open this issue before creating a PR.

If anyone needs it, here are some examples of earlier proposals for supporting PHP syntaxes from before PHP 8.4:

• Proposal for supporting PHP 8.2+ Disjunctive Normal Form Types in the Tokenizer #387
• Reminder: revisit yield from tokenization #529 (comment)

Timeline

For anyone coming to this ticket to see when syntax support for PHP 8.4 syntaxes is expected, please do NOT spam this ticket with repeated comments asking for a timeline.
It will be ready when it's ready and whatever is ready will be included in a release forthwith.

Seriously, to add syntax support for PHP 8.4, a completely silly amount of work will be needed and people need to sleep once in a while as well (and should get a life), so please be patient.

To quote myself from #731 (comment):

... property hooks is the bane of my existence and will be a nightmare to add support for in PHPCS. The tokenizer support alone is going to take months of work, as it is not 1 syntax they have added, but over a dozen new syntaxes, all under one RFC...
Only after syntax support is available through the Tokenizer can we start fixing individual sniffs and I expect 60-70% of sniffs in the wider PHPCS field (including external standards) will need updates, or at the very least additional tests with property hooks.

I spoke up about this at the time (and about other idiocracies in property hooks, it's a massive footgun), but unfortunately to no avail.
If you want to read my full analysis of the problems with property hooks, have a read through here: https://externals.io/message/123048#123080

Oh and this comment from me on the Internals mailinglist should also help illuminate why so many sniffs will need updates and how problematic property hooks are: https://externals.io/message/123048#123118

All in all, please be patient, I expect it will be quite a while before PHPCS will have full support for property hooks.

Funding for this would help. Finding a way to clone me, would help even more.