Skip to content

Fix inconsistency between J9Class and annotation data when redefine - #13424

Merged
DanHeidinga merged 1 commit into
eclipse-openj9:masterfrom
EricYangIBM:fixAnnotationCache
Oct 28, 2021
Merged

DanHeidinga merged 1 commit into
eclipse-openj9:masterfrom
EricYangIBM:fixAnnotationCache

Conversation

@EricYangIBM

@EricYangIBM EricYangIBM commented Sep 1, 2021 •

Copy link
Copy Markdown
Contributor

If a class is redefined between annotation fetch and parse, a
Wrong type at constant pool index or Constant pool index out of bounds
exception may be thrown due to inconsistency between the annotation data
and J9Class. Pass the J9ConstantPool along with the annotation data to
ensure that parseAnnotations looks up the correct constant pool.

  • Add new class InternalConstantPool that ConstantPool will hold an
    instance of (instead of InternalRamClass)
  • Change getConstantPool to take InternalConstantPool or Class
    (if Class, create InternalConstantPool from it).
  • Change sun/reflect/ConstantPool natives to take an InternalConstantPool
    argument so that the correct constant pool is looked up (e.g. after HCR)
  • Add a jithelper to get J9ConstantPool from J9Class
  • Add test that redefines a class while fetching its annotations

Closes: #10026
Signed-off-by: Eric Yang eric.yang@ibm.com

@EricYangIBM

Copy link
Copy Markdown
Contributor Author

@tajila I am confused about the existing code. This line https://github.com/EricYangIBM/openj9-openjdk-jdk11/blob/openj9/src/java.base/share/classes/sun/reflect/annotation/AnnotationParser.java#L119 implies that the start of the byte array holds the number of annotations. However this doesn't copy the number of annotations to the byte array, just the data. I'm not sure how parseAnnotations is getting the number of annotations. Is it somehow looking at the array header?

@tajila

tajila commented Sep 2, 2021

Copy link
Copy Markdown
Contributor

However this doesn't copy the number of annotations to the byte array, just the data.

The first two bytes of the data is the number of annotations. See findRuntimeVisibleAnnotation for the format of the data.

@EricYangIBM

Copy link
Copy Markdown
Contributor Author

Is there a way to find the start of a J9Class reference that is appended to the byte array? I.e. I think I have to check the size of the reference and subtract it from the length.

@tajila

tajila commented Sep 2, 2021

Copy link
Copy Markdown
Contributor

Is there a way to find the start of a J9Class reference that is appended to the byte array? I.e. I think I have to check the size of the reference and subtract it from the length.

yes basically once you know the size of the array, then you can extract the j9class using the technique you mentioned. The size of j9class will be J9VMTHREAD_OBJECT_HEADER_SIZE(currentThread)

@EricYangIBM

Copy link
Copy Markdown
Contributor Author

J9VMTHREAD_OBJECT_HEADER_SIZE(currentThread)

I meant in the java code to pass to getConstantPool

@tajila

tajila commented Sep 6, 2021

Copy link
Copy Markdown
Contributor

com.ibm.oti.vm.VM.FJ9OBJECT_SIZE

@EricYangIBM
EricYangIBM force-pushed the fixAnnotationCache branch 2 times, most recently from 17946de to aa7f323 Compare September 7, 2021 17:16
@EricYangIBM
EricYangIBM marked this pull request as ready for review September 7, 2021 17:21
@EricYangIBM EricYangIBM changed the title [WIP] Fix inconsistency between J9Class and annotation data when redefine Fix inconsistency between J9Class and annotation data when redefine Sep 7, 2021
@EricYangIBM
EricYangIBM requested a review from tajila September 7, 2021 17:22
@tajila

tajila commented Sep 7, 2021

Copy link
Copy Markdown
Contributor

Can you add a test that verifies there is no timing hole with redefinition.

Take a look at openj9/test/functional/cmdLineTests/jvmtitests/src/com/ibm/jvmti/tests/retransformClasses* and /openj9/test/functional/cmdLineTests/jvmtitests/src/com/ibm/jvmti/tests/redefineClasses* for examples of this

@EricYangIBM
EricYangIBM force-pushed the fixAnnotationCache branch 5 times, most recently from 6009562 to e423e0e Compare September 15, 2021 18:16
@EricYangIBM

Copy link
Copy Markdown
Contributor Author

I'm trying to set the annotationCache to null in the test so that the fetch and parse path is taken. My attempt at nulling the cache using reflection doesn't seem to do anything. I think I may have to set it using the annotationCacheOffset like in getAnnotationCache()? However when I try using Unsafe.getUnsafe() I get a security exception. Is there a way around this?

@pshipton

pshipton commented Sep 15, 2021 •

Copy link
Copy Markdown
Member

I don't see why using reflection wouldn't work. The annotationCache doesn't seem to be filtered from reflection. Unsafe and the annotationCacheOffset are used to avoid synchronization when reading the annotationCache field.

Regardless, I think you can work around the Unsafe.getUnsafe() restriction by using reflection to get the Unsafe.theUnsafe field.

@pshipton

Copy link
Copy Markdown
Member

@EricYangIBM

Copy link
Copy Markdown
Contributor Author

After adding this code (https://github.com/eclipse-openj9/openj9/pull/13424/files#diff-f52ad38ef1cc6b03312e51701549d9d406fca56be88db1989fe47a4aa08970e1R44-R47), the not null cache path (I added a print) was still taken 99 times out of 100. Perhaps it was overwritten by other threads?

@pshipton

pshipton commented Sep 16, 2021 •

Copy link
Copy Markdown
Member

Yes, if any annotation is requested the cache would be recreated.

@EricYangIBM

Copy link
Copy Markdown
Contributor Author

Setting the cache through the offset doesn't work either. If I add annotationCache = null; at the start of getAnnotationCache() the test fails without this PR's code changes and passes with them. I'm just not sure how to set it in the test code.

@pshipton

pshipton commented Sep 17, 2021 •

Copy link
Copy Markdown
Member

Use reflection to set Class.reflectCacheEnabled to false, and afterwards it won't be used any more.

@pshipton

Copy link
Copy Markdown
Member

It would be good to restore it afterwards for subsequent testing.

@pshipton

Copy link
Copy Markdown
Member

Sorry, ignore that, it's for a different cache.

@pshipton

Copy link
Copy Markdown
Member

The VM nulls the annotationCache appropriately when classes are redefined, You can look at callers of flushClassLoaderReflectCache(), the test shouldn't need to (and shouldn't) do it. If it's not happening you need to dig into why the VM isn't doing it for your test.

If you run the test without the fix, does it reproduce the problem?

@EricYangIBM

EricYangIBM commented Sep 17, 2021 •

Copy link
Copy Markdown
Contributor Author

If you run the test without the fix, does it reproduce the problem?

No, the prints show that the cache is not null and the test passes.

@EricYangIBM
EricYangIBM force-pushed the fixAnnotationCache branch 2 times, most recently from cb88a22 to 9f2f5b4 Compare October 21, 2021 19:32
@EricYangIBM

EricYangIBM commented Oct 21, 2021 •

Copy link
Copy Markdown
Contributor Author

Added a test for type annotations for implemented interfaces and extended superclasses, passes with the changes, fails without. (Same "constantpool index out of bounds" exception)

I had an issue where the second test run would not execute the redefines properly, not sure why. The first test would fail (without the changes) and the second would pass because the redefines were not done. This did not depend on either test; I renamed tests to change the order of execution and still the second test would pass.
I fixed this by adding different classes to test the annotation type changes on.

@tajila

tajila commented Oct 27, 2021

Copy link
Copy Markdown
Contributor

I had an issue where the second test run would not execute the redefines properly, not sure why.

Did you get an error message?

@EricYangIBM

Copy link
Copy Markdown
Contributor Author

No error messages. It just passed with printlns of the @Deprecated annotation (the test/functional/cmdLineTests/jvmtitests/src/com/ibm/jvmti/tests/redefineClasses/rc021_testAnnotationCache_R.java class)

@tajila

tajila commented Oct 27, 2021

Copy link
Copy Markdown
Contributor

Jenkins test sanity win jdk8

@tajila

tajila commented Oct 27, 2021

Copy link
Copy Markdown
Contributor

Jenkins test sanity,extended plinux jdk11

If a class is redefined between annotation fetch and parse, a
`Wrong type at constant pool index` or `Constant pool index out of bounds`
exception may be thrown due to inconsistency between the annotation data
and J9Class. Pass the J9ConstantPool along with the annotation data to
ensure that `parseAnnotations` looks up the correct constant pool.

- Add new class `InternalConstantPool` that `ConstantPool` will hold an
  instance of (instead of `InternalRamClass`)
- Change `getConstantPool` to take `InternalConstantPool` or `Class`
  (if `Class`, create `InternalConstantPool` from it).
- Change `sun/reflect/ConstantPool` natives to take an `InternalConstantPool`
  argument so that the correct constant pool is looked up (e.g. after HCR)
- Add a jithelper to get `J9ConstantPool` from `J9Class`
- Add test that redefines a class while fetching its annotations

Closes: eclipse-openj9#10026
Signed-off-by: Eric Yang <eric.yang@ibm.com>
@EricYangIBM

Copy link
Copy Markdown
Contributor Author

@tajila

tajila commented Oct 27, 2021

Copy link
Copy Markdown
Contributor

Jenkins test sanity,extended plinux jdk11

@tajila

tajila commented Oct 27, 2021

Copy link
Copy Markdown
Contributor

JDK8 tests passed

@DanHeidinga DanHeidinga left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm (didn't review the test code)

@DanHeidinga

Copy link
Copy Markdown
Member

@tajila any further concerns? Otherwise I'll merge this

@DanHeidinga DanHeidinga self-assigned this Oct 28, 2021
@tajila

tajila commented Oct 28, 2021

Copy link
Copy Markdown
Contributor

LGTM @DanHeidinga you can go ahead

@DanHeidinga
DanHeidinga merged commit c430223 into eclipse-openj9:master Oct 28, 2021
@EricYangIBM
EricYangIBM deleted the fixAnnotationCache branch October 28, 2021 15:22
@keithc-ca

Copy link
Copy Markdown
Contributor

Did anyone try a 32-bit Windows build? I expect the compiler to complain about casts between signed and unsigned types of differing sizes:

jlong JNICALL
Java_com_ibm_oti_vm_VM_getJ9ConstantPoolFromJ9Class(JNIEnv *env, jclass unused, jlong j9clazz)
{
	J9Class *clazz = (J9Class *)(UDATA)j9clazz;
	return (jlong)(UDATA)clazz->ramConstantPool;
}

I think those casts should use IDATA instead of UDATA.

@pshipton

Copy link
Copy Markdown
Member

Guessing this is the cause of #13825. I'm creating a revert PR to use for confirming this.

@pshipton

pshipton commented Oct 29, 2021 •

Copy link
Copy Markdown
Member

Grinders show this is not the cause.

@tajila

tajila commented Oct 29, 2021

Copy link
Copy Markdown
Contributor

@EricYangIBM Please create a PR to address Keiths comment above

EricYangIBM added a commit to EricYangIBM/openj9 that referenced this pull request Oct 29, 2021
Change UDATA casts to use IDATA when casting to and from jlong.

PR: eclipse-openj9#13424
Signed-off-by: Eric Yang <eric.yang@ibm.com>
@pshipton

Copy link
Copy Markdown
Member

I didn't realize the failing test was added by this PR. Reverting as this is the cause of #13825 and also the cause of an internal test failure.

Note in the nightly testing the failure #13825 didn't occur on plinux on either jdk8 or jdk11. Maybe more intermittent or doesn't occur on this platform. The failures also didn't occur on jdk17.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make Class's annotation cache more resilient across redefines

6 participants