Skip to content

Fix ConcurrentModificationException #6732 #6746

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from
Closed

Fix ConcurrentModificationException #6732 #6746

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
ab84470
Fix ConcurrentModificationException #6732
neugartf Sep 25, 2024
b115c5a
Move to stringPropertyNames in order to iterate on a stable list.
neugartf Sep 26, 2024
af48863
Improve stream handling of system property
neugartf Sep 26, 2024
329c05d
Merge branch 'open-telemetry:main' into main
neugartf Sep 26, 2024
d736406
Cloning properties in order to safely iterate over them
neugartf Sep 29, 2024
File filter

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 12 additions & 5 deletions api/all/src/main/java/io/opentelemetry/api/internal/ConfigUtil.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@

import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import java.util.Properties;
Comment on lines 8 to +11
Copy link
Contributor

@shalk shalk Sep 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import java.util.Properties;
import java.util.Collections;
import java.util.Locale;
import java.util.Map;
import javax.annotation.Nullable;

/**
Expand All @@ -33,12 +35,17 @@ private ConfigUtil() {}
*/
public static String getString(String key, String defaultValue) {
String normalizedKey = normalizePropertyKey(key);

// Cloning in order to avoid ConcurrentModificationException
// see https://github.com/open-telemetry/opentelemetry-java/issues/6732
String systemProperty =
System.getProperties().entrySet().stream()
.filter(entry -> normalizedKey.equals(normalizePropertyKey(entry.getKey().toString())))
.map(entry -> entry.getValue().toString())
.findFirst()
.orElse(null);
((Properties) System.getProperties().clone())
Copy link
Member

@jack-berg jack-berg Oct 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to have a test recreating this CME and demonstrating that this fixes it. However, I spent a decent amount of time trying to hammer this with concurrent reads and writes and was unable to recreate it, so it could be specific to some version of java or the jdk.

I'm content give this a shot even without strong proof that it resolves it.
Copy link
Contributor

@laurit laurit Oct 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like it does not reproduces on jdk 11+ probably because of https://github.com/openjdk/jdk11u/blob/cee8535a9d3de8558b4b5028d68e397e508bef71/src/java.base/share/classes/java/util/Properties.java#L159-L165
On jdk8 it reproduces with

import java.util.concurrent.Executor; import java.util.concurrent.Executors; import java.util.concurrent.atomic.AtomicInteger; public class Main { public static void main(String[] args) { int threads = 4; AtomicInteger counter = new AtomicInteger(); Executor executor = Executors.newFixedThreadPool(threads); for (int i = 0; i < threads; i++) { executor.execute(new Runnable() { @Override public void run() { while (true) { String property = "prop " + counter.getAndIncrement(); System.setProperty(property, "a"); System.getProperties().remove(property); } } }); } while (true) { System.getProperties().entrySet().stream() .filter( entry -> "x".equals(entry.getKey().toString()) ) .map(entry -> entry.getValue().toString()) .findFirst() .orElse(null); } } }

ConcurrentModificationException does not reproduce with the proposed fix. It doesn't reproduce with

Properties properties = System.getProperties(); synchronized (properties) { properties.entrySet().stream() .filter( entry -> "x".equals(entry.getKey().toString()) ) .map(entry -> entry.getValue().toString()) .findFirst() .orElse(null); }

either. Unlike the proposed fix this version does not make extra copies of system properties.
Copy link
Member

@jack-berg jack-berg Oct 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good find @laurit!

I know there was some skepticism about taking a lock on System.getProperties(), but it seems ok to me. The work we're doing inside the lock is narrow and its hard for me to imagine how we could end up in a dead lock scenario.
.stringPropertyNames().stream()
.filter(propertyName -> normalizedKey.equals(normalizePropertyKey(propertyName)))
.map(System::getProperty)
.filter(Objects::nonNull)
.findFirst()
.orElse(null);
Comment on lines +42 to +48
Copy link
Contributor

@shalk shalk Sep 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
((Properties) System.getProperties().clone())
.stringPropertyNames().stream()
.filter(propertyName -> normalizedKey.equals(normalizePropertyKey(propertyName)))
.map(System::getProperty)
.filter(Objects::nonNull)
.findFirst()
.orElse(null);
String systemProperty =
Collections.unmodifiableSet(System.getProperties().stringPropertyNames()).stream()
.filter(propertyName -> normalizedKey.equals(normalizePropertyKey(propertyName)))
.map(System::getProperty)
.findFirst()
.orElse(null);
Copy link
Member

@jack-berg jack-berg Oct 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unmodifiable set is superfluous since there's no attempt to modify the contents.
if (systemProperty != null) {
return systemProperty;
}
Expand Down
Loading