Download as ODP, PPTX
S313352 optimizing java device testing with automatic feature discovering
- 1.
- 2. Optimizing Java DeviceTesting with Automatic Feature Discover y Fyodor Romanov, Software Engineer
- 3. Agenda Java devicetesting challenges.
- 4.
- 5. Approach how tooptimize and shorten device test-cycle using Diagnostic tests to explore device capabilities.
- 6. Demo showing Diagnostictests usage and benefits
- 7.
- 8. Problem description Nearly3 Billion devices run Java ME in the world today.
- 9. Before a manufacturercan make a device commercially available, they must perform a significant amount of testing to ensure a high-quality, reliable product that delivers customer satisfaction.
- 10. Java Device qualitytesting helps Java technology wireless device manufacturers and network operators maximize product quality and minimize time to market.
- 11. Problem description Thereare a lot of (more then 30) Java Platform, Micro Edition (Java ME platform) technologies (and combinations):
- 12. Problem description Alot of technologies, a lot of tests. More then 11,000 tests are developed currently in JDTS and number is growing.
- 13. Up to 3weeks for guru to run all test on one device.
- 14. Challenge of managingand, if possible, lowering quality testing costs.
- 15. The most challengingtask for test execution is to configure tests for particular device.
- 16. Express testing: evaluatedevice quality in 4 hours.
- 17. Problem description Notto test technologies which device does not support. Negative false cases complicate failure analysis
- 18. Problem statement Devicemanufacture knows, but testing house doesn't.
- 19. Before Automatic featurediscovery.
- 20.
- 21.
- 22. Solution implementation Implementationexample will be based on JDTF test framework however it should be applicable to any Java device testing framework. The Java Device Test Framework (JDTF) is a test framework based on Oracle‘ s commercial Java Device Test Suite (JDTS) product. JDTF is a general purpose, fully-featured, flexible, and configurable test framework suited to assess various aspects of Java Platform, Micro Edition (Java ME) device implementation quality .
- 23. JDTF overall testsstructure JDTF organizes tests into hierarchical structure s called test packs , a collection of tests that are functionally related and have common setup requirements :
- 24. JDTF testing structureHarness is using tests and different configuration files to execute test on device. Test execution on device is controlled by MicroAgent which starts/stops test and communicates with Harness (sends results, logs, etc.)
- 25.
- 26. How to distinguishdiagnostic tests. JDTF provides keywords to filter (subset for execution) tests in test packs. So two options available to distinguish diagnostic tests: Mark them with special keyword, for example ‘diagnostic’
- 27. Move them intospecial dedicated test pack.
- 28. Writing diagnostic test.Changes in MicroAgent. Diagnostic tests find out whether the device supports an optional feature and export a property by using Logger class as follows. public class Logger { ... public void exportProperty(String name, String value) { LogRecord rec = new LogRecord (...); rec.setProperty(EXPORTED_PROPERTY_PREFIX + name, value); ... } }
- 29. Writing diagnostic testTest name should clear indicate what optional feature is tested and placed in appropriate package, for example: ConnectionTest.testHTTP
- 30. Test should bemarked with ‘diagnostic’ keyword
- 31. Test should tryto use optional feature and call Logger.exportProperty(String name, String value)
- 32. If feature issupported, return PASSED and if feature is not supported return FAILED. This helps to visualize test result.
- 33.
- 34. Writing diagnostic test.Code Sample. public class ConnectionTest implements AutomatedTest { ... public boolean testHTTP() { Connection c = null; try { ... c = Connector.open(url); logger.exportProperty("HTTPSupport", "true"); return true; } catch (IllegalArgumentException iae) { return false; } ... }
- 35.
- 36. Adding Template Buildersto TestPacks
- 37.
- 38. Final touch. AdjustingRelevance Filters Relevance filter should filter out not relevant test providing right reason. Code sample: public String LapiDependencyProvider.isRelevant(final TestId id) throws TestDependencyProviderException { final boolean createDeleteLmsSupported = getBoolean(TestPackProperty.CreatingAndDeletingLandma rkStoresSupported, id); if (!createDeleteLmsSupported) { if (fqn.endsWith("#addLandmarkScenario2") || fqn.endsWith("#deleteLandmarkFromAnotherLMS") { return "Test not relevant because Landmark Store create/delete operations configured to be not supported."; } }
- 39. Automatic Feature DiscoveryEfficiency. Proved to be very valuable.
- 40. Reduces 1 -2 days work for one person (depending of person's experience) to 30 minutes.
- 41. Before AutomaticFeature Discovery feature customers were taking a complete unknown device and painstakingly reading logs to find out what it supports .
- 42.
- 43. Useful links JDTFProject home page https://jdtf.dev.java.net with source code
- 44. JDTS home pagewith API detailed description
- 45.
- 46. Session k eypoints How Diagnostic tests can help in Java device quality testing Tips how to develop such tests
- 47. How to process Diagnostic tests output in testing framework JDTF test framework was used for demonstration <Insert Picture Here>
- 48.
- 49.
Editor's Notes
- #6 The growing complexity and diversity of devicebswith their varying operating systems, processors, and memory configurationbsincreases the need for thorough testing. At the same time, service providers and manufacturers face the challenge of managing and, if possible, lowering in-house costs. The Java Device testing helps device manufacturers and service providers ensure their reputation for quality, while building customer satisfaction and loyalty.
- #7 Java ME technology was originally created in order to deal with the constraints associated with building applications for small devices. For this purpose Sun defined the basics for Java ME technology to fit such a limited environment and make it possible to create Java applications running on small devices with limited memory, display and power capacity. Java ME platform is a collection of technologies and specifications that can be combined to construct a complete Java runtime environment specifically to fit the requirements of a particular device or market. The Java ME technology is based on three elements; * a configuration provides the most basic set of libraries and virtual machine capabilities for a broad range of devices, * a profile is a set of APIs that support a narrower range of devices, and * an optional package is a set of technology-specific APIs.
- #9 Failture analisys becomes complicated because even if a technology is supported, most of them have optional features which also can be unsupported. For example, JSR293 Location API 2.0 declares following optional features: Creating and deleting landmark stores, including private landmark stores Adding, removing and renaming landmark categories Adding and removing LandmarkStoreListener
- #10 Usually there are no proper available documentation for device and it’s manual and error-prone task to discover what technologies and optional features device supports and update special configuration file (template) accordingly.
- #11 Identifying what technologies device supports includes writing so-called Diagnostic tests that check particular optional feature by trying to use it and then analyze tests logs. Based on that user should set properties in device configuration file (template). I t is time-consuming to have to read through the log file and then interpret and w rite values generated there into configuration file. T hings that make it cumbersome are : Value key names may not match correctly: example Media Format names like MP4 Values are in different places, i.e. not grouped by feature/function/JSR.
- #12 Use Diagnostic tests to explore a device’s capabilities, such as : the presence of optional APIs, the presence of JSRs using Class.forName() canvas size, a number of colors ,… etc and report back to the harness. After running the automated diagnostic tests on a device, AUTOMATICALLY create a template that is customized for the device. In the generated template tests that cannot pass because of a missing feature, are filtered out. Using such a template saves testing time by running only tests that are relevant for a device.
- #17 When configuration is finished and test execution is not started, JDTF harness calls an optional test pack method for each test case. This method inspects the configuration and returns null if the test is relevant or a list of reasons describing why the test is irrelevant. An irrelevant test is one that is pointless to run because o ne or more properties in the configuration, which essentially represents the capabilities of the test device, makes a test irrelevant. For example, a property representing the presence of an optional device capability is set to false.
- #23 * Represents an entity that can convert device profile properties * to the test pack configuration values. * This interface intended to be implemented by the API clients. * This method maps Device Profile properties to test pack properties which * are put in the device profile template. * @param diagnosticsProperties a map which binds a test. pack id with a map * consisting of device profile properties and their values. * @return a map of test pack properties and their values.
- #27 Demo includes running JDTF with Automatic Feature Discovery feature implemented, running couple of diagnostic tests, generating template and demonstrating that appropriate tests are filtered out with proper reason.