Add tool for sending mobile integration tests to FTL. #117
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Building an integration test generates 400-1000+ megabytes of unnecessary dependencies and intermediates files. To improve performance and avoid OOM errors, these are deleted after each build that generates them.
Test scripts were moved to scripts/gha, but the workflows were not updated to reflect this.
This tool sends testapps (apks and ipas) to Firebase Test Lab, and then uses their log output to validate them. This is a heavily modified/simplified version of the tool that exists in google3 (firebase/app/client/unity/testapp_builder/firebase_test_lab). At a high level the tool works as follows: - Authorize the use of a Google Cloud project to use its storage bucket. - Search a directory for apks/ipas. - Send them to FTL for testing, using the gcloud api. FTL will store their log output in the GCS bucket. - Once they're done, find the testapp logs in GCS using gsutil, a command line tool installed as part of the Cloud SDK. - Validate the test results in the testapp logs and provide a summary of all the results. Changes to the tool in google3 include: - Significantly less code. - Makes far fewer assumptions about where FTL puts stuff in cloud storage. No longer requires hard-coded information about FTL's devices (which change over time). - Cleans up the output. - Prints the output from FTL when sending testapps, instead of storing it in a file on GCS. - Prints more helpful information if something unexpected occurs, like being unable to find the testapp log on GCS. This includes a recursive dump of everything added to the GCS bucket for this run, in case anything moved around. This will also make it easier to find items of interest, such as video links. "gsutil cp gs://<etc> <local_path>" can be used to copy the relevant items for further debugging. - Does not require Linux anymore. Should work on all three platforms, though I haven't tried it on Windows. Using the tool requires installation of the Google Cloud SDK, which is a requirement of using FTL from the command line. There's a convenient Github action which handles this, though we may add some automation to install this later to simplify local setup.
* add secret files for instance_id integration test * disable auth integration test on linux due to keychain access Co-authored-by: cynthiajiang <cynthiajiang@google.com> Co-authored-by: Cynthia J <cynthiajoan@users.noreply.github.com>
The workaround is to build twice. We capture the output and check for the presence of the failed task associated with the issue in question. If present, we build again. Otherwise, we raise an error that mimicks the error that would have been raised by running the subprocess with check=True, to maintain the same behaviour. This will make it easier to remove this workaround if/when the underlying issue is fixed.
…ebase/firebase-cpp-sdk into feature/ak-integration-tests
vimanyu reviewed Sep 1, 2020
DellaBitta reviewed Sep 1, 2020
Summary consisted of a single character instead of everything starting from that character. Plus a couple of small documentation improvements/fixes.
…ebase/firebase-cpp-sdk into feature/ak-integration-tests
Feature/ddb fix msg auth
anonymous-akorn dismissed stale reviews from vimanyu and DellaBitta via 1606450 DellaBitta approved these changes Sep 1, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
4 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
The main change is the addition of the Firebase Test Lab tool for testing mobile testapps. The third commit (c0f3bc7) has a detailed commit message explaining the tool and its differences from its counterpart in google3.
Another important change is to remove intermediate deps/objects after each integration test. These were found to contribute 400-1200 megabytes per project and are not needed after building. This should address the issue with Windows running out of disk space for the integration test workflow.