[llvm]Add a simple Telemetry framework (#102323)

Objective: - Provide a common framework in LLVM for collecting various usage metrics - Characteristics: - Extensible and configurable by: - tools in LLVM that want to use it - vendors in their downstream codebase - tools users (as allowed by vendor) Background: The framework was originally proposed only for LLDB, but there were quite a few requests to move it to llvm/lib given telemetry is a common use case in a lot of tools, not just LLDB. See more details on the design and discussions here on the RFC: https://discourse.llvm.org/t/rfc-lldb-telemetry-metrics/64588/20?u=oontvoo --------- Co-authored-by: Alina Sbirlea <alina.g.simion@gmail.com> Co-authored-by: James Henderson <James.Henderson@sony.com> Co-authored-by: Pavel Labath <pavel@labath.sk>

Author: 4 people

llvm/docs/Telemetry.rst

@@ -0,0 +1,257 @@
+===========================
+Telemetry framework in LLVM
+===========================
+
+.. contents::
+ :local:
+
+.. toctree::
+ :hidden:
+
+Objective
+=========
+
+Provides a common framework in LLVM for collecting various usage and performance
+metrics.
+It is located at ``llvm/Telemetry/Telemetry.h``.
+
+Characteristics
+---------------
+* Configurable and extensible by:
+
+ * Tools: any tool that wants to use Telemetry can extend and customize it.
+ * Vendors: Toolchain vendors can also provide custom implementation of the
+ library, which could either override or extend the given tool's upstream
+ implementation, to best fit their organization's usage and privacy models.
+ * End users of such tool can also configure Telemetry (as allowed by their
+ vendor).
+
+Important notes
+---------------
+
+* There is no concrete implementation of a Telemetry library in upstream LLVM.
+ We only provide the abstract API here. Any tool that wants telemetry will
+ implement one.
+ 
+ The rationale for this is that all the tools in LLVM are very different in
+ what they care about (what/where/when to instrument data). Hence, it might not
+ be practical to have a single implementation.
+ However, in the future, if we see enough common pattern, we can extract them
+ into a shared place. This is TBD - contributions are welcome.
+
+* No implementation of Telemetry in upstream LLVM shall store any of the
+ collected data due to privacy and security reasons:
+ 
+ * Different organizations have different privacy models:
+ 
+ * Which data is sensitive, which is not?
+ * Whether it is acceptable for instrumented data to be stored anywhere?
+ (to a local file, what not?)
+ 
+ * Data ownership and data collection consents are hard to accommodate from
+ LLVM developers' point of view:
+ 
+ * E.g., data collected by Telemetry is not necessarily owned by the user
+ of an LLVM tool with Telemetry enabled, hence the user's consent to data
+ collection is not meaningful. On the other hand, LLVM developers have no
+ reasonable ways to request consent from the "real" owners.
+
+
+High-level design
+=================
+
+Key components
+--------------
+
+The framework consists of four important classes:
+
+* ``llvm::telemetry::Manager``: The class responsible for collecting and
+ transmitting telemetry data. This is the main point of interaction between the
+ framework and any tool that wants to enable telemetry.
+* ``llvm::telemetry::TelemetryInfo``: Data courier
+* ``llvm::telemetry::Destination``: Data sink to which the Telemetry framework
+ sends data.
+ Its implementation is transparent to the framework.
+ It is up to the vendor to decide which pieces of data to forward and where
+ to forward them to for their final storage.
+* ``llvm::telemetry::Config``: Configurations for the ``Manager``.
+ 
+.. image:: llvm_telemetry_design.png
+
+How to implement and interact with the API
+------------------------------------------
+
+To use Telemetry in your tool, you need to provide a concrete implementation of the ``Manager`` class and ``Destination``.
+
+1) Define a custom ``Serializer``, ``Manager``, ``Destination`` and optionally a subclass of ``TelemetryInfo``
+
+.. code-block:: c++
+
+ class JsonSerializer : public Serializer {
+ public:
+ json::Object *getOutputObject() { return Out.get(); }
+
+ Error init() override {
+ if (Started)
+ return createStringError("Serializer already in use");
+ started = true;
+ Out = std::make_unique<json::Object>();
+ return Error::success();
+ }
+
+ // Serialize the given value.
+ void write(StringRef KeyName, bool Value) override {
+ writeHelper(KeyName, Value);
+ }
+
+ void write(StringRef KeyName, int Value) override {
+ writeHelper(KeyName, Value);
+ }
+
+ void write(StringRef KeyName, unsigned int Value) override {
+ writeHelper(KeyName, Value);
+ }
+
+ void write(StringRef KeyName, unsigned long Value) override {
+ writeHelper(KeyName, Value);
+ }
+
+ void write(StringRef KeyName, long Value) override {
+ writeHelper(KeyName, Value);
+ }
+
+ void write(StringRef KeyName, long long Value ) override {
+ writeHelper(KeyName, Value);
+ }
+
+ void write(StringRef KeyName, unsigned long long Value) override {
+ writeHelper(KeyName, Value);
+ }
+
+ void write(StringRef KeyName, StringRef Value) override {
+ writeHelper(KeyName, Value);
+ }
+ 
+ void beginObject(StringRef KeyName) override {
+ Children.push_back(json::Object());
+ ChildrenNames.push_back(KeyName.str());
+ }
+ 
+ void endObject() override {
+ assert(!Children.empty() && !ChildrenNames.empty());
+ json::Value Val = json::Value(std::move(Children.back()));
+ std::string Name = ChildrenNames.back();
+
+ Children.pop_back();
+ ChildrenNames.pop_back();
+ writeHelper(Name, std::move(Val));
+ }
+ 
+ Error finalize() override {
+ if (!Started)
+ return createStringError("Serializer not currently in use");
+ Started = false;
+ return Error::success();
+ }
+
+ private:
+ template <typename T> void writeHelper(StringRef Name, T Value) {
+ assert(Started && "serializer not started");
+ if (Children.empty())
+ Out->try_emplace(Name, Value);
+ else
+ Children.back().try_emplace(Name, Value);
+ }
+ bool Started = false;
+ std::unique_ptr<json::Object> Out;
+ std::vector<json::Object> Children;
+ std::vector<std::string> ChildrenNames;
+ };
+ 
+ class MyManager : public telemery::Manager {
+ public:
+ static std::unique_ptr<MyManager> createInstatnce(telemetry::Config *Config) {
+ // If Telemetry is not enabled, then just return null;
+ if (!Config->EnableTelemetry)
+ return nullptr;
+ return std::make_unique<MyManager>();
+ }
+ MyManager() = default;
+
+ Error preDispatch(TelemetryInfo *Entry) override {
+ Entry->SessionId = SessionId;
+ return Error::success();
+ }
+ 
+ // You can also define additional instrumentation points.
+ void logStartup(TelemetryInfo *Entry) {
+ // Add some additional data to entry.
+ Entry->Msg = "Some message";
+ dispatch(Entry);
+ }
+ 
+ void logAdditionalPoint(TelemetryInfo *Entry) {
+ // .... code here
+ }
+ 
+ private: 
+ const std::string SessionId;
+ };
+
+ class MyDestination : public telemetry::Destination {
+ public:
+ Error receiveEntry(const TelemetryInfo *Entry) override {
+ if (Error Err = Serializer.init())
+ return Err;
+ 
+ Entry->serialize(Serializer);
+ if (Error Err = Serializer.finalize())
+ return Err;
+ 
+ json::Object Copied = *Serializer.getOutputObject();
+ // Send the `Copied` object to wherever.
+ return Error::success();
+ }
+
+ private:
+ JsonSerializer Serializer;
+ };
+
+ // This defines a custom TelemetryInfo that has an additional Msg field.
+ struct MyTelemetryInfo : public telemetry::TelemetryInfo {
+ std::string Msg;
+ 
+ Error serialize(Serializer &Serializer) const override {
+ TelemetryInfo::serialize(serializer);
+ Serializer.writeString("MyMsg", Msg);
+ }
+ 
+ // Note: implement getKind() and classof() to support dyn_cast operations.
+ };
+
+ 
+2) Use the library in your tool.
+
+Logging the tool init-process:
+
+.. code-block:: c++
+
+ // In tool's initialization code.
+ auto StartTime = std::chrono::time_point<std::chrono::steady_clock>::now();
+ telemetry::Config MyConfig = makeConfig(); // Build up the appropriate Config struct here.
+ auto Manager = MyManager::createInstance(&MyConfig);
+
+ 
+ // Any other tool's init code can go here.
+ // ...
+ 
+ // Finally, take a snapshot of the time now so we know how long it took the
+ // init process to finish.
+ auto EndTime = std::chrono::time_point<std::chrono::steady_clock>::now();
+ MyTelemetryInfo Entry;
+
+ Entry.Start = StartTime;
+ Entry.End = EndTime;
+ Manager->logStartup(&Entry);
+
+Similar code can be used for logging the tool's exit.

llvm/docs/UserGuides.rst

@@ -72,6 +72,7 @@ intermediate LLVM representation.
  SupportLibrary
  TableGen/index
  TableGenFundamentals
+ Telemetry
  Vectorizers
  WritingAnLLVMPass
  WritingAnLLVMNewPMPass
@@ -293,3 +294,6 @@ Additional Topics
 
 :doc:`Sandbox IR <SandboxIR>`
  This document describes the design and usage of Sandbox IR, a transactional layer over LLVM IR.
+
+:doc:`Telemetry`
+ This document describes the Telemetry framework in LLVM.