Feature: Ability to Export Dashboard Data

[Arc676]

Tell us more.

Motivation

Suppose you have a large number of interdependent repositories with dependencies kept up to date by Renovate. Each repository has a Dependency Dashboard that shows what each individual application depends on, but there is no central data source from which you can see how the repositories are all connected. One possible solution is to use Renovate's postUpgradeTasks; after each update, store the dependency information and upload it to some central repository. While functional, this is comparatively inefficient because an action has to be taken for each individual dependency. Also, the central repository would only be updated when Renovate commits changes. If you want to record version information in the central repository, it will always be either ahead of or behind the true state of the applications (unless you always automerge Renovate's changes).

In fact, the Dependency Dashboard is the ideal data source for building this central repository, since it contains all of a repository's dependencies in one place. Thus, the feature request: make it possible to configure additional behavior after creating the dashboard such that this data can be exported somehow (see also #Alternative Solutions at the end of this post).

Concrete Example

My own end goal here is to build a dependency graph for a large number of repositories. The dependency graph I'm building is accessible via REST API, so the approaches mentioned above would be curl in the postUpgradeTasks or fetch in Renovate itself. I've included the patch for my specific use case below to illustrate the relevant change to Renovate. For general use, the exact behavior would have to be configurable in some way. It should also be noted that this feature could be implemented with more isolated changes, but I wanted to avoid copying the dependency list multiple times.

diff --git a/lib/workers/repository/dependency-dashboard.ts b/lib/workers/repository/dependency-dashboard.ts
index 17d895c35..0e345520d 100644
--- a/lib/workers/repository/dependency-dashboard.ts
+++ b/lib/workers/repository/dependency-dashboard.ts
@@ -422,12 +422,22 @@ export async function ensureDependencyDashboard(
 
   // fit the detected dependencies section
   const footer = getFooter(config);
-  issueBody += PackageFiles.getDashboardMarkdown(
+  let [body, deps] = PackageFiles.getDashboardMarkdown(
     GitHubMaxPrBodyLen - issueBody.length - footer.length,
   );
 
+  issueBody += body;
   issueBody += footer;
 
+  await fetch("http://storage.domain.tld/path/to/endpoint", {
+    method: "POST",
+    headers: {"Content-Type": "application/json"},
+    body: JSON.stringify({
+      App: config.repository,
+      Dependencies: deps
+    })
+  });
+
   if (config.dependencyDashboardIssue) {
     // If we're not changing the dashboard issue then we can skip checking if the user changed it
     // The cached issue we get back here will reflect its state at the _start_ of our run
diff --git a/lib/workers/repository/package-files.ts b/lib/workers/repository/package-files.ts
index 73614ef88..e727e0e17 100644
--- a/lib/workers/repository/package-files.ts
+++ b/lib/workers/repository/package-files.ts
@@ -31,7 +31,7 @@ export class PackageFiles {
    * @param maxLength
    * @param setHeader
    */
-  static getDashboardMarkdown(maxLength: number, setHeader = true): string {
+  static getDashboardMarkdown(maxLength: number, setHeader = true): [string, Object[]] {
     const note =
       '> ℹ **Note**\n> \n> Detected dependencies section has been truncated\n\n';
     const title = `## Detected dependencies\n\n`;
@@ -41,6 +41,7 @@ export class PackageFiles {
     const mdMaxLength = maxLength - maxHeaderLen;
 
     let md: string;
+    let deps: Object[] = [];
     let header = '';
     let removed = false;
     let truncated = false;
@@ -58,7 +59,11 @@ export class PackageFiles {
 
     do {
       // shorten markdown until it fits
-      md = PackageFiles.getDashboardMarkdownInternal(data);
+      let d: Object[];
+      [md, d] = PackageFiles.getDashboardMarkdownInternal(data, !removed);
+      if (!removed) {
+        deps = d;
+      }
       if (md.length > mdMaxLength) {
         // truncate data
         removed = PackageFiles.pop(data);
@@ -71,7 +76,7 @@ export class PackageFiles {
     header += title;
     header += truncated ? note : '';
 
-    return (setHeader ? header : '') + md;
+    return [(setHeader ? header : '') + md, deps];
   }
 
   /**
@@ -80,10 +85,12 @@ export class PackageFiles {
    */
   private static getDashboardMarkdownInternal(
     data: Map<string, Record<string, PackageFile[]> | null>,
-  ): string {
+    buildDepArray: boolean = false,
+  ): [string, Object[]] {
     const none = 'None detected\n\n';
     const pad = data.size > 1; // padding condition for a multi base branch repo
     let deps = '';
+    var depArray: Object[] = []
 
     for (const [branch, packageFiles] of data) {
       deps += pad
@@ -113,6 +120,14 @@ export class PackageFiles {
               ver && digest ? `${ver}@${digest}` : `${digest ?? ver!}`;
             // TODO: types (#22198)
             deps += ` - \`${dep.depName!} ${version}\`\n`;
+            if (buildDepArray) {
+              depArray.push({
+                Name: dep.depName,
+                Version: dep.currentValue,
+                Manager: manager,
+                PackageFile: packageFile.packageFile
+              });
+            }
           }
           deps += '\n</details>\n\n';
         }
@@ -121,7 +136,7 @@ export class PackageFiles {
       deps += pad ? '</blockquote>\n</details>\n\n' : '';
     }
 
-    return deps;
+    return [deps, depArray];
   }
 
   /**

Alternative Solutions

It is not strictly necessary for Renovate itself to upload the dashboard data. The end goal is also achievable by adding an option to have Renovate dump the Dependency Dashboard (or, preferably, a JSON-encoded list of the dependencies as constructed in the patch above) to disk once it's done processing each individual repository. The key difference from postUpgradeTasks is providing access to the metadata of all the dependencies at once, as well as running strictly once per repository only.

While this feature could just be implemented in a fork, that would result in a separate codebase to maintain and keep up to date with Renovate's upstream changes, which isn't ideal.

It must also be acknowledged that this could be done without modifying Renovate by just writing another crawler and searching for the Dependency Dashboard in each repository, but this would mean visiting all the repositories twice. In particular, the crawler would have to query the dashboards even if Renovate didn't find any outdated dependencies, since it has no way of knowing which repositories had their dependencies updated. The goal of this discussion is to probe the interest in having this feature (or something similar) built directly into Renovate itself so that the end goal can be achieved more efficiently.

[secustor]

I think you are looking for this: https://docs.renovatebot.com/self-hosted-configuration/#reporttype

[Arc676]

Thanks! I was not aware of this configuration option, it looks perfect for my use case. Is there some more extensive documentation on the report? Even if not, looking at the report does explain most things, but just to be sure: Renovate generates this report at the very end, right? I see it includes the repository name, so if one instance of Renovate checks multiple repositories, it only generates one report at the very end that contains all the information, correct?

[secustor]

There is not much documtation as this feature is experimental and we give no guarantees that the report format not changes.
That is because it mostly put out internal data formats.

Yes, these reports are per run and NOT per repository. That being said different repositories repositories are under different keys, so you will be able to differentiate the state of different repos.

[Arc676]

Great, thanks. I'll keep an eye on the changelog for the format then. Although even if the format changes, as long as the same information is still available that's fine.