🪟🎉 Connector builder: Substream slicer and cartesian slicer (#20861)

* improve some types * improve further * clean up a bit more * refactor loading state * move loading state up * remove isLoading references * remove unused props and make fetch connector error work * remove special component for name * remove top level state for unifinished flows * start removing uiwidget * Update airbyte-webapp/src/views/Connector/ConnectorCard/ConnectorCard.module.scss Co-authored-by: Tim Roes <tim@airbyte.io> * remove undefined option for selected id * remove unused prop * fix types * remove uiwidget state * clean up * adjust comment * handle errors in a nice way * do not respect default on oneOf fields * rename to formblock * reduce re-renders * pass error to secure inputs * simplify and improve styling * align top * code review * remove comment * review comments * rename file * be strict about boolean values * add example * track form error in error boundary * review comments * handle unexpected cases better * enrich error with connector id * 🪟🎉 Add copy stream button (#20577) * add copy stream button * review comments * rename prop * 🪟🎉 Connector builder: Integrate connector form for test input (#20385) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * fix small stuff * add warning label * review comments * adjust translation Co-authored-by: lmossman <lake@airbyte.io> * use request_body_json instead of request_body_data * 🪟 🎨 Move `Add` button into the line of Connector Builder key value list fields (#20699) * move add button into line * add stories for empty with control, and content + control * change button name to Control * 🪟🎉 Connector builder: Allow defining inputs (#20431) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * handle stored form values that don't contain new fields properly * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * 🪟🎉 Connector builder authentication (#20645) * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * fix keys * 🪟🎉 Connector builder: Session token and oauth authentication (#20712) * session token and oauth authentication * fill in session token variable * typos * make sure validation error does not go away * 🪟🎉 Connector builder: Always validate inputs form (#20664) * validate user input outside of form * review comments Co-authored-by: lmossman <lake@airbyte.io> Co-authored-by: lmossman <lake@airbyte.io> * fix merge conflict with dropdown prop being renamed to control * [Connector Builder] Add paginator (#20698) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * handle stored form values that don't contain new fields properly * session token and oauth authentication * fill in session token variable * fix merge of default values * add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts * add cursor and primary key fields to ui * save * add page size and token option inputs * fixes after rebase * add pagination * fix pagination types * handle empty field_name better * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * typos * make sure validation error does not go away * make primary key and cursor optional, and reorder * save toggle group progress * fix style of toggle label * handle empty values better * fix page size/token option field validation and rendering * handle cursor pagination page size option correctly Co-authored-by: Joe Reuter <joe@airbyte.io> * [Connector Builder] Add stream slicer (#20748) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * handle stored form values that don't contain new fields properly * session token and oauth authentication * fill in session token variable * fix merge of default values * add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts * add cursor and primary key fields to ui * save * add page size and token option inputs * fixes after rebase * add pagination * fix pagination types * handle empty field_name better * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * typos * make sure validation error does not go away * make primary key and cursor optional, and reorder * save toggle group progress * fix style of toggle label * handle empty values better * fix page size/token option field validation and rendering * handle cursor pagination page size option correctly * save stream slicer progress * finish stream slicer * fix stream slicer fields and validation Co-authored-by: Joe Reuter <joe@airbyte.io> * debounce form builder values update to reduce load * 🪟🔧 Connector builder: use new lowcode manifest (#20715) * use new manifest yaml * Update airbyte-webapp/src/components/connectorBuilder/types.ts Co-authored-by: Lake Mossman <lake@airbyte.io> * use updated manifest types Co-authored-by: Lake Mossman <lake@airbyte.io> * add substream slicer * add substream and cartesian slicer * debounce validation as well * akways show stream test button in error state if there are errors * fix type of oauth input * add validation schema for add stream form * validate all views on test click * add type to prevent console warning * do not allow path for substream slicer request option * do not show request option for substream slicer * rewrite stream slice field tooltip Co-authored-by: Tim Roes <tim@airbyte.io> Co-authored-by: lmossman <lake@airbyte.io>

Author: 3 people

airbyte-webapp/package-lock.json

@@ -40,6 +40,7 @@
  "@sentry/tracing": "^6.19.6",
  "@tanstack/react-table": "^8.7.0",
  "@types/segment-analytics": "^0.0.34",
+ "@types/uuid": "^9.0.0",
  "classnames": "^2.3.1",
  "dayjs": "^1.11.3",
  "firebase": "^9.8.2",
@@ -80,6 +81,7 @@
  "styled-components": "^5.3.5",
  "typesafe-actions": "^5.1.0",
  "unist-util-visit": "^4.1.0",
+ "uuid": "^9.0.0",
  "yup": "^0.32.11"
  },
  "devDependencies": {

airbyte-webapp/package.json

@@ -3,6 +3,7 @@ import merge from "lodash/merge";
 import { useState } from "react";
 import React from "react";
 import { FormattedMessage, useIntl } from "react-intl";
+import { v4 as uuid } from "uuid";
 import * as yup from "yup";
 
 import { Button } from "components/ui/Button";
@@ -55,6 +56,7 @@ export const AddStreamButton: React.FC<AddStreamButtonProps> = ({ onAddStream, b
  ...initialValues,
  name: values.streamName,
  urlPath: values.urlPath,
+ id: uuid(),
  }),
  ]);
  setIsOpen(false);

airbyte-webapp/src/components/connectorBuilder/Builder/AddStreamButton.tsx

@@ -0,0 +1,13 @@
+@use "scss/variables";
+@use "scss/colors";
+
+$removeButtonWidth: 20px;
+
+.itemWrapper {
+ display: flex;
+ gap: variables.$spacing-md;
+}
+
+.itemContainer {
+ flex-grow: 1;
+}

airbyte-webapp/src/components/connectorBuilder/Builder/BuilderList.module.scss

@@ -0,0 +1,55 @@
+import { faPlus } from "@fortawesome/free-solid-svg-icons";
+import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
+import { useField } from "formik";
+import React, { ReactElement, useMemo } from "react";
+import { FormattedMessage } from "react-intl";
+
+import { Button } from "components/ui/Button";
+
+import styles from "./BuilderList.module.scss";
+import { RemoveButton } from "./RemoveButton";
+
+interface BuilderListProps {
+ children: (props: { buildPath: (path: string) => string }) => ReactElement;
+ basePath: string;
+ emptyItem: object;
+}
+
+export const BuilderList: React.FC<BuilderListProps> = ({ children, emptyItem, basePath }) => {
+ const [list, , helpers] = useField<object[]>(basePath);
+
+ const buildPathFunctions = useMemo(
+ () =>
+ new Array(list.value.length).fill(undefined).map((_value, index) => {
+ return (path: string) => `${basePath}[${index}]${path !== "" ? "." : ""}${path}`;
+ }),
+ [basePath, list.value.length]
+ );
+
+ return (
+ <>
+ {buildPathFunctions.map((buildPath, currentItemIndex) => (
+ <div className={styles.itemWrapper} key={currentItemIndex}>
+ <div className={styles.itemContainer}>{children({ buildPath })}</div>
+ <RemoveButton
+ onClick={() => {
+ const updatedItems = list.value.filter((_, index) => index !== currentItemIndex);
+ helpers.setValue(updatedItems);
+ }}
+ />
+ </div>
+ ))}
+ <div>
+ <Button
+ variant="secondary"
+ icon={<FontAwesomeIcon icon={faPlus} />}
+ onClick={() => {
+ helpers.setValue([...list.value, { ...emptyItem }]);
+ }}
+ >
+ <FormattedMessage id="connectorBuilder.addNewSlicer" />
+ </Button>
+ </div>
+ </>
+ );
+};

airbyte-webapp/src/components/connectorBuilder/Builder/BuilderList.tsx

@@ -11,7 +11,7 @@ interface Option {
  default?: object;
 }
 
-interface OneOfOption {
+export interface OneOfOption {
  label: string; // label shown in the dropdown menu
  typeValue: string; // value to set on the `type` field for this component - should match the oneOf type definition
  default?: object; // default values for the path

airbyte-webapp/src/components/connectorBuilder/Builder/BuilderOneOf.tsx

@@ -21,19 +21,3 @@ $removeButtonWidth: 20px;
 .kvLabel {
  color: colors.$grey-400;
 }
-
-.removeButton {
- border: none;
- background-color: transparent;
- color: colors.$grey;
- cursor: pointer;
- padding: 0;
- width: $removeButtonWidth;
- height: $removeButtonWidth;
- font-size: 18px;
- transition: variables.$transition;
-
- &:hover {
- color: colors.$red;
- }
-}

airbyte-webapp/src/components/connectorBuilder/Builder/KeyValueListField.module.scss

@@ -1,5 +1,3 @@
-import { faXmark } from "@fortawesome/free-solid-svg-icons";
-import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
 import { useField } from "formik";
 import { FormattedMessage } from "react-intl";
 
@@ -10,6 +8,7 @@ import { Input } from "components/ui/Input";
 import { Text } from "components/ui/Text";
 
 import styles from "./KeyValueListField.module.scss";
+import { RemoveButton } from "./RemoveButton";
 
 interface KeyValueInputProps {
  keyValue: [string, string];
@@ -32,9 +31,7 @@ const KeyValueInput: React.FC<KeyValueInputProps> = ({ keyValue, onChange, onRem
  </Text>
  <Input value={keyValue[1]} onChange={(e) => onChange([keyValue[0], e.target.value])} />
  </div>
- <button type="button" className={styles.removeButton} onClick={onRemove}>
- <FontAwesomeIcon icon={faXmark} size="1x" />
- </button>
+ <RemoveButton onClick={onRemove} />
  </div>
  );
 };

airbyte-webapp/src/components/connectorBuilder/Builder/KeyValueListField.tsx

@@ -0,0 +1,20 @@
+@use "scss/variables";
+@use "scss/colors";
+
+$removeButtonWidth: 20px;
+
+.removeButton {
+ border: none;
+ background-color: transparent;
+ color: colors.$grey;
+ cursor: pointer;
+ padding: 0;
+ width: $removeButtonWidth;
+ height: $removeButtonWidth;
+ font-size: 18px;
+ transition: variables.$transition;
+
+ &:hover {
+ color: colors.$red;
+ }
+}

airbyte-webapp/src/components/connectorBuilder/Builder/RemoveButton.module.scss

@@ -0,0 +1,12 @@
+import { faXmark } from "@fortawesome/free-solid-svg-icons";
+import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
+
+import styles from "./RemoveButton.module.scss";
+
+export const RemoveButton = ({ onClick }: { onClick: () => void }) => {
+ return (
+ <button className={styles.removeButton} onClick={onClick}>
+ <FontAwesomeIcon icon={faXmark} />
+ </button>
+ );
+};