You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -343,6 +363,7 @@ Examples of this include sendEmail, calculatePrice, etc.
343
363
- Start Time (string, optional): Indicates the begining time to start polling from (defaults to the begining of time)
344
364
- End Time (string, optional): If provided, don’t fetch records modified after this time (defaults to never)
345
365
- Size of Polling Page (optional; positive integer) Indicates the size of pages to be fetched. Defaults to 1000.
366
+
- Single Page per Interval (dropdown/checkbox: yes/no; default yes) Indicates that if the number of changed records exceeds the maximum number of results in a page, instead of fetching the next page immediately, wait until the next flow start to fetch the next page.
346
367
347
368
##### Input Metadata
348
369
@@ -357,12 +378,22 @@ N/A
357
378
snapshot.pageNumber = snapshot.pageNumber || 0;
358
379
let lastSeenTime = previousLastModified;
359
380
do {
381
+
let whereCondition;
382
+
if (previousLastModified === cfg.startTime || new Date(0)){
- If `previousLastModified` is set to `lastSeenTime` and we have `lastModified >= previousLastModified` then each execution will include records from previous execution. But if at the first execution `previousLastModified` could be equal `cfg.startTime` and we have `lastModified > previousLastModified` then we will lose objects whose last modified date is equal to the `cfg.startTime`. This is why we compare `previousLastModified` and `cfg.startTime || new Date(0)` and if they are equal, use condition `lastModified >= previousLastModified,` else: `lastModified > previousLastModified,`
425
+
- We use `lastModified <= maxTime` as it is more understandable for user.
426
+
- We have `Single Page per Interval` default to yes because it is slightly safer.
As an integrator who has multiple production flows which move large amounts of
4
+
data, I will get a large amount of errors. (E.g. if I have a flow that runs
5
+
every three minutes and has an error rate of 0.5%, then I would expect 2 errors
6
+
per day for that flow.) I want better tooling so that I can:
7
+
8
+
* Fix data records which are not in the expected state due a flow encountering an error before completion
9
+
* Understand if a flow is broken or if errors are simply occurring at an expected rate
10
+
* Triage errors so that their root cause can be addressed
11
+
* Be alerted to any possible misconfiguration of my flows or their related systems
12
+
13
+
Below are some possible mechanisms that can be used to achieve these goals.
14
+
15
+
# Possible Error Features
16
+
## Manual Replay on Error
17
+
*(Not particularly applicable to Request-Reply flows)*
18
+
If an error occurs during the execution, it would be nice if an integrator could
19
+
edit the input message and then click a button to resubmit the error message to
20
+
that point in the flow or opt to dismiss the error message.
21
+
22
+
## Group by error view
23
+
If it would be possible to do group by's on reported errors based on
24
+
flow/step/account/stack trace/error message and provide a frequency count of
25
+
each error, an integrator would have a better view of what is happening in the
26
+
system.
27
+
28
+
## Auto-retry
29
+
*(Not particularly applicable to Request-Reply flows)*
30
+
If a component throws an exception (and the component can tell the
31
+
platform/sailor that retry-ing this input may be successful) then sailor/the
32
+
platform should automatically invoke the `rebound` case without requiring the
33
+
component developer to call rebound.
34
+
35
+
## On-Error Action for Flow Component
36
+
If an error happens somewhere in a flow and the user wants to have that error
37
+
reporting/recovery partially or fully handled by an external system in an
38
+
automated way, we could add an 'On Error' step for a flow which is a component
39
+
which is invoked and passed the error information for any errors which happen
40
+
from any step in the flow.
41
+
42
+
## Track Last Execution Date
43
+
Allow integrators to set a "Alert me if not run in {Some Time Span}" setting on
44
+
a flow. Send an alert if time span elapses without any executions.
45
+
46
+
# Triage on error type & Semi-automated error recovering
47
+
Depending on the root cause of the error (e.g. authentication error vs system
48
+
error vs business validation error) a corporate customer may want different
49
+
individuals to be responsible for managing different errors. If the component
50
+
developer has the ability to categorize the errors the component emits and the
51
+
platform can respect that categorization, then the platform can put different
52
+
errors into different manual error resolution queues to be picked up by
53
+
different individuals at an organization.
54
+
55
+
Consider the list of possible error causes and a (suggested) way that the platform could respond to that error:
56
+
57
+
## Authentication Error
58
+
(A previously valid account starts throwing authentication errors when trying to
59
+
access it because a token has expired, a password changed, a system user
60
+
deleted, etc.)
61
+
1. Platform starts sending error alerts very loudly so that they are not missed by individuals responsible for monitoring integrations.
62
+
1. All messages arriving at any step in any non request-reply flow are stored.
63
+
1. Once a user has fixed the error-ed account, the user can click a button to process failed messages.
64
+
1. The platform will then fees the previously held messages into the previously paused steps.
65
+
66
+
## Business Validation Error
67
+
(A step emits error because the API rejects the request with a `400`-like error
68
+
because the incoming message does not conform to some business rule enforced by
69
+
the system.)
70
+
1. Place message into business validation resolution queue for manual correction before being retried.
71
+
72
+
## System Error
73
+
(A step emits error because the API returns with a `500` response code, timeout or failure to connect.)
74
+
1. Automatically start retrying.
75
+
1. If this failure is observed for all (or a high %) of interactions with this system/account (across multiple input messages), flag the system is down.
76
+
1. Platform starts sending error alerts very loudly so that they are not missed by individuals responsible for monitoring integrations.
77
+
1. All messages arriving at any step in any non request-reply flow are stored.
78
+
1. Once a user has fixed the error-ed account, the user can click a button to process failed messages.
79
+
1. The platform will then fees the previously held messages into the previously paused steps.
80
+
**Aside: One could build an additional mechanism into the platform to track external system health.**
81
+
82
+
## Dev Bug
83
+
(A step emits an error that is not caught by the developer (e.g. Null pointer
84
+
exception, maybe out of memory) or explicitly thrown by the developer (e.g. enum
85
+
in invalid state))
86
+
1. Platform sends stacktrace to component developer via email or bug tracking system so that component developer is aware of the problem and can fix the bug.
87
+
88
+
## API starvation error
89
+
(A component fails to complete an API call because the API call limit has been reached.)
90
+
1. Component tells the platform when more API calls will be available.
91
+
1. Platform retries at that time.
92
+
1. Number of such starvation errors for an account is stored. If this number is increasing, then the platform should alert the integrator.
93
+
**Aside: One could build an additional mechanism into the platform to track API call limit usage.**
94
+
95
+
## Emit Warning
96
+
(A component receives some warnings from some API that they are calling.)
97
+
1. Component passes warning to platform.
98
+
1. Platform places warning in message queue for the user so that the user can resolve or dismiss the warning.
99
+
100
+
## Incoming message too large
101
+
(Courtesy of @zubairov : a component with 512 MB RAM emits a 300 MB message which can not be loaded into the following component with only 256 MB RAM.)
0 commit comments