Real-time Insights, powered by Reactive Programming

1.
Real-time Insights powered byReactive Programming Jay Phelps | @_jayphelps

2.
Jay Phelps |@_jayphelps Let's talk why

4.
Jay Phelps |@_jayphelps Time is money

5.
Jay Phelps |@_jayphelps 127+ million person-hours lost per year $500k — $1+ million per hour average hourly cost of critical failure https://devops.com/real-cost-downtime/ Fortune 500

6.
Jay Phelps |@_jayphelps Every second counts

7.
Jay Phelps |@_jayphelps Not just outages...

8.
Jay Phelps |@_jayphelps debugging, testing, and even Information Security too

9.
Jay Phelps |@_jayphelps improving Developer Experience

10.
Senior Software Engineer| @_jayphelps Jay Phelps

11.
Jay Phelps |@_jayphelps debugging, testing, and InfoSec

12.
Jay Phelps |@_jayphelps InfoSec “preventing unauthorized access, use, or disruption of information”

13.
Jay Phelps |@_jayphelps stopping hackers

14.
Jay Phelps |@_jayphelps InfoSec

15.
Jay Phelps |@_jayphelps

16.
Jay Phelps |@_jayphelps I'm on it.

17.
Jay Phelps |@_jayphelps We can block exploits using our gateway proxy

18.
Jay Phelps |@_jayphelps ...we need to know it's working...

19.
Jay Phelps |@_jayphelps ...we want to watch attackers try...

20.
Jay Phelps |@_jayphelps ...in real-time

21.
Jay Phelps |@_jayphelps We need real-time insights For debugging, testing, and InfoSec

22.
Jay Phelps |@_jayphelps Logging is the answer

23.
Jay Phelps |@_jayphelps LOG ALL THE THINGSLOG ALL THE THINGSLOG ALL THE THINGSLOG ALL THE THINGSLOG ALL THE THINGS

24.
Jay Phelps |@_jayphelps One little problem...

25.
Jay Phelps |@_jayphelps We have millions of devices...

26.
Jay Phelps |@_jayphelps We have thousands of servers...

27.
Jay Phelps |@_jayphelps We need to process them in real-time

28.
Jay Phelps |@_jayphelps “Netﬂix is a log generating company that happens to stream movies” - Adrian Cockroft

29.
Jay Phelps |@_jayphelps Massiveamount of streaming logs

30.
Jay Phelps |@_jayphelps Reactive Extensions

31.
Jay Phelps |@_jayphelps Rx

32.
Jay Phelps |@_jayphelps The best ideas from the Observer pattern, the Iterator pattern, and functional programming

33.
Jay Phelps |@_jayphelps “lodash for events”

34.

35.

36.
Jay Phelps |@_jayphelps High-level intro

37.
Jay Phelps |@_jayphelps .map(value => value * 10) .forEach(value => { console.log(value); }); [1, 2, 3]

38.

39.

40.

41.

42.
Jay Phelps |@_jayphelps .map(value => value * 10) .forEach(value => { console.log(value); }); [1, 2, 3] // 10

43.
Jay Phelps |@_jayphelps .map(value => value * 10) .forEach(value => { console.log(value); }); [1, 2, 3] // 10 // 20

44.
Jay Phelps |@_jayphelps .map(value => value * 10) .forEach(value => { console.log(value); }); [1, 2, 3] // 10 // 20 // 30

45.
Jay Phelps |@_jayphelps [1, 2, 3] // 10 // 20 // 30 .map(value => value * 10) .forEach(value => { console.log(value); });

46.
Jay Phelps |@_jayphelps [1, 2, 3] // 10 // 20 // 30 .map(value => value * 10) .forEach(value => { console.log(value); });

47.
Jay Phelps |@_jayphelps Observable.of(1, 2, 3) .map(value => value * 10) .forEach(value => { console.log(value); });

48.
Jay Phelps |@_jayphelps Observable.of(1, 2, 3) .map(value => value * 10) .forEach(value => { console.log(value); });

49.
Jay Phelps |@_jayphelps Observable.of(1, 2, 3) .map(value => value * 10) .subscribe(value => { console.log(value); });

50.
Jay Phelps |@_jayphelps Observable.of(1, 2, 3) .map(value => value * 10) .subscribe(value => { console.log(value); });

51.
Jay Phelps |@_jayphelps Observable.of(1, 2, 3) // 10 .map(value => value * 10) .subscribe(value => { console.log(value); });

52.
Jay Phelps |@_jayphelps Observable.of(1, 2, 3) // 10 // 20 .map(value => value * 10) .subscribe(value => { console.log(value); });

53.
Jay Phelps |@_jayphelps Observable.of(1, 2, 3) // 10 // 20 // 30 .map(value => value * 10) .subscribe(value => { console.log(value); });

54.
Jay Phelps |@_jayphelps Array is a collection

55.
Jay Phelps |@_jayphelps Observable is a collection that arrives over time

56.
Jay Phelps |@_jayphelps Represents a stream

57.
Jay Phelps |@_jayphelps button.addEventListener('click', event => { console.log('you clicked!'); });

58.
Jay Phelps |@_jayphelps Observable.fromEvent(button, 'click') .subscribe(event => { console.log('you clicked!'); });

59.
.subscribe(event => { console.log('you clicked!'); }); Jay Phelps |@_jayphelps Observable.fromEvent(button, 'click') .debounceTime(500)

60.
.subscribe(event => { console.log('you clicked!'); }); Jay Phelps |@_jayphelps Observable.fromEvent(button, 'click') .debounceTime(500)

61.
Jay Phelps |@_jayphelps Observable.fromEvent(button, 'click') .debounceTime(500)

62.

63.

64.

65.

66.
Observable.fromEvent(button, 'click') .debounceTime(500) Jay Phelps |@_jayphelps

67.

68.

69.

70.

71.

72.
Observable.fromEvent(button, 'click') .debounceTime(500) Jay Phelps |@_jayphelps 500 ms

73.

74.

75.

76.

77.

78.

79.

80.
Observable.fromEvent(button, 'click') .debounceTime(500) Jay Phelps |@_jayphelps 500 ms 500 ms

81.

82.

83.

84.
Observable.fromEvent(button, 'click') .debounceTime(500) Jay Phelps |@_jayphelps 500 ms 500 ms500 ms

85.

86.

87.

88.
Jay Phelps |@_jayphelps Stream of logs

89.
Jay Phelps |@_jayphelps { "path": "/something", "status": 500 } Log message in JSON

90.
Jay Phelps |@_jayphelps import { webSocket } from "rxjs/observable/webSocket"; let streamOfLogs = webSocket("ws://logs.netflix.com")

91.
.subscribe(msg => { console.log(msg); }); Jay Phelps |@_jayphelps streamOfLogs .filter(msg => msg.status !== 200)

92.
.subscribe(msg => { console.log(msg); }); Jay Phelps |@_jayphelps streamOfLogs .filter(msg => msg.status !== 200)

93.
500 200 404301 Jay Phelps | @_jayphelps 500 streamOfLogs .filter(msg => msg.status !== 200) 404

94.
Jay Phelps |@_jayphelps Learning Rx isn't easy

95.
Jay Phelps |@_jayphelps It's worth it!

96.
Jay Phelps |@_jayphelps It's worth it! Jay!!

97.
Jay Phelps |@_jayphelps Case Study Scaling high volume logs in real-time

98.
Jay Phelps |@_jayphelps We're going to log JSON

99.
Jay Phelps |@_jayphelps { "path": "/something", "status": 200 } Device { "event": "StartPlay", "device": "XBox 360" } Server

100.
Jay Phelps |@_jayphelps Stream them all through a single pipeline

101.
Jay Phelps |@_jayphelps How do you ﬁnd and process the logs you want

102.
Jay Phelps |@_jayphelps Write a custom job in Java?

103.
Jay Phelps |@_jayphelps Job: a unit of work

104.
Jay Phelps |@_jayphelps { "path": "/something", "status": 500, "duration": 135, // etc ... } Server log message

105.
Jay Phelps |@_jayphelps streamOfLogs .filter(event -> event.get("status") != 200) .map(event -> new JSONObject() .put("path", event.get("path")) .put("status", event.get("status")) );

106.
Jay Phelps |@_jayphelps Powerful, but... not ideal

107.
Jay Phelps |@_jayphelps Query Language

108.
Jay Phelps |@_jayphelps SELECT path,status WHERE status != 200

109.
Jay Phelps |@_jayphelps Just-in-time (JIT) Compilation

110.
Jay Phelps |@_jayphelps SELECT path,status,duration WHERE status != 200

111.
Jay Phelps |@_jayphelps SELECT path,status,duration WHERE status != 200

112.
Jay Phelps |@_jayphelps streamOfLogs .filter(event -> event.get("status") != 200) .map(event -> new JSONObject() .put("path", event.get("path")) .put("status", event.get("status")) ); SELECT path,status,duration WHERE status != 200

113.
Jay Phelps |@_jayphelps What's next?

114.
Jay Phelps |@_jayphelps 8+ million messages per second, peakmessagespersecond Time

115.
Jay Phelps |@_jayphelps How do we scale this?

116.
Jay Phelps |@_jayphelps Distribute work via autoscaling

117.
Jay Phelps |@_jayphelps A Source B Server B Server 50% 50% Load balancing a job

118.
Jay Phelps |@_jayphelps Chain jobs together

119.
Jay Phelps |@_jayphelps SELECT path,status WHERE source == "API" Chain jobs together

120.
Jay Phelps |@_jayphelps A Job B C Job Job Chain jobs together

121.
Jay Phelps |@_jayphelps High-volume, distributed systems have a problem...

122.
Jay Phelps |@_jayphelps Backpressure

123.
Jay Phelps |@_jayphelps “pressure opposed to the desired ﬂow of gases in conﬁned places such as a pipe” - wikipedia.org

124.
Jay Phelps |@_jayphelps A B C Server Server Server 100 rps 75 rps

125.
Jay Phelps |@_jayphelps A B C Server Server Server 60 sec * 25 rps = 1,500 rpm! 100 rps 75 rps

126.
Jay Phelps |@_jayphelps A B C Server Server Server 100 rps 75 rps 60 sec * 60 min * 25 rps = 90,000 rph!

127.
Jay Phelps |@_jayphelps Buffer or Drop

128.
Jay Phelps |@_jayphelps onBackpressureBuffer

129.
.onBackpressureBuffer() Jay Phelps |@_jayphelps streamOfLogs .subscribe(value -> { // something expensive Thread.sleep(100); });

130.

131.

132.
Jay Phelps |@_jayphelps 1 2 3 4 5 .onBackpressureBuffer() [3, 4, 5] 1 2

133.
Jay Phelps |@_jayphelps onBackpressureDrop

134.
.onBackpressureDrop() streamOfLogs .subscribe(value -> { // something expensive Thread.sleep(100); }); Jay Phelps |@_jayphelps

135.

136.

137.
Jay Phelps |@_jayphelps 1 2 3 4 5 .onBackpressureDrop() 1 4

138.
Jay Phelps |@_jayphelps Job authors choose which

139.
Jay Phelps |@_jayphelps 700+ jobs running 24/7 8+ million events/second Autoscaling Mantis Low-latency, high throughput stream-processing job platform http://techblog.netﬂix.com

140.
Jay Phelps |@_jayphelps We need somewhere to submit and view query results

141.
Jay Phelps |@_jayphelps Query Builder UI

142.
Jay Phelps |@_jayphelps SELECT path,status WHERE status != 200

143.

144.

145.

146.

147.

148.

153.
Jay Phelps |@_jayphelps Can be extremely high-volume 100k+ rps

154.
Jay Phelps |@_jayphelps We simply can’t render that fast!

155.
Jay Phelps |@_jayphelps Even if we could, it would be bad UX

156.
Jay Phelps |@_jayphelps Performance solutions are often driven by UX

157.
Jay Phelps |@_jayphelps So what do we do?

158.
Jay Phelps |@_jayphelps UI virtualization aka Virtual Table?

159.
VirtualizedNOT Virtualized vs.

160.
Jay Phelps |@_jayphelps We still can’t update that virtual table 100k per second

161.
Jay Phelps |@_jayphelps This is also backpressure

162.
Jay Phelps |@_jayphelps Buffer or Drop

163.
Jay Phelps |@_jayphelps Buffer for 1 second

164.
Jay Phelps |@_jayphelps getWebSocket() .bufferTime(1000)

165.
Jay Phelps |@_jayphelps Buffer size is unbounded

166.
Jay Phelps |@_jayphelps See what your users actually do

167.
Jay Phelps |@_jayphelps Users want a sample

168.
Jay Phelps |@_jayphelps Batch sampling

169.
Jay Phelps |@_jayphelps Drop after reaching a certain threshold

170.

171.

172.

173.

174.
Jay Phelps |@_jayphelps let buffer = getWebSocket() .bufferTime(1000); let gate = new BehaviorSubject(true); let batchSize = 50; let batchSizeCounter = 0; let results = gate$ .switchMap(enabled => enabled ? buffer : Observable.never()) .do(buffer => { batchSizeCounter += buffer.length; if (batchSizeCounter >= batchSize) { // truncates the array, if it's over batchSize buffer.length = batchSize; // turns on the gate, pausing the stream gate.next(false); batchSizeCounter = 0; } }); https://goo.gl/DMOqBA

175.

176.

177.

178.

179.

180.

181.
Jay Phelps |@_jayphelps Works for low-volume queries too

182.
Jay Phelps |@_jayphelps Raven

183.
Jay Phelps |@_jayphelps RavenSweet Christmas

184.
Jay Phelps |@_jayphelps improved debugging, testing, and InfoSec

185.
Jay Phelps |@_jayphelps Netﬂix loves Rx

186.
Jay Phelps |@_jayphelps Rx is powerful, cross-platform

187.
Thanks! @_jayphelps

Real-time Insights, powered by Reactive Programming

More Related Content

Viewers also liked

Similar to Real-time Insights, powered by Reactive Programming

More from Jay Phelps

Recently uploaded

Real-time Insights, powered by Reactive Programming