In my experience with JavaScript/Node.js devs, I find out that Streams is a topic still mostly unexplored and sometimes even feared. This is probably one of the most beautiful features of Node.js and it can be adopted also in the browser. Once you grasp the fundamentals, you'll be able to solve some ordinary programming challenges in a much more elegant and efficient way, so this is a very interesting topic. In this talk, I will cover the following topics: Streams: when and how; Different types of streams; Built-in and custom streams; Composability; Stream utils; Streams in the browser.

1. Luciano Mammino (@loige)
IT’S ABOUT TIME TOIT’S ABOUT TIME TO
EMBRACE STREAMSEMBRACE STREAMS
loige.link/streams-cityjs
London - May 3, 2019
1

2. // buffer-copy.js
const {
readFileSync,
writeFileSync
} = require('fs')
const [,, src, dest] = process.argv
// read entire file content
const content = readFileSync(src)
// write that content somewhere else
writeFileSync(dest, content)
@loige2

3. // buffer-copy.js
const {
readFileSync,
writeFileSync
} = require('fs')
const [,, src, dest] = process.argv
// read entire file content
const content = readFileSync(src)
// write that content somewhere else
writeFileSync(dest, content)
@loige2

4. // buffer-copy.js
const {
readFileSync,
writeFileSync
} = require('fs')
const [,, src, dest] = process.argv
// read entire file content
const content = readFileSync(src)
// write that content somewhere else
writeFileSync(dest, content)
@loige2

5. // buffer-copy.js
const {
readFileSync,
writeFileSync
} = require('fs')
const [,, src, dest] = process.argv
// read entire file content
const content = readFileSync(src)
// write that content somewhere else
writeFileSync(dest, content)
@loige2

6. @loige3

7. WE DO THIS ALL THE TIMEWE DO THIS ALL THE TIME
@loige3

8. WE DO THIS ALL THE TIMEWE DO THIS ALL THE TIME
AND IT'S OKAND IT'S OK
@loige3

9. WE DO THIS ALL THE TIMEWE DO THIS ALL THE TIME
AND IT'S OKAND IT'S OK
BUT SOMETIMES ...BUT SOMETIMES ...
@loige3

10. @loige
ERR_FS_FILE_TOO_LARGE! ERR_FS_FILE_TOO_LARGE!
File size is greater than possible Buffer
4

11. BUT WHY?BUT WHY?
@loige5

12. IF BYTES IF BYTES WEREWERE BLOCKS... BLOCKS...@loige
6

13. MARIO CAN LIFTMARIO CAN LIFT
FEW BLOCKSFEW BLOCKS
@loige
7

14. BUT NOT TOO MANY...BUT NOT TOO MANY...@loige
?!
8

15. WHAT CAN WE DO IF WE HAVE TOWHAT CAN WE DO IF WE HAVE TO
MOVE MANY BLOCKS?MOVE MANY BLOCKS?
@loige9

16. WE CAN MOVE THEM ONE BY ONE!WE CAN MOVE THEM ONE BY ONE!
@loige
we stream them...
10

17. 11

18. HELLO, I AM LUCIANO! HELLO, I AM LUCIANO!
11

19. HELLO, I AM LUCIANO! HELLO, I AM LUCIANO!
11

20. HELLO, I AM LUCIANO! HELLO, I AM LUCIANO!
11

21. HELLO, I AM LUCIANO! HELLO, I AM LUCIANO!
11

22. HELLO, I AM LUCIANO! HELLO, I AM LUCIANO!
Cloud Architect
11

23. HELLO, I AM LUCIANO! HELLO, I AM LUCIANO!
Cloud Architect
Blog:
Twitter:
GitHub:
loige.co
@loige
@lmammino
11

24. HELLO, I AM LUCIANO! HELLO, I AM LUCIANO!
Cloud Architect
Blog:
Twitter:
GitHub:
loige.co
@loige
@lmammino
11

25. code: loige.link/streams-examples
loige.link/streams-cityjs
12

26. 01. BUFFERS VS01. BUFFERS VS
STREAMS STREAMS
@loige13

27. BUFFERBUFFER: DATA STRUCTURE TO STORE AND: DATA STRUCTURE TO STORE AND
TRANSFER ARBITRARY BINARY DATATRANSFER ARBITRARY BINARY DATA
@loige
*Note that this is loading all the content of the file in memory
*
14

28. STREAMSTREAM: ABSTRACT INTERFACE FOR: ABSTRACT INTERFACE FOR
WORKING WITH STREAMING DATAWORKING WITH STREAMING DATA
@loige
*It does not load all the data straight away
*
15

29. FILE COPY: FILE COPY: THE BUFFER WAYTHE BUFFER WAY
@loige
// buffer-copy.js
const {
readFileSync,
writeFileSync
} = require('fs')
const [,, src, dest] = process.argv
const content = readFileSync(src)
writeFileSync(dest, content)
16

30. FILE COPY: FILE COPY: THE STREAM WAYTHE STREAM WAY
// stream-copy.js
const {
createReadStream,
createWriteStream
} = require('fs')
const [,, src, dest] = process.argv
const srcStream = createReadStream(src)
const destStream = createWriteStream(dest)
srcStream.on('data', (data) => destStream.write(data))
@loige
* Careful: this implementation is not optimal
*
17

31. FILE COPY: FILE COPY: THE STREAM WAYTHE STREAM WAY
// stream-copy.js
const {
createReadStream,
createWriteStream
} = require('fs')
const [,, src, dest] = process.argv
const srcStream = createReadStream(src)
const destStream = createWriteStream(dest)
srcStream.on('data', (data) => destStream.write(data))
@loige
* Careful: this implementation is not optimal
*
17

32. FILE COPY: FILE COPY: THE STREAM WAYTHE STREAM WAY
// stream-copy.js
const {
createReadStream,
createWriteStream
} = require('fs')
const [,, src, dest] = process.argv
const srcStream = createReadStream(src)
const destStream = createWriteStream(dest)
srcStream.on('data', (data) => destStream.write(data))
@loige
* Careful: this implementation is not optimal
*
17

33. FILE COPY: FILE COPY: THE STREAM WAYTHE STREAM WAY
// stream-copy.js
const {
createReadStream,
createWriteStream
} = require('fs')
const [,, src, dest] = process.argv
const srcStream = createReadStream(src)
const destStream = createWriteStream(dest)
srcStream.on('data', (data) => destStream.write(data))
@loige
* Careful: this implementation is not optimal
*
17

34. MEMORY COMPARISON (~600MB FILE)MEMORY COMPARISON (~600MB FILE)
node ­­inspect­brk buffer­copy.js assets/poster.psd ~/Downloads/poster.psd
@loige18

35. MEMORY COMPARISON (~600MB FILE)MEMORY COMPARISON (~600MB FILE)
node ­­inspect­brk stream­copy.js assets/poster.psd ~/Downloads/poster.psd
@loige19

36. LET'S TRY WITH A BIG FILE (~10GB)LET'S TRY WITH A BIG FILE (~10GB)
node ­­inspect­brk stream­copy.js assets/the­matrix­hd.mkv ~/Downloads/the­matrix­hd.mkv
@loige20

37. STREAMS VS BUFFERS STREAMS VS BUFFERS
Streams keep a low memory footprint
even with large amounts of data
Streams allows you to process data as
soon as it arrives
@loige21

38. 03. STREAM TYPES03. STREAM TYPES
& APIS & APIS
@loige22

39. ALL STREAMS ARE ALL STREAMS ARE EVENT EMITTERSEVENT EMITTERS
A stream instance is an object that emits events when its internal
state changes, for instance:
s.on('readable', () => {}) // ready to be consumed
s.on('data', (chunk) => {}) // new data is available
s.on('error', (err) => {}) // some error happened
s.on('end', () => {}) // no more data available
The events available depend from the type of stream
@loige23

40. READABLEREADABLE STREAMS STREAMS
A readable stream represents a source from which data is consumed.
Examples:
fs readStream
process.stdin
HTTP response (client-side)
HTTP request (server-side)
AWS S3 GetObject (data field)
It supports two modes for data consumption: flowing and paused (or non-
flowing) mode.
@loige24

41. READABLE STREAMSREADABLE STREAMS
Data is read from source automatically and chunks are emitted as soon
as they are available.
@loige25

42. @loige
1
2
3
Source data
Readable stream in
flowing mode
data listener
READABLE STREAMSREADABLE STREAMS
Data is read from source automatically and chunks are emitted as soon
as they are available.
26

43. @loige
12
3
Source data
Readable stream in
flowing mode
Read
data listener
READABLE STREAMSREADABLE STREAMS
Data is read from source automatically and chunks are emitted as soon
as they are available.
27

44. @loige
12
3
Source data
Readable stream in
flowing mode
data listener
data
READABLE STREAMSREADABLE STREAMS
Data is read from source automatically and chunks are emitted as soon
as they are available.
28

45. @loige
2
3
Source data
Readable stream in
flowing mode
data listener
Read
READABLE STREAMSREADABLE STREAMS
Data is read from source automatically and chunks are emitted as soon
as they are available.
29

46. @loige
2
3
Source data
Readable stream in
flowing mode
data listener
data
READABLE STREAMSREADABLE STREAMS
Data is read from source automatically and chunks are emitted as soon
as they are available.
30

47. @loige
3
Source data
Readable stream in
flowing mode
data listener
Read
READABLE STREAMSREADABLE STREAMS
Data is read from source automatically and chunks are emitted as soon
as they are available.
31

48. @loige
3
Source data
Readable stream in
flowing mode
data listener
data
READABLE STREAMSREADABLE STREAMS
Data is read from source automatically and chunks are emitted as soon
as they are available.
32

49. @loige
Source data
Readable stream in
flowing mode
Read
data listener
(end)
READABLE STREAMSREADABLE STREAMS
Data is read from source automatically and chunks are emitted as soon
as they are available.
33

50. @loige
Source data
Readable stream in
flowing mode
data listener
end
(end)
When no more data is available, end is emitted.
READABLE STREAMSREADABLE STREAMS
Data is read from source automatically and chunks are emitted as soon
as they are available.
34

51. // count-emojis-flowing.js
const { createReadStream } = require('fs')
const { EMOJI_MAP } = require('emoji') // from npm
const emojis = Object.keys(EMOJI_MAP)
const file = createReadStream(process.argv[2])
let counter = 0
file.on('data', chunk => {
for (let char of chunk.toString('utf8')) {
if (emojis.includes(char)) {
counter++
}
}
})
file.on('end', () => console.log(`Found ${counter} emojis`))
file.on('error', err => console.error(`Error reading file: ${err}`))
@loige35

52. // count-emojis-flowing.js
const { createReadStream } = require('fs')
const { EMOJI_MAP } = require('emoji') // from npm
const emojis = Object.keys(EMOJI_MAP)
const file = createReadStream(process.argv[2])
let counter = 0
file.on('data', chunk => {
for (let char of chunk.toString('utf8')) {
if (emojis.includes(char)) {
counter++
}
}
})
file.on('end', () => console.log(`Found ${counter} emojis`))
file.on('error', err => console.error(`Error reading file: ${err}`))
@loige35

53. loige.link/st_patrick
@loige36

54. WRITABLEWRITABLE STREAMS STREAMS
A writable stream is an abstraction that allows to write data over a destination
Examples:
fs writeStream
process.stdout, process.stderr
HTTP request (client-side)
HTTP response (server-side)
AWS S3 PutObject (body parameter)
@loige37

55. // writable-http-request.js
const http = require('http')
const req = http.request(
{
hostname: 'enx6b07hdu6cs.x.pipedream.net',
method: 'POST'
},
resp => {
console.log(`Server responded with "${resp.statusCode}"`)
}
)
req.on('finish', () => console.log('request sent'))
req.on('close', () => console.log('Connection closed'))
req.on('error', err => console.error(`Request failed: ${err}`))
req.write('writing some content...n')
req.end('last write & close the stream')
@loige38

56. // writable-http-request.js
const http = require('http')
const req = http.request(
{
hostname: 'enx6b07hdu6cs.x.pipedream.net',
method: 'POST'
},
resp => {
console.log(`Server responded with "${resp.statusCode}"`)
}
)
req.on('finish', () => console.log('request sent'))
req.on('close', () => console.log('Connection closed'))
req.on('error', err => console.error(`Request failed: ${err}`))
req.write('writing some content...n')
req.end('last write & close the stream')
@loige38

57. // writable-http-request.js
const http = require('http')
const req = http.request(
{
hostname: 'enx6b07hdu6cs.x.pipedream.net',
method: 'POST'
},
resp => {
console.log(`Server responded with "${resp.statusCode}"`)
}
)
req.on('finish', () => console.log('request sent'))
req.on('close', () => console.log('Connection closed'))
req.on('error', err => console.error(`Request failed: ${err}`))
req.write('writing some content...n')
req.end('last write & close the stream')
@loige38

58. // writable-http-request.js
const http = require('http')
const req = http.request(
{
hostname: 'enx6b07hdu6cs.x.pipedream.net',
method: 'POST'
},
resp => {
console.log(`Server responded with "${resp.statusCode}"`)
}
)
req.on('finish', () => console.log('request sent'))
req.on('close', () => console.log('Connection closed'))
req.on('error', err => console.error(`Request failed: ${err}`))
req.write('writing some content...n')
req.end('last write & close the stream')
@loige38

59. @loige39

60. loige.link/writable-http-req
@loige40

61. BACKPRESSUREBACKPRESSURE
When writing large amounts of data you
should make sure you handle the stop write
signal and the drain event
loige.link/backpressure
@loige41

62. // stream-copy-safe.js
const { createReadStream, createWriteStream } = require('fs')
const [, , src, dest] = process.argv
const srcStream = createReadStream(src)
const destStream = createWriteStream(dest)
srcStream.on('data', data => {
const canContinue = destStream.write(data)
if (!canContinue) {
// we are overflowing the destination, we should pause
srcStream.pause()
// we will resume when the destination stream is drained
destStream.once('drain', () => srcStream.resume())
}
})
@loige42

63. // stream-copy-safe.js
const { createReadStream, createWriteStream } = require('fs')
const [, , src, dest] = process.argv
const srcStream = createReadStream(src)
const destStream = createWriteStream(dest)
srcStream.on('data', data => {
const canContinue = destStream.write(data)
if (!canContinue) {
// we are overflowing the destination, we should pause
srcStream.pause()
// we will resume when the destination stream is drained
destStream.once('drain', () => srcStream.resume())
}
})
@loige42

64. // stream-copy-safe.js
const { createReadStream, createWriteStream } = require('fs')
const [, , src, dest] = process.argv
const srcStream = createReadStream(src)
const destStream = createWriteStream(dest)
srcStream.on('data', data => {
const canContinue = destStream.write(data)
if (!canContinue) {
// we are overflowing the destination, we should pause
srcStream.pause()
// we will resume when the destination stream is drained
destStream.once('drain', () => srcStream.resume())
}
})
@loige42

65. // stream-copy-safe.js
const { createReadStream, createWriteStream } = require('fs')
const [, , src, dest] = process.argv
const srcStream = createReadStream(src)
const destStream = createWriteStream(dest)
srcStream.on('data', data => {
const canContinue = destStream.write(data)
if (!canContinue) {
// we are overflowing the destination, we should pause
srcStream.pause()
// we will resume when the destination stream is drained
destStream.once('drain', () => srcStream.resume())
}
})
@loige42

66. // stream-copy-safe.js
const { createReadStream, createWriteStream } = require('fs')
const [, , src, dest] = process.argv
const srcStream = createReadStream(src)
const destStream = createWriteStream(dest)
srcStream.on('data', data => {
const canContinue = destStream.write(data)
if (!canContinue) {
// we are overflowing the destination, we should pause
srcStream.pause()
// we will resume when the destination stream is drained
destStream.once('drain', () => srcStream.resume())
}
})
@loige42

67. OTHER TYPES OF STREAMOTHER TYPES OF STREAM
Duplex Stream
streams that are both Readable and Writable.
(net.Socket)
Transform Stream
Duplex streams that can modify or transform the data as it is written
and read.
(zlib.createGzip(), crypto.createCipheriv())
@loige43

68. ANATOMY OF A TRANSFORM STREAMANATOMY OF A TRANSFORM STREAM
1. write data
transform stream
3. read transformed data2. transform the data
(readable stream) (writable stream)
@loige44

69. GZIP EXAMPLEGZIP EXAMPLE
1. write data
transform stream
3. read transformed data2. transform the data
(readable stream) (writable stream)
@loige
Uncompressed data Compressed data
compress
zlib.createGzip()
45

70. gzipStream.on('data', data => {
const canContinue = destStream.write(data)
if (!canContinue) {
gzipStream.pause()
destStream.once('drain', () => {
gzipStream.resume()
})
}
})
gzipStream.on('end', () => {
destStream.end()
})
// ⚠ TODO: handle errors!
// stream-copy-gzip.js
const {
createReadStream,
createWriteStream
} = require('fs')
const { createGzip } = require('zlib')
const [, , src, dest] = process.argv
const srcStream = createReadStream(src)
const gzipStream = createGzip()
const destStream = createWriteStream(dest)
srcStream.on('data', data => {
const canContinue = gzipStream.write(data)
if (!canContinue) {
srcStream.pause()
gzipStream.once('drain', () => {
srcStream.resume()
})
}
})
srcStream.on('end', () => {
// check if there's buffered data left
const remainingData = gzipStream.read()
if (remainingData !== null) {
destStream.write()
}
gzipStream.end()
})
@loige46

71. gzipStream.on('data', data => {
const canContinue = destStream.write(data)
if (!canContinue) {
gzipStream.pause()
destStream.once('drain', () => {
gzipStream.resume()
})
}
})
gzipStream.on('end', () => {
destStream.end()
})
// ⚠ TODO: handle errors!
// stream-copy-gzip.js
const {
createReadStream,
createWriteStream
} = require('fs')
const { createGzip } = require('zlib')
const [, , src, dest] = process.argv
const srcStream = createReadStream(src)
const gzipStream = createGzip()
const destStream = createWriteStream(dest)
srcStream.on('data', data => {
const canContinue = gzipStream.write(data)
if (!canContinue) {
srcStream.pause()
gzipStream.once('drain', () => {
srcStream.resume()
})
}
})
srcStream.on('end', () => {
// check if there's buffered data left
const remainingData = gzipStream.read()
if (remainingData !== null) {
destStream.write()
}
gzipStream.end()
})
@loige46

72. 03. PIPE()03. PIPE()
@loige47

73. readable
.pipe(tranform1)
.pipe(transform2)
.pipe(transform3)
.pipe(writable)
readable.pipe(writableDest)
@loige
Connects a readable stream to a writable stream
A transform stream can be used as a destination as well
It returns the destination stream allowing for a chain of pipes
48

74. // stream-copy-gzip-pipe.js
const {
createReadStream,
createWriteStream
} = require('fs')
const { createGzip } = require('zlib')
const [, , src, dest] = process.argv
const srcStream = createReadStream(src)
const gzipStream = createGzip()
const destStream = createWriteStream(dest)
srcStream
.pipe(gzipStream)
.pipe(destStream)
@loige49

75. // stream-copy-gzip-pipe.js
const {
createReadStream,
createWriteStream
} = require('fs')
const { createGzip } = require('zlib')
const [, , src, dest] = process.argv
const srcStream = createReadStream(src)
const gzipStream = createGzip()
const destStream = createWriteStream(dest)
srcStream
.pipe(gzipStream)
.pipe(destStream)
@loige49

76. // stream-copy-gzip-pipe.js
const {
createReadStream,
createWriteStream
} = require('fs')
const { createGzip } = require('zlib')
const [, , src, dest] = process.argv
const srcStream = createReadStream(src)
const gzipStream = createGzip()
const destStream = createWriteStream(dest)
srcStream
.pipe(gzipStream)
.pipe(destStream)
@loige49

77. readable
.pipe(decompress)
.pipe(decrypt)
.pipe(convert)
.pipe(encrypt)
.pipe(compress)
.pipe(writeToDisk)
Setup complex pipelines with pipe
@loige
This is the most common way to use streams
50

78. readable
.on('error', handleErr)
.pipe(decompress)
.on('error', handleErr)
.pipe(decrypt)
.on('error', handleErr)
.pipe(convert)
.on('error', handleErr)
.pipe(encrypt)
.on('error', handleErr)
.pipe(compress)
.on('error', handleErr)
.pipe(writeToDisk)
.on('error', handleErr)
Handling errors (correctly)
@loige51

79. readable
.on('error', handleErr)
.pipe(decompress)
.on('error', handleErr)
.pipe(decrypt)
.on('error', handleErr)
.pipe(convert)
.on('error', handleErr)
.pipe(encrypt)
.on('error', handleErr)
.pipe(compress)
.on('error', handleErr)
.pipe(writeToDisk)
.on('error', handleErr)
Handling errors (correctly)
@loige
handleErr should end and destroy the streams
(it doesn't happen automatically)
51

80. readable
.on('error', handleErr)
.pipe(decompress)
.on('error', handleErr)
.pipe(decrypt)
.on('error', handleErr)
.pipe(convert)
.on('error', handleErr)
.pipe(encrypt)
.on('error', handleErr)
.pipe(compress)
.on('error', handleErr)
.pipe(writeToDisk)
.on('error', handleErr)
Handling errors (correctly)
@loige
handleErr should end and destroy the streams
(it doesn't happen automatically)
51

81. 04. STREAM UTILITIES04. STREAM UTILITIES
@loige52

82. // stream-copy-gzip-pipeline.js
const { pipeline } = require('stream')
const { createReadStream, createWriteStream } = require('fs')
const { createGzip } = require('zlib')
const [, , src, dest] = process.argv
pipeline(
createReadStream(src),
createGzip(),
createWriteStream(dest),
function onEnd (err) {
if (err) {
console.error(`Error: ${err}`)
process.exit(1)
}
console.log('Done!')
}
)
stream.pipeline(...streams, callback)
@loige53

83. // stream-copy-gzip-pipeline.js
const { pipeline } = require('stream')
const { createReadStream, createWriteStream } = require('fs')
const { createGzip } = require('zlib')
const [, , src, dest] = process.argv
pipeline(
createReadStream(src),
createGzip(),
createWriteStream(dest),
function onEnd (err) {
if (err) {
console.error(`Error: ${err}`)
process.exit(1)
}
console.log('Done!')
}
)
stream.pipeline(...streams, callback)
@loige
Can pass multiple streams (they will be piped)
53

84. // stream-copy-gzip-pipeline.js
const { pipeline } = require('stream')
const { createReadStream, createWriteStream } = require('fs')
const { createGzip } = require('zlib')
const [, , src, dest] = process.argv
pipeline(
createReadStream(src),
createGzip(),
createWriteStream(dest),
function onEnd (err) {
if (err) {
console.error(`Error: ${err}`)
process.exit(1)
}
console.log('Done!')
}
)
stream.pipeline(...streams, callback)
@loige
Can pass multiple streams (they will be piped)
The last argument is a callback. If invoked with an
error, it means the pipeline failed at some point.
All the streams are ended and destroyed correctly.
53

85. readable-stream -
Npm package that contains the latest version of Node.js stream library.
It also makes Node.js streams compatible with the browser (can be used with
Webpack and Broswserify)
npm.im/readable-stream
@loige
* yeah, the name is misleading. The package offers all the functionalities in the official 'stream'
package, not just readable streams.
*
54

86. 04. WRITING CUSTOM 04. WRITING CUSTOM
STREAMS STREAMS
@loige55

87. @loige
EmojiStream Uppercasify DOMAppend
56

88. @loige
EmojiStream Uppercasify DOMAppend
Lemon
56

89. @loige
EmojiStream Uppercasify DOMAppend
LEMON
56

90. @loige
EmojiStream Uppercasify DOMAppend
LEMON
56

91. @loige
EmojiStream Uppercasify DOMAppend
LEMON Banana
56

92. @loige
EmojiStream Uppercasify DOMAppend
LEMON BANANA
56

93. @loige
EmojiStream Uppercasify DOMAppend
LEMON
BANANA
56

94. @loige
EmojiStream Uppercasify DOMAppend
LEMON
class EmojiStream
extends Readable {
_read() {
// ...
}
}
BANANA
56

95. @loige
EmojiStream Uppercasify DOMAppend
LEMON
class EmojiStream
extends Readable {
_read() {
// ...
}
}
class Uppercasify
extends Transform {
_transform(
chunk,
enc,
done
) {
// ...
}
}
BANANA
56

96. @loige
EmojiStream Uppercasify DOMAppend
LEMON
class EmojiStream
extends Readable {
_read() {
// ...
}
}
class Uppercasify
extends Transform {
_transform(
chunk,
enc,
done
) {
// ...
}
}
class DOMAppend
extends Writable {
_write(
chunk,
enc,
done
) {
// ...
}
}
BANANA
56

97. @loige
EmojiStream Uppercasify DOMAppend
LEMON
class EmojiStream
extends Readable {
_read() {
// ...
}
}
class Uppercasify
extends Transform {
_transform(
chunk,
enc,
done
) {
// ...
}
}
class DOMAppend
extends Writable {
_write(
chunk,
enc,
done
) {
// ...
}
}
BANANA
this.push(data)
pass data to the next step
56

98. // emoji-stream.js (custom readable stream)
const { EMOJI_MAP } = require('emoji') // from npm
const { Readable } = require('readable-stream') // from npm
const emojis = Object.keys(EMOJI_MAP)
function getEmojiDescription (index) {
return EMOJI_MAP[emojis[index]][1]
}
function getMessage (index) {
return emojis[index] + ' ' + getEmojiDescription(index)
}
class EmojiStream extends Readable {
constructor (options) {
super(options)
this._index = 0
}
_read () {
if (this._index >= emojis.length) {
return this.push(null)
}
return this.push(getMessage(this._index++))
}
}
module.exports = EmojiStream
@loige57

99. // emoji-stream.js (custom readable stream)
const { EMOJI_MAP } = require('emoji') // from npm
const { Readable } = require('readable-stream') // from npm
const emojis = Object.keys(EMOJI_MAP)
function getEmojiDescription (index) {
return EMOJI_MAP[emojis[index]][1]
}
function getMessage (index) {
return emojis[index] + ' ' + getEmojiDescription(index)
}
class EmojiStream extends Readable {
constructor (options) {
super(options)
this._index = 0
}
_read () {
if (this._index >= emojis.length) {
return this.push(null)
}
return this.push(getMessage(this._index++))
}
}
module.exports = EmojiStream
@loige57

100. // emoji-stream.js (custom readable stream)
const { EMOJI_MAP } = require('emoji') // from npm
const { Readable } = require('readable-stream') // from npm
const emojis = Object.keys(EMOJI_MAP)
function getEmojiDescription (index) {
return EMOJI_MAP[emojis[index]][1]
}
function getMessage (index) {
return emojis[index] + ' ' + getEmojiDescription(index)
}
class EmojiStream extends Readable {
constructor (options) {
super(options)
this._index = 0
}
_read () {
if (this._index >= emojis.length) {
return this.push(null)
}
return this.push(getMessage(this._index++))
}
}
module.exports = EmojiStream
@loige57

101. // emoji-stream.js (custom readable stream)
const { EMOJI_MAP } = require('emoji') // from npm
const { Readable } = require('readable-stream') // from npm
const emojis = Object.keys(EMOJI_MAP)
function getEmojiDescription (index) {
return EMOJI_MAP[emojis[index]][1]
}
function getMessage (index) {
return emojis[index] + ' ' + getEmojiDescription(index)
}
class EmojiStream extends Readable {
constructor (options) {
super(options)
this._index = 0
}
_read () {
if (this._index >= emojis.length) {
return this.push(null)
}
return this.push(getMessage(this._index++))
}
}
module.exports = EmojiStream
@loige57

102. // emoji-stream.js (custom readable stream)
const { EMOJI_MAP } = require('emoji') // from npm
const { Readable } = require('readable-stream') // from npm
const emojis = Object.keys(EMOJI_MAP)
function getEmojiDescription (index) {
return EMOJI_MAP[emojis[index]][1]
}
function getMessage (index) {
return emojis[index] + ' ' + getEmojiDescription(index)
}
class EmojiStream extends Readable {
constructor (options) {
super(options)
this._index = 0
}
_read () {
if (this._index >= emojis.length) {
return this.push(null)
}
return this.push(getMessage(this._index++))
}
}
module.exports = EmojiStream
@loige57

103. // emoji-stream.js (custom readable stream)
const { EMOJI_MAP } = require('emoji') // from npm
const { Readable } = require('readable-stream') // from npm
const emojis = Object.keys(EMOJI_MAP)
function getEmojiDescription (index) {
return EMOJI_MAP[emojis[index]][1]
}
function getMessage (index) {
return emojis[index] + ' ' + getEmojiDescription(index)
}
class EmojiStream extends Readable {
constructor (options) {
super(options)
this._index = 0
}
_read () {
if (this._index >= emojis.length) {
return this.push(null)
}
return this.push(getMessage(this._index++))
}
}
module.exports = EmojiStream
@loige57

104. // uppercasify.js (custom transform stream)
const { Transform } = require('readable-stream')
class Uppercasify extends Transform {
_transform (chunk, encoding, done) {
this.push(chunk.toString().toUpperCase())
done()
}
}
module.exports = Uppercasify
@loige58

105. // uppercasify.js (custom transform stream)
const { Transform } = require('readable-stream')
class Uppercasify extends Transform {
_transform (chunk, encoding, done) {
this.push(chunk.toString().toUpperCase())
done()
}
}
module.exports = Uppercasify
@loige58

106. // uppercasify.js (custom transform stream)
const { Transform } = require('readable-stream')
class Uppercasify extends Transform {
_transform (chunk, encoding, done) {
this.push(chunk.toString().toUpperCase())
done()
}
}
module.exports = Uppercasify
@loige58

107. // uppercasify.js (custom transform stream)
const { Transform } = require('readable-stream')
class Uppercasify extends Transform {
_transform (chunk, encoding, done) {
this.push(chunk.toString().toUpperCase())
done()
}
}
module.exports = Uppercasify
@loige58

108. // uppercasify.js (custom transform stream)
const { Transform } = require('readable-stream')
class Uppercasify extends Transform {
_transform (chunk, encoding, done) {
this.push(chunk.toString().toUpperCase())
done()
}
}
module.exports = Uppercasify
@loige58

109. // uppercasify.js (custom transform stream)
const { Transform } = require('readable-stream')
class Uppercasify extends Transform {
_transform (chunk, encoding, done) {
this.push(chunk.toString().toUpperCase())
done()
}
}
module.exports = Uppercasify
@loige58

110. // dom-append.js (custom writable stream)
const { Writable } = require('readable-stream')
class DOMAppend extends Writable {
_write (chunk, encoding, done) {
const elem = document.createElement('li')
const content = document.createTextNode(chunk.toString())
elem.appendChild(content)
document.getElementById('list').appendChild(elem)
done()
}
}
module.exports = DOMAppend
@loige59

111. // dom-append.js (custom writable stream)
const { Writable } = require('readable-stream')
class DOMAppend extends Writable {
_write (chunk, encoding, done) {
const elem = document.createElement('li')
const content = document.createTextNode(chunk.toString())
elem.appendChild(content)
document.getElementById('list').appendChild(elem)
done()
}
}
module.exports = DOMAppend
@loige59

112. // dom-append.js (custom writable stream)
const { Writable } = require('readable-stream')
class DOMAppend extends Writable {
_write (chunk, encoding, done) {
const elem = document.createElement('li')
const content = document.createTextNode(chunk.toString())
elem.appendChild(content)
document.getElementById('list').appendChild(elem)
done()
}
}
module.exports = DOMAppend
@loige59

113. // dom-append.js (custom writable stream)
const { Writable } = require('readable-stream')
class DOMAppend extends Writable {
_write (chunk, encoding, done) {
const elem = document.createElement('li')
const content = document.createTextNode(chunk.toString())
elem.appendChild(content)
document.getElementById('list').appendChild(elem)
done()
}
}
module.exports = DOMAppend
@loige59

114. // dom-append.js (custom writable stream)
const { Writable } = require('readable-stream')
class DOMAppend extends Writable {
_write (chunk, encoding, done) {
const elem = document.createElement('li')
const content = document.createTextNode(chunk.toString())
elem.appendChild(content)
document.getElementById('list').appendChild(elem)
done()
}
}
module.exports = DOMAppend
@loige59

115. // dom-append.js (custom writable stream)
const { Writable } = require('readable-stream')
class DOMAppend extends Writable {
_write (chunk, encoding, done) {
const elem = document.createElement('li')
const content = document.createTextNode(chunk.toString())
elem.appendChild(content)
document.getElementById('list').appendChild(elem)
done()
}
}
module.exports = DOMAppend
@loige59

116. 05. STREAMS IN THE 05. STREAMS IN THE
BROWSER BROWSER
@loige60

117. // browser/app.js
const EmojiStream = require('../emoji-stream')
const Uppercasify = require('../uppercasify')
const DOMAppend = require('../dom-append')
const emoji = new EmojiStream()
const uppercasify = new Uppercasify()
const append = new DOMAppend()
emoji
.pipe(uppercasify)
.pipe(append)
@loige61

118. // browser/app.js
const EmojiStream = require('../emoji-stream')
const Uppercasify = require('../uppercasify')
const DOMAppend = require('../dom-append')
const emoji = new EmojiStream()
const uppercasify = new Uppercasify()
const append = new DOMAppend()
emoji
.pipe(uppercasify)
.pipe(append)
@loige61

119. // browser/app.js
const EmojiStream = require('../emoji-stream')
const Uppercasify = require('../uppercasify')
const DOMAppend = require('../dom-append')
const emoji = new EmojiStream()
const uppercasify = new Uppercasify()
const append = new DOMAppend()
emoji
.pipe(uppercasify)
.pipe(append)
@loige61

120. // browser/app.js
const EmojiStream = require('../emoji-stream')
const Uppercasify = require('../uppercasify')
const DOMAppend = require('../dom-append')
const emoji = new EmojiStream()
const uppercasify = new Uppercasify()
const append = new DOMAppend()
emoji
.pipe(uppercasify)
.pipe(append)
@loige61

121. npm i --save-dev webpack webpack-cli
node_modules/.bin/webpack src/browser/app.js
# creates dist/main.js
mv dist/main.js src/browser/app-bundle.js
@loige
Let's use webpack to build this app for the browser
62

122. <!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta
name="viewport"
content="width=device-width, initial-scale=1, shrink-to-fit=no"
/>
<title>Streams in the browser!</title>
</head>
<body>
<ul id="list"></ul>
<script src="app.bundle.js"></script>
</body>
</html> @loige
Finally let's create an index.html
63

123. <!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta
name="viewport"
content="width=device-width, initial-scale=1, shrink-to-fit=no"
/>
<title>Streams in the browser!</title>
</head>
<body>
<ul id="list"></ul>
<script src="app.bundle.js"></script>
</body>
</html> @loige
Finally let's create an index.html
63

124. <!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta
name="viewport"
content="width=device-width, initial-scale=1, shrink-to-fit=no"
/>
<title>Streams in the browser!</title>
</head>
<body>
<ul id="list"></ul>
<script src="app.bundle.js"></script>
</body>
</html> @loige
Finally let's create an index.html
63

125. @loige64

126. @loige64

127. 06. CLOSING06. CLOSING
@loige65

128. Streams have low memory footprint
Process data as soon as it's available
Composition through pipelines
Streams are abstractions:
Readable = Input
Transform = Business Logic
Writable = Output
@loige
TLDR;TLDR;
66

129. IF YOU WANT TO LEARN (EVEN) MOAR IF YOU WANT TO LEARN (EVEN) MOAR
ABOUT STREAMS...ABOUT STREAMS...
nodejs.org/api/stream.html
github.com/substack/stream-handbook
@loige67

130. IF YOU ARE NOT CONVINCED YET...IF YOU ARE NOT CONVINCED YET...
@loige
curl parrot.live
68

131. @loige
github.com/hugomd/parrot.live
Check out the codebase
69

132. @loige70

133. @loige
THANKS!THANKS!
loige.link/streams-cityjs
We are hiring, talk to me! :)
71

134. CREDITSCREDITS
on for the cover picture
for the amazing St. Patrick emoji art
The internet for the memes! :D
Dan Roizer Unsplash
emojiart.org
SPECIAL THANKSSPECIAL THANKS
, , ,
, ,
@StefanoAbalsamo @mariocasciaro @machine_person
@Podgeypoos79 @katavic_d @UrsoLuca
@loige72