Dynamically extend pipe chains in Node.js

Question

I'm currently practicing with streams in Node.js using the modules through2, concat-stream and request.

I set up a pipe chain like this:

request(process.argv[2]) .pipe(through(write, end)) .pipe(/*And */) .pipe(/* Some */) .pipe(/* more */) .on("finish",function (){ #Do some stuff }); function write(buf, enc, nxt){ /*Push some data after some processing*/ } function end(done){ done() }

This is a static chain of pipes. Is it possible to, through some form of user input, dynamically specify a pipe-chain?

In pseudo code:

array_of_user_pipes = from_some_input; pipe_library = {pipe_name:pipe_callback} object, loaded into application logic perform the url request (fetch .txt file over the internet) for all pipe in array_of_user_pipes do fetch pipe from pipe_library (simple key look-up) chain pipe to chain execute the (dynamic) pipe chain

It looks totally doable. Have you encountered any specific problems implementing this approach? — Yury Tarabanko
– Yury Tarabanko, Commented Jun 9, 2017 at 12:13
Just not really sure how to set it up, because static chains are set up through method chaining, this isn't possible if you're iterating, or is there another way to add a pipe to a chain ? — lennyklb
– lennyklb, Commented Jun 9, 2017 at 12:16

Yury Tarabanko · Accepted Answer · 2017-06-09 14:22:20Z

No problems in translating your pseudo code to js

const pipesMap = { pipeName1: require('pipe-module'), pipeName2: function(){}, ... }

Define a piper function that takes pipe names and returns a function that pipe initial event stream through each of them.

const piper = pipes => request => pipes.reduce((piped, pipe) => piped.pipe(pipesMap[pipe]), request) const userInputPipesArray = ['pipeName1', 'pipeName2'] piper(userInputPipesArray)(request(process.argv[2])).on('finish')

Edit

You could do the same using for-loop

let piped = request(process.argv[2]) for(let pipe of userInputPipesArray) { piped = piped.pipe(pipeMap[pipe]) } piped.on('finish', ...)

Thanks a lot, I'll try it out. Does this approach have a certain name? I'd like to read up on it.
Not sure if there is any special name for when reduce aka fold is used this way. This could be done w/o reduce using for-loop.
Thanks. This looks exactly like the way I wanted, perhaps I was too pessimistic :)

Collectives™ on Stack Overflow

Dynamically extend pipe chains in Node.js

1 Answer 1

3 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Related