Hi all! I'm very new to the DEV community. Today, I'd like to introduce data stream transfer between almost every device over HTTP/HTTPS made for use with Unix Pipe and also for browser!

What do I want to solve?

We sometimes want to transfer data between Mac, Windows, Linux, Unix, iPhone, Android... Although we have AirDrop, Google Drive, Dropbox, Slack, WhatsApp, Skype, netcat, ssh or etc, we need to

• Find a common service used by both sender and receiver
• Install additional GUI software
• Sign up some service
• Solve NAT traversal

In addition, almost existing data transfer services are not CUI-friendly or not GUI-friendly.

So, I made a solution to the problems above. The system allows you to

• Transfer data with almost every device
• Use it without additional installation
• Use it with Unix/Linux Pipe and it's engineer-friendly

Actual Usages

I'd like to introduce how to use the system I made.

Here is a demo to transfer seq 100000.

Send: seq 100000 | curl -T - https://ppng.io/myseq
Get: curl https://ppng.io/myseq

Here is a demo to transfer a 100MB file.

Send: cat 100MB.dat | curl -T - https://ppng.io/mydata
Get: curl https://ppng.io/mydata > mydata

You can transfer any data by using pipe. For example,

Compress & Send: cat myfile | gzip | curl -T ...
Get & Decompress: curl ... | zcat > myfile
and
Encrypt & Send: cat myfile | openssl aes-256-cbc | curl -T ...
Get & Decript: curl ... | openssl aes-256-cbc -d > myfile
and
Send directory(zip): zip -q -r - ./mydir | curl -T - https://ppng.io/mydir.zip
Send directory(tar.gz): tar zfcp - ./mydir | curl -T - https://ppng.io/mydir.tar.gz

Here is a demo to transfer data to a browser.

Here is a demo to transfer data from a browser.

This means you can transfer a file from Android/iPhone/iPad... to another device!

Transfer to Multiple Receivers

Terminal Screen Sharing!

Share: script -f >(curl -T - https://ppng.io/myscript >&/dev/null)
View: curl https://ppng.io/myscript
(Use script -F in Mac instead of script -f)

GitHub Repository

The project is Piping Server found in the link below.
GitHub: https://github.com/nwtgck/piping-server

# Send
echo 'hello, world' | curl -T - https://ppng.io/hello

# Get
curl https://ppng.io/hello > hello.txt

This project is written in TypeScript.

Public Servers

Here are public servers. My recommendation is https://ppng.io because it has a short name.

• https://ppng.io/
• https://ppng.herokuapp.com
• https://piping-92sr2pvuwg14.runkit.sh/

In my policy, you can freely run the servers. I'd like to describe how to run later.

You can get engineer-friendly help from the following.

$ curl https://ppng.io/help
<Help will be displayed here>

How it works

I think Piping Server is like a TURN server in Pear-to-Pear. Piping Server relays your data to another device to solve NAT traversal.

A body of POST/PUT HTTP request is passed to a body of GET HTTP response.

How to Run Piping Server

We have several ways to run Piping Server. I'd like to introduce one by one.

Heroku

Push [Deploy to Heroku] button in https://github.com/nwtgck/piping-server.

Portable Binary Executable

The following commands make a Piping Server run on http://localhost:8888.

wget https://github.com/nwtgck/piping-server-pkg/releases/download/v0.8.7/piping-server-macos
chmod +x piping-server-macos
./piping-server-macos --http-port=8888

You can also find other platform releases in GitHub Releases in https://github.com/nwtgck/piping-server-pkg. The portable executables are produced by zeit/pkg.

Docker

A docker run make a Piping Server run on http://localhost:8888.

docker run -d --restart=always -p 8888:80 nwtgck/piping-server --http-port=80

npm

The following commands make Piping Server run on http://localhost:8888.

# install
npm install -g piping-server
# run
piping-server --http-port=8888

Possibility of Data Transfer over HTTP

I'm conducting an experiment to reveal how many data can be transferred over HTTP.
I started the experiment 55 days and 7 hours ago and over 1000 terabyte data were transferred.

The method of the experiment is as follows.
Infinitely send: cat /dev/urandom | curl -T - localhost:8888/rand
Infinitely discard: curl localhost:8888/rand > /dev/null

Transfer Speed Compared with Go

Here is a simple comparison in transfer speed between TypeScript/Node.js and Go implementations. As a result of the videos, there seem to be almost no differences.

TypeScript/Node.js:

Go:

Repo Go ver.: https://github.com/nwtgck/go-piping-server

Japanese Post

Here is my original Japanese post: https://qiita.com/nwtgck/items/78309fc529da7776cba0.
You can get more information there.

Thank you very much for reading. Have a wonderful day!