Introduction

You all probably know previews in a lot of apps. For example, when you share something in Microsoft Teams and you can preview the webpage without actually visiting it within your chat. For a while I have been wondering how the behaviour of preview bots and crawlers differs between the different services and what their possible security and privacy implications would be. End to end encryption would also mean, that the preview generation would need to happen client-side, as the server never has access to the message content to actually fetch a preview, for example.

Content

Experiment Setup

In order to properly track the requests I set up an nginx reverse proxy with the following paths:

• / (just a simple html as a standard usecase)
• /same-site-redirect (redirects to /)
• /external-redirect (redirect to https://example.com)
• /redirect-loop (redirect to /redirect-loop - this was a good one)
• /ssrf-redirect-magic (redirect to the magic ip )
• /ssrf-redirect-local (redirect to good ol' localhost)

For easier testing I have also configured the nginx to log JSON format and set up a small webserver that would read the JSON and display it in a nice data table. All the source code is more or less documented available on github.

Findings summary

1. All of the apps that issued requests, did this from their datacenter, meaning they have access to the message content in some form.

2. Whenever you post a link on twitter (aka X), the AppleBot will pay you a visit within a matter of seconds. At least they are kind enough to check the robots.txt

3.When you add a link to your instagram bio, the facebook bot will crawl it to get to know you better
4.The facebook bot handles the redirect-loop the worst (requested it like 20 times...). Interestingly it used different IP Adresses for handling the redirects.

5. When you click an instagram bio link, facebook automatically adds a parameter "fbclid" to track your interaction

6. Signal was one of the few apps that did not cause any request to the nginx

Results Table

Following some notes from the observations. Feel free to check out the referenced github repo and try it yourself.