This article aims to identify the different Internet censorship methods utilized to block Clubhouse in Jordan, based on some schematic analysis we performed since Clubhouse-related issues were first reported on March 15th.
It is important to note that not all the ISPs were, or are currently, blocking Clubhouse, specifically, as of April 18th, JCS (AS44702) is currently not restricting access to the application. On the other hand, Zain (AS48832), on its 4G network, Orange (AS8376), VTel (AS50670) and DAMAMAX (AS47887) have restricted access to Clubhouse, on a continuous basis since mid-March, while Umniah apparently started implementing the blocking in a later phase around end of March.
Blocking access to certain websites in Jordan is mainly done through two different methods depending on the ISP. While Umniah (AS9038) and DAMAMAX (AS47887) are using DNS tampering, analysis on the traffic on Zain (AS48832) and Orange (AS8376) suggests the use of Deep Packet Inspection (DPI) techniques, here we are presenting some findings from the analysis.
DNS Tampering
Altering responses from the DNS is a common technique to block access to websites, this is done by interfering with DNS and providing clients with altered responses, specifically, in ‘DNS hijacking’, the DNS resolver ‘lies’ and returns the wrong IP address to the client, a behaviour which we encountered on Umniah’s network. Although Umniah apparently allowed access to Clubhouse for a number of days, it is currently blocking the domain joinclubhouse.com by responding to the DNS query by a NXDOMAIN status (No such name), informing the client that the queried domain name does not exist in the DNS:
$ dig joinclubhouse.com; <<>> DiG 9.16.1-Ubuntu <<>> joinclubhouse.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 705
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;joinclubhouse.com. IN A
;; Query time: 67 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: XX02 14:26:49 EEST 2021
;; MSG SIZE rcvd: 46
Querying the Clubhouse’s domain on a different name server, 1.1.1.1 in this example, will provide a different answer, where the domain name is correctly resolved to 104.18.21.150 and 104.18.20.150, Clubhouse’s IP addresses on Cloudflare.
$ dig @1.1.1.1 joinclubhouse.com; <<>> DiG 9.16.1-Ubuntu <<>> @1.1.1.1 joinclubhouse.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23688
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;joinclubhouse.com. IN A
;; ANSWER SECTION:
joinclubhouse.com. 300 IN A 104.18.21.150
joinclubhouse.com. 300 IN A 104.18.20.150
;; Query time: 155 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: XX02 14:29:19 EEST 2021
;; MSG SIZE rcvd: 78
Umniah are providing their 4G customers a Huawei E5577 Wifi router with a built-in DHCP server and a DNS server, accessible by the same router’s IP address, 192.168.8.1, once a blocked domain is queried, an SOA (Start of Authority) resource record is provided to the client with a name of the zone blacklist.umniah.local. In our query, we assumed that the serial number of the zone follows the recommendations in RFC 1912, therefore, the date of the last change as announced by the record was March 29, 2021.
$ dig @192.168.8.1 joinclubhouse.com SOA +noall +authorityblacklist.umniah.local. 3600 IN SOA f5-a.umniah.com.blacklist.umniah.local. hostmaster.f5-a.umniah.com.blacklist.umniah.local. 2021032902 10800 3600 604800 86400
It doesn’t seem that traffic on Umniah presents other anomalies, in this case and similar cases, users could access Clubhouse by changing their DNS resolver.
On DAMAMAX, the domain joinclubhouse.com fails to resolve too, even after changing the DNS into an open/public DNS resolver like 1.1.1.1 or 8.8.8.8. Specifically, no answers are received to the DNS queries to Clubhouse’s domain.
$ dig @1.1.1.1 joinclubhouse.com; <<>> DiG 9.10.6 <<>> @1.1.1.1 joinclubhouse.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
A DNS query response with the same transaction ID (0xa4b6 in the example below) and an address would have been expected, however no response was received, this might suggest DNS responses containing joinclubhouse.com might be filtered out.
$ tshark -i any host 1.1.1.1 and port 53 16 49.269871 192.168.x.x → 1.1.1.1 DNS 74 Standard query 0xa4b6 A joinclubhouse.com OPT
17 54.271342 192.168.x.x → 1.1.1.1 DNS 74 Standard query 0xa4b6 A joinclubhouse.com OPT
18 59.275237 192.168.x.x → 1.1.1.1 DNS 74 Standard query 0xa4b6 A joinclubhouse.com OPT
After this finding, an attempt to resolve the domain through DoH (DNS-over-HTTPS) has been conducted, in DoH, the domain name is sent to a DoH-compatible DNS server using an encrypted HTTPS connection instead of a plain text one. The domain is correctly resolved, as shown in the last connection of the following response:
$ curl -v --doh-url https://cloudflare-dns.com/dns-query https://joinclubhouse.com…
* Connection #1 to host cloudflare-dns.com left intact
* a DOH request is completed, 0 to go
* DOH Host name: joinclubhouse.com
* TTL: 198 seconds
* DOH A: 104.18.21.150
* DOH A: 104.18.20.150
* DOH AAAA: 2606:4700:0000:0000:0000:0000:6812:1596
* DOH AAAA: 2606:4700:0000:0000:0000:0000:6812:1496
* Trying 104.18.21.150...
* TCP_NODELAY set
* Connected to joinclubhouse.com (104.18.21.150) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
…
Deep Packet Inspection
Now, let’s have a look at some quick analysis we have done on Zain. While a TCP connection can be established successfully to https://joinclubhouse.com (IP: 104.18.21.150), a “Connection reset by peer” error is received:
$ curl -v https://joinclubhouse.com* Trying 104.18.21.150:443...
* TCP_NODELAY set
* Connected to joinclubhouse.com (104.18.21.150) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: Connection reset by peer in connection to joinclubhouse.com:443
* Closing connection 0
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to joinclubhouse.com:443
The error is generated due to the reception of a RST/ACK packet after we have sent the initial packet of the TLS handshake (Client Hello), as can be seen in our captured packets (PCAP) while analyzing Clubhouse:
$ tshark -r clubhouse.pcap 1 0.000000000 192.168.x.x → 104.18.21.150 TCP 74 42864 → 443 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=3151920416 TSecr=0 WS=128
2 0.056612164 104.18.21.150 → 192.168.x.x TCP 66 443 → 42864 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1400 SACK_PERM=1 WS=1024
3 0.056642802 192.168.x.x → 104.18.21.150 TCP 54 42864 → 443 [ACK] Seq=1 Ack=1 Win=64256 Len=0
4 0.067406532 192.168.x.x → 104.18.21.150 TLSv1 571 Client Hello
5 0.096786915 104.18.21.150 → 192.168.x.x TCP 54 443 → 42864 [RST, ACK] Seq=1 Ack=518 Win=32890880 Len=0
Previous network measurements on Zain strongly indicated the utilization of DPI techniques to block websites and apps. The specific use of Sandvine Packetlogic devices, produced by the U.S.-based company Sandvine, can be seen in one of our reports, published in cooperation with OONI, by looking at some fingerprint elements that are considered specific to these devices, namely, the injected packet received by the client is an empty RST/ACK packet of which the IPID was 0x00003412, a highly distinctive fingerprint of Sandvine Packetlogic according to a CitizenLab report. In October 2020, a report from Bloomberg mentioned the use of such devices in Jordan, according to one current and two former employees of Sandvine, their equipment has enabled repeated government-ordered internet shutdowns.
However, recent packets captured while scanning Clubhouse do not show the same fingerprint, although an empty RST/ACK packet is still received, the IPIDs are not fixed:
$ tshark -T fields -e frame.number -e ip.id -E separator=: -r clubhouse_1.pcap1:0x0000ab10
2:0x00000000
3:0x0000ab11
4:0x0000ab12
5:0x0000c0f1$ tshark -T fields -e frame.number -e ip.id -E separator=: -r clubhouse_2.pcap1:0x0000f590
2:0x00000000
3:0x0000f591
4:0x0000f592
5:0x0000d030
Compared to previous analysis (in 2018 and early 2019) that always shows a fixed IPID of 0x00003412 in the fifth packet:
$ tshark -T fields -e frame.number -e ip.id -E separator=: -r facebook_2018.pcap1:0x00003b7b
2:0x00000000
3:0x00003b7c
4:0x00003b7d
5:0x00003412
A not-fixed IPID is indeed the normal behaviour, there are several possible explanations on the reason why the previous IPID (0x00003412) is not showing in more recent analysis of blocked websites on Zain, this might include the potential recent use of newer versions of Packetlogic devices or alternative devices or techniques not characterized by the aforementioned fingerprint.
In contrast to Zain, the website joinclubhouse.com is accessible on Orange, which in turn bans access to the app by blocking access to its API endpoints available at https://www.clubhouseapi.com/api, analysis of the traffic to Clubhouse on Orange shows, similarly to Zain, that the connections are closed by injecting an RST/ACK packet with not-fixed IDs.
Conclusion
As not all the ISPs are implementing the blocking of Clubhouse equally, and different techniques are in place, it would be safe to confirm that the blocking of Clubhouse is not implemented on a central level, but rather independently by each service provider.
The use of alternative DNS resolvers and/or the use of DNS over HTTPS/TLS should allow users to bypass blocking on operators adopting DNS tampering techniques (like Umniah and DAMAMAX).
The use of VPN can circumvent both the Internet Censorship methods used in Jordan. As Clubhouse was first reported blocked on some networks on March 15th, it was not strange that instantly after facing issues connecting to the app, multiple users in Jordan resorted to VPNs to circumvent the blocking. In a matter of minutes, many people returned to the app just by enabling their already-installed VPNs, tools that became extremely popular in the last years as a result of a not-so-shiny history of Internet censorship in this country, consequently, some of these VPN tools were also affected to some blocking, including TunnelBear, NordVPN and ExpressVPN.
(UPDATE APRIL 18, 2021: Analysis on Orange shows the app is blocked, the article was modified to reflect that)
(UPDATE JUNE 16, 2021: More recent analysis shows the app is still blocked by the ISPs listed in this article)
