As it may appear surprising that a TV station can be forced to stop broadcasting after having its website defaced and social network accounts controlled by some hackers, I’ve tried to collect publicly available technical information and improve my understanding of this interesting issue. Below you will find my own technical analysis of infrastructure components that may be in use at TV5Monde, I’m not an expert in TV Broadcasting, my main speciality being IT Infrastructure, however as we will see those two domains are becoming closer.
What is TV5 Monde (TV5 World in English)?
TV5 Monde is a global television network, editing and broadcasting French language contents, mostly taken from French-speaking countries (France Television for France, RTBF for Belgium, RTS from Switzerland, and Radio-Canada). TV5 also produces some contents, like news magazines. TV5 is headquartered in Paris (Avenue de Wagram).
Being a French national, working and living abroad for years, I’m a subscriber of TV5 Monde through my local TV provider (StarHub in Singapore). It is quite popular with French-speakers around the world, but of less interest for French national residing in France (mostly being an extract of programmes already available on French channels).
From the news reports, TV5 Monde started to lose control of their Twitter and Facebook accounts, then the tv5monde website was defaced. Shortly after that, the email system seems to have stopped working, then the TV broadcast system. This is not confirmed and only taken from public news sources. The video broadcast was stopped for approx. 3 hours. It was then restored for pre-recorded contents only.
In this investigation I will not focus on the Twitter / Facebook hack or website defacement, but will try to get a better understanding of the TV5Monde infrastructure used in general.
Let’s move on to the technical part. I will split this study in 2 parts. The first one will be looking at the “Corporate Network”, the day-to-day general network used by staff, and supporting emails, internet browsing, phones and so on.
The second part will be focusing on the “Broadcasting Network”, used by staff to produce (and post-produce) video casts, and stream them globally.
What can be remotely discovered on TV5 Monde IT Infrastructure, without directly performing any scan on their systems (which are still mostly down at this time)?
This part is what I will call the “Corporate Infrastructure”, supporting users and general infrastructure (email, fileservers, and corporate applications).
Let’s begin with the DNS. Several DNS domains seems to be in use:
tv5monde.com, tv5monde.org, tv5.org, tv5paris.org
tv5monde.com, tv5monde.org and tv5.org DNS are managed by Gandi, a well-known French provider, with a serious reputation. Gandi DNS provides dual-authentication for your DNS management portal.
tv5paris.org DNS are managed by another company crt.fr, which I don’t personally know. I don’t know either which kind of security they provide to clients to administer their DNS.
The MX records are shown below. A MX record is a Mail Exchanger Record
tv5.org MX xcs1.tv5paris.org 220.127.116.11 Preference: 10
tv5.org MX xcs2.tv5paris.org 18.104.22.168 Preference: 20
tv5monde.com MX mail.tv5monde.com 22.214.171.124 Preference: 5
tv5monde.org MX xcs1.tv5paris.org 126.96.36.199 Preference: 10
tv5monde.org MX xcs2.tv5paris.org 188.8.131.52 Preference: 20
tv5paris.org MX mail.tv5paris.org 184.108.40.206 Preference: 10
Apart from tv5monde.com, all the MX are pointing to tv5paris.org – which is maybe a legacy domain (TV5 changed names several times in the past years).
From a Google search, the most frequent publicly communicated email domain seems to be tv5monde.org, so using xcs1.tv5paris.org and xcs2.tv5paris.org as MX.
xcs1.tv5paris.org and xcs2.tv5paris.org are still up, they are WatchGuard network security appliances, with their admin web interface externally responding on port 443(with an expired and incorrect certificate).
Using the IP’s of the MX we can now find the public ranges from the RIPE public database.
2 public Internet ranges owned by TV5 can be found: a /24 TV5-NET and a /28 FR-TV5-MONDE.
220.127.116.11 France TV5-NET TV5 Monde 18.104.22.168 22.214.171.124 126.96.36.199/24 Yes Jean-Pierre VERINES 131, avenue de Wagram, 75017 Paris, France email@example.com firstname.lastname@example.org +33 1 44 18 55 55 +33 1 44 18 49 38 RIPE NCC xcs1.tv5paris.org
188.8.131.52 France FR-TV5-MONDE TV5-MONDE 184.108.40.206 220.127.116.11 18.104.22.168/28 Yes Vincent FLEURY TV5 MONDE, FR-PARIS, France email@example.com + 33 1 44 18 55 23 RIPE NCC
A 3rd one, from Claranet (French ISP) range is not fully owned by TV5 (ISP range).
22.214.171.124 France TYPHON Typhon 126.96.36.199 188.8.131.52 184.108.40.206/24 Yes Claranet Network Operations Center 18-20, rue du Faubourg du Temple, 75011 PARIS firstname.lastname@example.org email@example.com +33 1 70 13 70 00 +33 1 70 13 70 01 RIPE NCC tv5monde.typhon.net
Open ports ?
We can now explore those network ranges with Shodan, an online search engine for Internet-connected devices.
A Shodan query on the TV5-NET network range reveals disturbing findings… (a query on the other range didn’t showed any results). It must be noted that the Shodan results are time stamped from a month ago.
https://www.shodan.io/search?query=net%3A220.127.116.11%2F24 (you need a free account to see the results).
(Partial) screenshots of the Shodan results:
A number of TV5 Internal IT components were externally accessible recently (March 2015, before the network shutdown), like a Google Search Appliance, a PowerShell Remote Management Interface, a Dreambox (digital television receiver), a Mac Mini, a Remote Desktop to a Windows server, a SMB file share, and so on.
As shown in the screenshot, an interesting system found is Isilon. I will develop later on why this maybe important.
At this stage, it seems that there was a significant Internet exposure of TV5Monde internal IT systems, from the Corporate Network.
What else can be learned on their systems, from public sources?
A set of TV5 Monde Tenders / Technical Requests for Proposals (“Appels d’Offres” in French) are available online. Those documents are RFP for vendors, to bid on the management of TV5 Monde internal Infrastructure and were published in mid-2013.
Those documents are public due to a European/French law on Public Tenders, which TV5 Monde must comply with (even being a Private company if I’m correct). They can be read here (in French only): http://telephonie-electricite-securite.avisdemarche.com/author/tv5-monde/
From the RFPs, we can identify the following equipment as being used in 2013 on their internal network:
- Checkpoint 4407 / Cisco ASA 5520 – Firewalls, VPN, maybe remote access , (the scope of usage is not specified in the RFP)
- WatchGuard XTM810 – Probably the email front ends seen earlier
- Nexus 7010 – Large core switches, probably connected to the SAN
- 2 Cisco 6513 – Access Switches
- IP Phones Mitel
- Rancid – Network Monitoring
- Cisco Fabric Interconnect – to the SAN
- Backup + File Servers
- 4D / FileMaker
- Oracle (1 production cluster, 1 Backup VM)
- SQL Server (3VM for IT , 2 active/passive clusters for production)
- MySQL (11 standalone VM, 4 clusters)
A rather classical set of Infrastructure hardware in a medium-size corporation.
The Internet Browsing is not explicitly detailed, there is no mention of Reverse-Proxy or Web Application Firewall being used (which doesn’t means there is not).
I haven’t found any public information regarding which type of antivirus is in use internally on Desktops and/or servers. Same for the Patch Management of Windows and Mac environment.
All this is useful, but doesn’t help to understand the TV Broadcasting infrastructure.
Through some online research, some of the equipment used (or still in use) can be identified, in what I will call the “Broadcasting Infrastructure”. The broadcasting is a long interconnected chain of processes, tools, and staff.
In Sept 2012, TV5 Monde renewed its outsourcing to Ericsson, “the biggest broadcast outsourcing contract in France”
Digital Broadcasting is heavily relying on a similar infrastructure than the traditional servers, but with significant differences in term of hardware and software, and connectivity. A very detailed technical article on the new TV5Monde Broadcasting centre can be found here (in French only, but the pictures are worth to check):
On top of the video contents produced internally at TV5Monde, incoming and outgoing video feeds will come and go from/to various providers (you can identify Arquiva on one of the screenshot, Arquiva is one communications company providing satellites accesses to media organisations. Most of those connectivity links should be private links (like Colt that we can identify on the screenshots of the article).
However TV5 Monde is also broadcasting on the Internet. It seems that they are using Tata Communication CDN
http://cdn.tatacommunications.com/download/tatacase-tv5monde__study3.4-x1a.pdf, I don’t know how the content is pushed to them.
A set of different software is running on dozens of processing servers in the Broadcast Infrastructure. Such servers are used to manage the massive amount of video contents, their commercial rights, their encoding, their meta-data and so on.
Those processes are called MAM: Media Asset Management and PAM: Production Asset Management. The software used are a mix of commercial and in-house developments. iNews from Avid, Mosart from Vizrt, Louise and SGT (third-party development). A third-party development called SYGEPS is used as an Orchestrator, it’s a Tomcat + Oracle DB running on CentOS (source: http://www.rs2i.fr/tv5-monde)
The production (mostly preparation of news reports) is done on 60 workstations (brand not specified). The post-production (tasks done on the content after the production itself) is done through 24 workstations (brand not specified, but maybe Mac). It is not specified where and to which Local Network those workstations are connected to. Are they running on the Corporate Network described earlier, or are they segregated?
From news video footage done during the incident in the TV5Monde premises, some of the staff in the office seems to have 2 desktops: 1 PC and 1 Mac.
An interesting TV5 Monde staff interview (in French again) is available here:
The TV5 Monde staff interviewed is explaining that he was preparing a newscast when the attack started. My translation: “I was expecting files from Gabon (Africa), and those files were not arriving. I was with the sender over the phone, as he was sending me the files through email – in fact it was simply a link to WeTransfer. And the sender kept telling me that the email was sent but nothing was coming through on my side”.
What’s interesting here is not that the email was not arriving or the download was not starting, which may be due to various reasons (linked to the attack or not), but the use of WeTransfer for preparing a newscast.
WeTransfer being a public cloud-based file-sharing service, does that means that the Production Workstations have direct access to Internet, as well as being on the same LAN than the Broadcast servers? Or the files are downloaded on the Corporate Network then safely transferred to the Production side?
For those who wants to explore those MAM/PAM software further, interesting readings can be found here http://www.rs2i.fr/tv5-monde (French) and there http://www.vizrt.com/products/viz_mosart/#
Overall, there is a long list of servers and software used in the production and post-production chain.
However, the focus of our interest is to better understand how the broadcast itself was stopped, during 3 hours.
Windows 7 for the Transmission Server ?
One of the component repeatedly mentioned by media during the crisis was the “Transmission server”. I’m quoting journalists and one TV5 Staff saying on TV: “The Transmission Server was hacked”. Obviously, this is far from being fully confirmed. However, we can find reference to what maybe a transmission server potentially in use at TV5Monde here:
http://www.amydv.gr/tv5-monde-adopts-nexio-volt-servers/?lang=en and there http://www.live-production.tv/news/products/tv5monde-modernizes-playout-center.html
The Nexio Volt (from Harris Broadcast) is a media server able to stream multiple feeds in SD and HD, it’s a 1U rack server, running on Windows 7 x64 Ultimate Edition. It is not specified if any hardening is done on the OS by default.
I don’t know the technical rationale behind this OS choice for a server, maybe due to graphics drivers’ availability for the video processing cards.
Nexio Volt specification details can be found here:
Connected to the Volt, the Farad is a SAN Storage system designed for broadcast and production facilities. http://www.imaginecommunications.com/products/playout/video-servers/nexio-farad
Another system which seems in use in TV5 Monde Broadcasting department is Pixel Power ChannelMaster:
You can find more details on the usage of a ChannelMaster here: http://www.pixelpower.com/sites/pixelpower.com/files/channelmaster_rev01_web_lowres.pdf
The Production SAN exposed on Internet ?
I haven’t been able to find the OS used or detailed tech specs behind ChannelMaster. However in the same article, we can find the following information:
“TV5 Monde has purchased Pixel Power’s ChannelMaster no compromise integrated playout technology. It has also installed Gallium, Pixel Power’s integrated, sophisticated and scalable scheduling, asset management and automation system”
“Gallium is also integrated with the broadcaster’s secondary storage – IBM and Isilon – to manage media transfer to the ChannelMaster local playout cache”
Isilon… Which we have seen earlier in the Shodan report, with an FTP exposed on Internet. Isilon is an EMC SAN storage system, specially designed to cope with video broadcasting storage constraints.
TV5 owns at least one, http://www.emc.com/collateral/customer-profiles/h10621-cp-tv5monde.pdf
This press release is showing a simplified diagram of where the Isilon seats in the Broadcasting chain.
Does this Isilon the same server than the one appearing in the Shodan report? I can’t confirm this, but with the probable high-cost of such equipment, it doesn’t seems economically sound to use an Isilon just as an external FTP server…
No Antivirus ?
Volicon Observer is another system part of the Broadcasting infrastructure.
An old admin guide manual can be found here (the user manuals requires a client account)
The client used for Volicon Observer can be any Web browser, where the Volicon Overser Web server seems to be using PHP (from the Admin guide screenshots page 17). The OS used seems to be Windows due to references to C:\ for file paths on the server (but I don’t know if this is a Server or Workstation edition). Page 94 provides an overview of services running on the box
An interesting remark found in the admin guide, section “What not to do on the server side”:
“Do not install Antivirus software until checking with the Volicon Support group. In addition see the Antivirus Excluded Storage Areas / Services to Scan”.
There is probably more similar systems in the Broadcasting infrastructure.
I don’t know if the Volt is the famous “Transmission Server” currently in use at TV5 Monde.
I can’t tell if this is the last server used before sending the content outside of TV5, and I can’t confirm if this was the system which went down. And last but not least, no information on the internal segregation between the Corporate and the Broadcasting infrastructure have been published.
But as detailed earlier, some of the broadcasting infrastructure components are third-party specialized hardware running Windows OS editions.
As a result, the same security constraints than on any Windows desktop or server will have to be taken in account (antivirus, anti-malware, patching, permissions, credentials, Internet browsing protection, etc).
It’s unfortunately not uncommon for such critical “vendor blackbox” to be managed as an independent third-party system, when – at then end – it’s only a regular Windows system running on the LAN. Maintenance and patching being left to the vendor’s responsibility (and rarely done at the same frequency than the official patch releases)… This doesn’t means that was the case at TV5 Monde but this remains a possibility…
From an external point of view, only using publicly available information, some questions are open:
- Why a number of IT services were available on Internet, increasing the surface attack on the Corporate Infrastructure?
- Which kind of Internet browsing security and desktop protection (Mac and PC) was provided to the staff?
- Was the Corporate network segregated with the Broadcasting network? How?
- How the specialised equipment used in Broadcasting (proprietary hardware but running commercial standard OS for some of them) were managed (patching, antivirus, credentials)?
Obviously, investigations are still at a very early stage. Various unconfirmed rumours on Internet are already spreading (VBScript virus). The IT staff, the vendors and the French government are working on it, and I hope we will see a detailed technical report in the near future.
Finally, this is one more reminder of the importance of some basic security principles in corporate environment:
- Reduce Internet exposure to the minimum
- Secure all exposed systems (SFTP and co.)
- Use DMZ & Bastions servers
- Install, configure, maintain and monitor IDS / IPS
- Apply patches & anti-virus, even on third-party systems
- Perform vulnerability scanning and penetration tests
- Educate users
Thanks – Pierre-Olivier Blu-Mocaer – FixSing Consulting
12/04/2015 : Reflets.info (French online journalism platform) mentioned this analysis here : https://reflets.info/piratage-de-tv5-monde-acte-3-grosses-boites-noires-et-sentiment-de-securite/
12/04/2015 : Added IDS / IPS recommendation ( Thanks @bluetouff )
14/04/2015 : LeMagIT (French IT magazine) mentioned this analysis here : http://www.lemagit.fr/actualites/4500244235/TV5-Monde-sur-la-piste-des-systemes-de-diffusion
15/04/2015 : Analysis mentioned in Silicon.fr : http://www.silicon.fr/tv5-monde-revelateur-failles-securite-teles-113904.html
14/05/2015 : One month after the attack, still no detailed report available. However, the latest Shodan’ scan shows that a large cleanup has been performed on the external firewall(s) rules.
Great analysis 🙂 I’m not working in such a big TV organization but I’ve been working in smaller video production & TV broadcast companies for 15+ years and here are some of my own guesses and comments.
First, having an internet access on video production systems is now very common. Back in the 90s it was forbidden to connect a video production/editing station to the internet (there were mostly Avid turnkey stations running under Mac OS 9 or Windows NT). They were all using specific hardware and drivers, and even network cards were specific cards.
But now is a completely different story. Video workstations are often connected to different LANs, at least one being the corporate network, and other a private LAN connected to the vendor’s SAN. Having an internet access is now part of those environments. It’s not recommended, but it’s the case. The biggest turnkey systems of the 90’s now run on regular hardware setups that became standard workstations years ago 🙂
Antivirus are not recommended because the maximum performance is needed on those workstations.
I can also confirm than WeTransfer and other public clouds are used to transfer data between remote locations and the main video production company (I even saw MEGA used). The reason is also simple: even if you provide a private server (like FTP, SFTP…), most people on the other end of the line won’t even understand how to use them. Download, configure and use even an FTP client is not an effort a lot of people want to make anymore. They don’t even understand how to use it. Add to this than quite often the transfer does not work very well between some countries or networks because of bandwidth or peering issues, and that you often have to transfer very large file that gets corrupted during transfer… and in the end you get an e-mail saying basically “f*ck your system, here is a WeTransfer link, deal with it”.
So if you need to install any client other than a web browser to transfer data, your brilliant server infrastructure won’t be used at all.
Okay now to the antivirus/firewall and Windows stuff. In my own experience, vendors of broadcast servers do sell “blackboxes” running under quite any Windows version you can think from Windows XP to the most recent Windows Server. This is not a joke, for instance in France there is Windows XP boxes in every TV broadcast facility, in the most critical location into the broadcast process – I won’t tell you why, but it’s true. Video servers also often run under Windows (Server mostly) and you also get IP video encoders & muxers running under Windows.
And yes, there is absolutely no antivirus, no firewall and no update made to these boxes. Why? Well, it’s a production environment. Those boxes run 24/7 during years until they’re replaced. If they work, you don’t need to update them, and you will NOT. Sometimes they have issues (like: know to crash after X days of uptime), so okay, you know the issues, the vendor knows the issues, and you do use workarounds (like: reboot each X-7 days of uptime). But you don’t patch anything, never, because else a lot of different issues may arise, and your box may also just stop to work because of specific software/hardware installed. And in any case, it’s the vendor’s job, not yours. Your box is certified to work under environment X and you don’t want to update to an uncertified environment X.0.1.
What else? Well, I saw somewhere that they were using an Avid workstation (running on Windows) with a “Media Offline” message. That means that somewhere there is an Avid media server (a SAN I guess). Okay. Then somewhere else, you need to have a way to push the final media to the broadcast server. After that, the video signal outputs through SDI (a point to point video protocol) and runs through various encoders to make MPEG-TS IP streams. From that point it’s all network stuff (hence the DCIM, to handle the various streams). Note that it’s often specific network stuff, that regular network people are not used to handle. Video streams are quite different to handle than a regular TCP-IP LAN. That’s also why network administrators relies on external contractors to install & maintain all the “video stuff”.
My guess, and it’s only a guess, is that the hackers wanted to have a look inside the network to get access to the social network accounts of TV5, that they ran various scans and found many Windows boxes or routers with vulnerabilities (I also didn’t mention it but all those boxes run http and SMTP servers, often very outdated). Linux boxes are very uncommon in the broadcast world. OSX boxes might be found, mostly for video conversion farms, they’re also not updated for the same reasons.
They might have found the equivalent of Disneyland for hackers.
FYI a few years ago a big French broadcast facility was infected by a Windows virus (I heard i came through an infected USB key used to transfer video media). A lot of video boxes were infected, and some TV channels stopped broadcasting because of that. It wasn’t even an attack, only a regular virus.
Oops, I made a mistake: I meant SNMP servers, not SMTP servers. Sorry 🙂
Merci beaucoup pour ces analyses dignes de ce nom!
“Disneyland for hackers” : yes, I really think we are in this case !
[…] TV5 Monde – a (tentative) technical analysis […]
[…] A (tentative) technical analysis of TV5 Monde infrastructure following the recent hack […]
[…] vous invitons à lire cette excellente analyse. De plus, on se propose dans la suite de cette article de faire un petit comparatif avec la […]
Fascinating analysis. As a supplier of one of these boxes sitting in the broadcast room, running Windows Server – our recommendation to IT people is not to run patches, antivirus – else it impacts severely the performance of the system, or even breaks functionality.
Our application is very sensitive to file access timing, CPU scheduling etc… We are dealing with high quality video and audio – and timeliness is critical.
After reading this article, we will have to go back to the white board and see how to design a hardened server, while still keeping the quality of service required. This is not an easy task, but in an interconnected world, you have to provide a fully secured service.
Thanks for this eye opening article.
[…] 11/04/2015. TV5Monde – A (tentative) technical analysis : http://www.fixsing.com/tv5monde-a-tentative-technical-analysis/ 14/04/2015. Une Cyber-facture pour TV5 Monde? : […]