VoDKA project

Video on Demand Kernel Architecture

vodka@lfcia.org

R LFCIA
FEDER Xunta UDC


VoDKA's History

This project was started on January 2000 under the umbrella of an European RDF (FEDER) Project where the cable & telecommunications company R acted as a partner of the LFCIA Lab. Additional effort has come from two other projects: one supported by the Galicia Government (Xunta de Galicia) - performance modeling - and the other by the University of A Coruña - design patterns.

On November 2000, the first version of the server: ERISTOFF (Early Release Initially Starting To Offer Functional Fun), previously known as ERICSStoff, was finished.

The second version was called SMIRNOFF (Second Monitoring/Marvelous and Improved Release Now Offering Further Features) and its first distillation was finished on March, 2001. The second distillation was ready on June 2001, and offered additional functionality and scalability. The third distillation was the first publicly tested, serving multimedia contents to the campus network.

A public press release was published on July, 12 2001. See the articles published in the following newspapers: El Ideal Gallego (announce), (article), La Opinion (1), (2), La Voz de Galicia (article), and Expansión (article).

From January, 2002 to July,2002 several trials and small scale testbeds took place, streaming video from a VoDKA server composed of several large storage nodes at a central place in the academic network to several caches both at the academic and external networks, each feeding one or several streamers. The current test nodes are placed at a variety of sites, ranging from our local gigabit backbone to the academic ATM backbone and a test network at the telecom operator. Clients have also been tested on a variety of platforms and connections, including DOCSIS cablemodems with and without bandwidth control. Further testing work is underway with the cooperation of R.

During 2003 work has been done in supporting different clients - from PACE set-top boxes, ISMA compliant players, and GCT Allwell reference designs to TV-over-IP hardware solutions. In addition, and funded by Spanish Ministry of Science and Technology (TIC 2002-02859), most of the system has been rewritten paying attention to distributed functional patterns identified in several applications. Hence, the quality of the system has been notably improved.

On February, 13th 2003, VoDKA was awarded the project of the year prize in the VII Galician ETI (Information Technologies Meeting) . ETI is organized by Computing magazine and PricewaterhouseCoopers with support from Sun Microsystems. We updated the public press release to include this award. Some articles on this event were published on El Ideal Gallego, Diario de Ferrol, Faro de Vigo, La Opinión, O Correo Galego, Expansión Galicia, La Voz de Galicia, and others.

In December, 2003 a spin-off company was created to further develop, industrialize and market VoDKA: LambdaStream. The research group at LFCIA (now MADS Group keeps a continued relationship with LambdaStream for further research projects

Main Goals
  • To build an extremely scalable, fault tolerant, multiprotocol, adaptable (to the network topology and end-user protocols) streaming server.
  • To use commodity hardware (Beowulf) clusters for the underlying architecture.
  • To use the functional programming paradigm and design patterns for the analysis, design and implementation of the distributed control system.
  • Scope Proposal and Services Model (Spanish) (PDF, PostScript) (note: This is obsolete nowadays, but stays here for completeness and perspective)
Brief Description

A Video-On-Demand (VoD) server is a system that provides video services in which a user can request a specific piece of video information (film-on-demand, remote learning, home shopping, interactive news,...) at any time. The growing demand of such services suggests the design of scalable VoD servers, both in storage capacity and bandwidth. A hierarchical storage system, based on a cheap Linux cluster , is proposed in this project to meet these requirements. The project is not focused in the end-user media delivery protocols, but in the distributed, extremely scalable, cost-efficient storage, location and transport of media objects.

The VoDKA server is composed of a set of distributed modules that can be deployed throughout a network of any size. Each module has a specialized purpose, like tape or disk storage, media caching, protocol-specific streaming, protocol adaptation, media conversion... plus negotiator modules whose purpose is to decide optimal media sources and routes, scheduling, cache placing...

As of this writing VoDKA implements modules performing on-disk storage, on-tape storage, HTTP storage, WEBDAV storage, erlang message passing transport, TCP transports, reliable multicast inter-node transport, multi-node and multi-level media caching, constant bitrate HTTP streaming, ICEcast streaming, RTSP/RTP streaming and RTP multicast, and SNMP management. All the modules are hot-pluggable and hot-swappable. Right now work is ongoing to add more streaming protocols and further fault tolerance features.

The concurrent and distributed language Erlang has been chosen for programming the prototypes as well as for most of the modules of the final system. Erlang is a functional language originally developed by Ericsson, that has suitable features for the implementation of soft real-time fault-tolerant distributed processing systems. For example, some development tools, some libraries for the creation of graphical user interfaces, and even a distributed database (Mnesia), have been used for this project. Currently many other companies like Alteon, (now part of Nortel Networks) Cellpoint, Corelatus, or One2One (now part of T-Mobile) use it. And, of course, many Ericsson products like the AXD301 and ANx are developed in whole or part in Erlang.

Funded Projects
  • ERDF/FEDER (1FD97-1759) Project Leader: J. L. Freire (2000-2001)
  • Galician Government (PGIDT99COM10502) Project Leader: Alberto Valderruten (2000-2001)
  • University of Coruña (2000-5050252026) Project Leader: Víctor M. Gulías (2001)
  • Pilot funded by R (Fundacion Universidade da Coruña) Project Leader Víctor M. Gulías (early 2002)
  • Spanish Ministry of Science and Technology (TIC 2002-02859) Project Leader: Víctor M. Gulías (2003-2005)
Original ERDF/FEDER Project Team
VoDKA Developers
Master's thesis done in related topics
  • Juan J. Sánchez Design of a scheduling subsystem --Erlang-- (Gulías, Sept 1999)
  • José M. Casanova Web-based end-user frontend --Java-- (Gulías, Sept 2001)
  • Rubén López RTP/RTSP Front-end Adapter --C-- (Gulías, Sept 2001)
  • Carlos Abalde Distributed payment system --Erlang-- (Gulías, Sept 2001)
  • José R. Gulías e-Commerce distributed application --Erlang-- (Freire, July 2001)
  • Iago Elizechea Personal media management --Java-- (Gulías, Sept 2002)
  • Javier París Network filesystem optimizations for streaming clients --C-- (Gulías, Sept 2002)
  • Carlos Varela SNMP Management --Erlang-- Gulías, Sept 2002)
  • David Iglesias MOV/MPEG4 SMIL composition tool --Java-- (Gulías, Febr 2003)
  • Samuel Rivas Linux set-top box client software --C-- (Gulías, Jul 2003)
  • Miguel Villares MHP middleware for an IP-based set-top box --Java-- (Gulías, Jul 2003)
  • Alfonso Rivero VoDKA Optimizations for peripheral clients --Erlang, Java-- (Gulías, Jul 2004)
  • Manoel Polo Design and Implementation of an ad-hoc DRM system for VoDKA --Erlang, C-- (Gulías, ??)
Public Diagrams
Note: these diagrams are extremely aging right now, and will be replaced as soon as the user documentation is completed. Ignore them.
  • ERISTOFF network topology (PDF)
  • SMIRNOFF two main modules (Management Subsystem & VoDka Subsystem) (PDF)
  • SMIRNOFF two level (storage & streaming) hierarchy schema (Artistic, PDF, Animated PDF)
  • SMIRNOFF flexible configuration with n levels (PDF)
  • SMIRNOFF distributed configuration example: two streaming levels and shared cache & storage levels (PDF)
  • SMIRNOFF monitoring schematics:
    • General monitoring philosophy (PDF)
    • Monitoring of pipes example (PDF)
Presentations
Publications
  • Víctor M. Gulmas, Carlos Abalde, Juan J. Sanchez Lambda goes to Hollywood Fifth International Simposium on Practical Aspects of Declarative Languages (PADL'03). Lecture Notes in Computer Science 2562. New Orleans, USA. January 2003.
  • Carlos Abalde, Víctor M. Gulías, Josi L. Freire, Juan J. Sanchez, A Cluster-based Payment Gateway System Developed Using a Distributed Functional Language. The First Eurasian Conference on Advances in Information and Communication Technology. Workshop on Formal Modeling of Intelligent Peripheral Systems (EurAsia ICT 2002). Proceedings of the First Eurasian Conference on Advances in Information and Communication Technology. Shiraz, Iran. October, 2002.
  • Thomas Arts, Juan J. Sanchez, Global Scheduler Properties derived from Local Restrictions. ACM Sigplan Erlang Workshop at the Principles, Logics, and Implementations of high-level programming languages (ACM Sigplan Erlang Workshop at PLI2002). ACM Sigplan Proceedings of the Erlang Workshop 2002. Pittsburgh, USA. October, 2002. Ed. ACM.
  • Juan J. Sanchez, Miguel Barreiro, Vmctor M. Gulías, Josi L. Freire, Exploiting Sequential Libraries on a Cluster of Computers. Erlang Workshop at the Principles, Logics, and Implementations of high-level programming languages (Erlang Workshop at PLI2001). Proceedings of the Erlang Workshop 2001. Florence, Italy. September, 2001. Ed. Bjarne Ddcker.
  • Juan J. Sanchez, Miguel Barreiro, Vmctor M. Gulías, Josi L. Freire, Javier Mosquera, Functional Scheduling in a Distributed VoD Server. 13th International Workshop on the Implementation of Functional Languages (An Erlang-based Hierarchical Distributed VoD System. Erlang User Conference, Stockholm, 2001
  • Alberto Valderruten, Víctor M. Gulías, Juan J. Sánchez, José L. Freire, Javier Mosquera, Implementación de un modelo de monitorización para un servidor de Vídeo bajo Demanda en Erlang. CLEI, Mérida, Venezuela, 2001
  • Miguel Barreiro, Víctor M. Gulías, Javier Mosquera, Juan J. Sánchez, Utilización de programación funcional distribuida y clusters Linux en el desarrollo de servidores de vídeo bajo demanda. SIT, A Coruña, 2001
  • Miguel Barreiro, Víctor M. Gulías, Juan J. Sánchez, Santiago Jorge, The Tertiary Level in a Functional Cluster-based Hierarchical VoD System. Eurocast, Las Palmas, 2001
  • Miguel Barreiro, Víctor M. Gulías, Juan J. Sánchez, A monitoring and instrumentation tool writen in Erlang. Erlang User Conference, Stockholm, 2000
  • Miguel Barreiro, Víctor M. Gulías, Juan J. Sánchez, Monitoring and Instrumentation of a Clustered VoD System (poster). Erlang User conference, Stockholm, 2000
  • Juan J. Sánchez, Javier Mosquera, Víctor M. Gulías, Alberto Valderruten, State of the Art and Design of VOD Systems. SCI/ISAS, Orlando, 2000.
  • Miguel Barreiro, Víctor M. Gulías, Cluster Setup and its Administration. In Rajkumar Buyya, editor, High Performance Cluster Computing, volume I. Prentice Hall, 1999
  • Miguel Barreiro, Víctor M. Gulías, Juan Quintela, Antonio Blanco, José L. Freire, Experiencias con un cluster Beowulf. SEID, Santiago, 1999
Related Links

Links have been moved to a separate page, since they were becoming huge.

vodka@lfcia.org