BIG Data Management on Apache Hadoop

Yahoo services in all areas – advertising, Web search, content – depend for their quality on analysis of data fed back from the serving systems. For example, logs from Web servers and other front-end as well as back-end systems. This is joined with dimensional data along with catalogs and indexes used to serve ads, search results, and customized content. The Hadoop Cluster makes orders of magnitude more of this data available in one place together with extensive computation resources. This enables teams to continuously optimize and improve the serving quality. As the number of data sources and data sets to be mirrored on the Hadoop Clusters has increased, the need for a common platform offering robust automation has become apparent. This data is collected and brought to the Hadoop Cluster using multiple steps in a pipeline. Our goal is to make sure this data reaches Hadoop Cluster in accordance with committed SLAs for latency and fidelity. The Data Management solution will automate the movement (Data In, Out, & Copy) and lifecycle management of data (Retention, Anonymization, Compliance Archival, etc.) on the Yahoo Hadoop Clusters. The solution addresses the problem of loading thousands of distinct data sets to a growing number of clusters in multiple data centers. It meets latency and data quality SLAs while requiring minimal operational staff and allows scaling with Hadoop. It helps the vast majority of Hadoop Cluster users depending on regular data availability with increased reliability.

- by Seetharam Venkatesh

Author`s Bio:
BIG Data Expert Hadoop Veteran 15+ years of industry experience