When you add nodes to a cluster, they start out empty. They work, but the data for them to work on isn’t co-located, so it’s not very efficient. Therefore, you want to tell HDFS to rebalance.
After adding new racks to our 70 node cluster, we noticed that it was taking several hours per terabyte to rebalance the nodes. You can copy a terabyte of data across a 10GbE network in under half an hour with SCP, so why should HDFS take several hours?
Z-Filters is a technique for listening to what millions of people are talking about.
If a hundred million people were to talk about a hundred million different things, making sense of it would be a hopeless task, but that’s not the way society operates. The number of things people are talking about at any given moment is much, much smaller— thousands, not millions—and by limiting results to subjects that are newly emerging or newly active, a comprehensive view of “what’s new” can be seen at approximately the data-flow of the Times Square news ticker.
hadoopoopadoop 