Hadoop Backend
A few friends knew that we chose Hadoop as the framework for FreeWheel's backend have been asking me: so, is it working out for you? How do you like it? Why have not you blogged about it yet? — Here you go.
Being at DoubleClick, built massive scale data aggregation system framework several times over taught me that I should try really hard to avoid it if possible. I was determined since day one that we were not going to build our own framework. We have our advantage, being a startup in video market, the traffic at beginning is going to be low comparing to established high traffic internet companies. We would have enough time to play around with a generic framework even if it turns out not as efficient.
I don't remember who first discussed Map/Reduce with me, maybe Vince, the best friend of mine in the last 10 years. Vince was the adserver architect at DoubleClick, who recently joined one of Dwight's companies. When I was checking out open source data aggregation framework, Vince told me about Map/Reduce. It seems to be what I wanted. The first day that my backend team was formed, I told them to investigate it. They ended up choosing Hadoop, it turns out to be the same framework that folks at DoubleClick use for a recent experimental project as well — Great minds think alike
For what we wanted, it worked out great. Without this framework, we would have to settle with a prototype data aggregator, which would need to be rebuilt fairly quickly. Hadoop not only provides the framework fundamentals enabled us to focus our resource on business logic, it also comes with a monitoring tool that satisfied our needs. We integrated it with Orion IpMonitor since day one so that we don't fly blind. Monitoring is such an important piece of running ASP, we don't treat it lightly.
Being at DoubleClick, built massive scale data aggregation system framework several times over taught me that I should try really hard to avoid it if possible. I was determined since day one that we were not going to build our own framework. We have our advantage, being a startup in video market, the traffic at beginning is going to be low comparing to established high traffic internet companies. We would have enough time to play around with a generic framework even if it turns out not as efficient.
I don't remember who first discussed Map/Reduce with me, maybe Vince, the best friend of mine in the last 10 years. Vince was the adserver architect at DoubleClick, who recently joined one of Dwight's companies. When I was checking out open source data aggregation framework, Vince told me about Map/Reduce. It seems to be what I wanted. The first day that my backend team was formed, I told them to investigate it. They ended up choosing Hadoop, it turns out to be the same framework that folks at DoubleClick use for a recent experimental project as well — Great minds think alike
For what we wanted, it worked out great. Without this framework, we would have to settle with a prototype data aggregator, which would need to be rebuilt fairly quickly. Hadoop not only provides the framework fundamentals enabled us to focus our resource on business logic, it also comes with a monitoring tool that satisfied our needs. We integrated it with Orion IpMonitor since day one so that we don't fly blind. Monitoring is such an important piece of running ASP, we don't treat it lightly.
Comments