Apr 09

(via xkcd: Heartbleed)

(via xkcd: Heartbleed)

Drones on demand -


Gofor imagines a future world where drones are cheap and ubiquitous. What sorts of things would we have personal drones do for us? Follow us home in unsafe neighborhoods? Personal traffic copters? Travel location scouting?

How long before someone uses a personal drone for the same…

Apr 07

“What we have seen is that a logically centralized, hierarchical control plane with a peer-to-peer data plane beats full decentralization,” explained Vahdat in his keynote. “All of these flew in the face of conventional wisdom,” he continued, referring to all of those projects above, and added that everyone was shocked back in 2002 that Google would, for instance, build a large-scale storage system like GFS with centralized control. “We are actually pretty confident in the design pattern at this point. We can build a fundamentally more efficient system by prudently leveraging centralization rather than trying to manage things in a peer-to-peer, decentralized manner.” — Google Lifts Veil On “Andromeda” Virtual Networking

Apr 05

“A collaboration between a Stanford ant biologist and a computer scientist has revealed that the behavior of harvester ants as they forage for food mirrors the protocols that control traffic on the Internet.” — Stanford biologist, computer scientist team up to discover the ‘anternet’

Apr 04

A reliable storage system is one that can be trusted to perform well under all states of operation, and that kind of predictable performance is difficult to achieve. In a predictable system, worst-case performance is crucial; average performance not so much. In a well implemented, correctly provisioned system, average performance is very rarely a cause of concern. But throughout the company we look at metrics like p999 and p9999 latencies, so we care how slow the 0.01% slowest requests to the system are. We have to design and provision for worst-case throughput. For example, it is irrelevant that steady-state performance is acceptable, if there is a periodic bulk job that degrades performance for an hour every day.

Because of this priority to be predictable, we had to plan for good performance during any potential issue or failure mode. The customer is not interested in our implementation details or excuses; either our service works for them and for Twitter or it does not. Even if we have to make an unfavorable trade-off to protect against a very unlikely issue, we must remember that rare events are no longer rare at scale.

With scale comes not only large numbers of machines, requests and large amounts of data, but also factors of human scale in the increasing number of people who both use and support the system. We manage this by focusing on a number of concerns:

- if a customer causes a problem, the problem should be limited to that customer and not spread to others
- it should be simple, both for us and for the customer, to tell if an issue originates in the storage system or their client
- for potential issues, we must minimize the time to recovery once the problem has been detected and diagnosed
- we must be aware of how various failure modes will manifest for the customer
- an operator should not need deep, comprehensive knowledge of the storage system to complete regular tasks or diagnose and mitigate most issues

And finally, we built Manhattan with the experience that when operating at scale, complexity is one of your biggest enemies. Ultimately, simple and working trumps fancy and broken. We prefer something that is simple but works reliably, consistently and provides good visibility, over something that is fancy and ultra-optimal in theory but in practice and implementation doesn’t work well or provides poor visibility, operability, or violates other core requirements.

” — Manhattan, our real-time, multi-tenant distributed database for Twitter scale | Twitter Blogs

Apr 03

“I think enterprises should care more about full stack independence to manage vertical risk. Why? When a technology is vertically prescriptive (i.e. Pivotal CF One requiring VMware), it hurts an organizations’ ability to manage supply chain risk by ensuring they can source vendors to fulfill needs at other parts in the stack. This means that a customer loses cost control leverage, feature leverage, and can be held hostage in the face of overall stack quality degradation. By optimizing for vertical independence, an enterprise ensures that vendor selection at any tier in the IT stack does not dictate what vendors they use above and below that part of the stack.” — The LIES That Come With Some Flavors of PaaS

“If you don’t have a strong VP Engineering who can manage estimates, task completion, assignments and kill scope creep you’re dead.” — How to Deliver More Software Projects On Time

Mar 31

“It used to be that the physical hardware was orders of magnitude more expensive than engineers but this hasn’t been true for decades now - it’s perfectly reasonable to look for ways to reduce yours costs especially if it can be done quickly but obsessing over hardware costs, especially while you’re still growing, is a red herring. Building large systems is tough and the fewer things you have to worry about the better - using AWS reduces the chance that you will run into a scenario where you’re just not able to do something without changing your host and rewriting your architecture.” — AWS is about infrastructure optionality

Mar 25

“A great office helps with recruiting. It helps with productivity and output. It helps with employee retention. A great office helps with the intangibles of “well being” that are psychologically hidden in the lower levels of Maslow’s Hierarchy of Needs. A good office / work environment is the foundation of establishing a strong company culture and team spirit. It’s not everything – but it helps.” — When Does Establishing a Good Startup Culture Outweigh Being Cheap?

“When I joined, it was it was a culture shock,” said Frankovsky. “Being a hardware guy at such a fast-moving software company. It was like if you’ve ever jumped out of a fast-moving boat, you get the wind knocked out of you when you hit the water. That’s what it felt like to be a slow-moving hardware guy with all of these software people who were used to iterating every few weeks. That’s why we started Open Compute so we could speed up the development of hardware.” — Facebook’s Open Compute guru Frank Frankovsky leaves to build optical storage startup — Tech News and Analysis