Amazon held another event focused on educating start-ups and existing tech companies about their web services offerings and how to integrate them into they current (or new) businesses. I attended a similar event previously at Mezzanine in SF, but this time around we’re in the Sir Francis Drake Hotel (near Union Square).
So far I would estimate around 90 – 95% males, and perhaps even a higher than SF-usual ratio of about 90% Mac laptops in use.
NOTE: You know you’re at a “corporate” rather than a “community” event when the only wifi available requires payment…
Opening Statements – Adam Selipsky, VP AWS
It all started with Amazon building our own application (amazon.com), and needing to scale it, provide performance, etc. We spent over a couple of billion dollars to provide the infrastructure. We learned lessons:
- Moved away from expensive hardware
- Moved to standard hardware and lots of it
- Service-based architecture
Perhaps other people have the same or similar problems? So we started talking internally and externally about what we could provide. Asked people what’s important for all of this stuff
- Reliability: It has to Just Work, not necessarily 100% of the time, but close,
- Scalability: Upward and downward scalability,
- Performance: It has to operate as quickly as if it were all in one data center
- Simplicity: don’t make people have to learn too much (make it as simple as possible)
Enabling business to build and get paid for scalable, high-performance applications. Good results so far, including over 400,000 registered developers and S3 contains over 22 billion objects. Why do people love this stuff so much?
- Get rid of the muck: don’t do things you don’t want to do or can’t do,
- Focus: allows you to focus on product development, marketing, sales strategy etc,
- Time to Market: allows start-ups to shave 2-6 months off delivery cycles (provisioning, configuration, etc)
Q: Is this a serious business or an experiment?
A: This is not an experiment, this is Amazon’s third business line. Retail Business, Seller Business (1.4 million active sellers) and now Developer Business.
Mention of Amazon Start-Up Challenge and invite people to check it out.
AWS Presentation – Jeff Barr, Sr. Evangelist AWS
As a consultant, I was invited to a small little conference hosted by Amazon. I was one of about 5 or 6 other freelancers there. I realized that they were talking about taking the covers off amazon.com and making the core of some of their system available to “everyone”. I was hired into a slightly different position and around 15 – 20% of his time was to “help out” when he could with AWS.
Summary of the 3 main business streams of Amazon.
Bandwidth consumed by AWS has surpassed the rest of all Amazon properties combined.
Why do people like this so much?
The “muck” or “undifferentiated heavy-lifting” can take up 70% of the time involved in building and scaling an online application.
Talks about the differential between projected usage and real usage, and how if you (traditionally) purchased assets to handle your projections, you will almost always have waste (or a deficit!).
Mentions “being famous” as showing up on: Techmeme, Reddit, Digg, Slashdot, TechCrunch
Amazon doesn’t usually see too many spikes across all users because of the averaging out effect of all of their clients’ usage cycles.
AWS principles: Easy to use, Fast, Elastic, Highly available, Secure, Pay as you go
Summaries of EC2 (+ EBS), S3 SimpleDB, DevPay, SQS, FPS, MTurk.
81 million Amazon.com users who you can enable to pay through FPS/DevPay.
Look at Podango, who have an impressive system that transcodes podcasts and automatically re-assembles them all with new advertising, announcements and raw content (daily). Everything is powered by AWS and automatically scales up and down to handle load (transcoding/assembly).
What has happened in the last year? FPS, DevPay, Elastic IPs, Availability Zones, Large Instances, CPU-Intensive Instances, EBS, S3 in EU, SimpleDB additions, Premium Developer Support, Service Health Dash, OpenSolaris/MySQL, plus more.
Trends We’re Seeing
- Systems using more than one service
- Massive datasets and large-scale parallel processing
- Enterprise adoption
- Increased need for support and transparency
- Running more sophisticated software in the cloud
Future Roadmap
- Security features and certifications
- Focus on operational excellence
- US and Int’l expansion
- Localization of technical resources (translations etc)
- EC2 GA (general availability) and SLA (we’re close)
- Windows Server support
- Additional services (we’re not anywhere near done!)
Start-Up Challenge
Over 900 entries last year in the competition to win $100,000 (cash + AWS credits). Ooyala won with their video content management system. Entrants are judged on:
- Creativity and Originality
- Likelihood of long-term success
- How well it addresses a market need
- Usage of AWS
Customer Presentations
Tap in Systems – Peter Loh
IT monitoring and systems management, powered by AWS. Their specialty is heterogeneous environments. They are recently funded and are now available under a limited beta. Monitoring usually comes last (requirements –> application –> monitoring), even though it’s a critical piece of the puzzle.
Deploy a monitoring solution in the cloud, capable of monitoring cloud devices or internal IT systems. Works like enterprise monitoring – gathers stats from all your systems, compiles them and provides reports. Their custom-developed systems are open source and will be released back to the community. Everything is priced per usage.
Provide all sorts of custom adapters/components to cater to specific services and their relevant metrics. Runs with Nagois, Ganglia, Cacti, Linux, Windows etc! They also ensure the security of your information because the EC2 instance runs under your AWS account, not theirs (they never see your data, just provide the engine to “do stuff with it”).
Vertica – Jerry Held
Grid-based, columnar DBMS for data warehousing and analytics. Founded in 2005 and now has 50+ customers worldwide, with great leadership/advisors. Moving some of their business to the cloud has provided a whole new playing field to expand their business etc.
Vertica provides:
- Scale out grid architecture
- Aggressive data compression
- Automatic high availability (failover, replication and recovery all in the cloud)
Some examples of what people are using it for:
- Analytic Software as a Service (companies providing specialized/assisted analysis of data)
- Short-term Analytics (competitve analysis, marketing campaign metrics, event post-mortem analysis etc)
- Frictionless Vertica Proof of Concepts (using the cloud lowers costs for Vertica AND for their customers)
- Vertica R&D (stress testing, benchmarking, etc)
“Don’t look at what you used to do, and figure out how to do it cheaper and faster in the cloud. Look at things that couldn’t be done before, but how this technology allows us to do something fundamentally different.”
Animoto – Brad Jefferson
Completely automated videos based on photographs and music. The Animoto engine analyzes the audio and syncs the video with the nuances of the music, rendering a completely custom video every time. It takes about 5 minutes to render a 30 second Animoto clip.
Halted development for 3 months and converted everything over to Amazon’s infrastructure. Probably the best decision they’ve made as a company. They use EC2, S3 and SQS. They don’t own a single server, and all work from laptops.
They have Editor, Producer and Renderer Queues to handle each component of the system, based on how it works in the real world.
Facebook application absolutely spiked them out – around 5 instances up to 3,600 instance in a matter of days. 25K users to 700K users on Facebook. It would have taken them 7,000 servers to handle this traffic spike.
Elastra – Stuart Charlton
A suite of software applications and components, provisioned and managed in the cloud using technologies created by Elastra. A company that’s around a year old, with 20-ish people. Software can manipulate hardware – game-changing.
Giving the system architects the direct ability to provision and manage the pieces of the system they design. EDML and ECML provide abstracted configuration of how this will all work together.
Q&A
Q: What about intellectual property rights etc?
A: It’s your data, it’s your code. We don’t look at it and we claim no ownership over it.
Q: Do you use Amazon for storage as well, or just for processing?
A: Everything is stored on S3.
Q: What is missing from AWS?
A: CDN performance, latency/initial data across the wire, EBS solves a lot of problems, Elastic IPs are a little limited, DevPay could use some more granularity on its reporting, a better way to handle software licensing.
Q: How are your experiences using EBS vs S3
A: Similar to Enterprise SAN or Network Storage. [Elastic Fox]. You can stripe, RAID, and performance is close to the local ephemeral storage. It will be the primary way that we configure our systems.
Q: How can you speed up the “spin-up” time for these services dealing with large volumes of data.
A: We’re looking at parallel loading of content using multiple instances to speed things up.
Salil Despandi – Bay Partners
We always ask if people are using EC2/S3. If not, why not? What about your architecture prevents you from working on AWS infrastructure? As a startup, you’re in a good position to re-architect so that you can.
“Java has become the COBOL of the 21st century.”
Issues (a year ago)
- ephemeral storage (now EBS!)
- cookie-cutter-ness of machines (more options now)
- scale out, not up (can get bigger sizes though)
- IP stability etc
- still very machine focused (not minutes/hours/etc)
- commercial software licensing is an unknown
- configuring and managing thousands of nodes is still difficult
Opportunities
- tools & higher layers of infrastructure (fault tolerance, load balancing, dashboard, etc etc)
- higher levels of service (like engineyard.com for RoR, ways to get data in, CDNs, security, etc)
- every layer of the stack, offered as a distributed scalable service in the cloud
- AWS-compatible private clouds
- open source and cloud intersection
- big ecosystem
Cocktails/Networking
Cheese/bread/fruit platter provided. Hors’ dourves came around care of the SFD staff and there were a few different beers and a red/white wine available. Not too bad at all. I had a few interesting conversations with a few interesting people, and it seemed like others were doing the same. Cards I collected were from people at:
- Right Scale: really interesting services providers who help you manage all your AWS stuff
- FathomDB: upcoming provider of database services, backed by AWS technologies (MySQL on EC2/S3)
- rPath: software appliances powered by AWS
- syncplicity: interesting synchronizing service that syncs not just web <—> desktop, but also desktop <—> web applications (think Flickr, Facebook, etc).
- Shoppertron: provides white-labeled storefronts which aggregate big providers (eBay, Shopping.com, etc) and integrate smaller providers as required
- Carticipate: cool iPhone App which helps you find car-pooling/car-sharing buddies who are near you, going in your direction.
Feedback
I find these specific events to be a great “intro to AWS”, but personally I’d love to see them do some more advanced events where they are quite technical and explore, in depth, some of the solutions customers have built, what their architecture looks like etc. I talked with Tracy Laxdal (AWS Marketing Manager) at length about this, and about how a lot of the people using their services are developers, and could benefit from more education on the systems architecture side of things.
In retrospect, I think there is also some great value to be had (for AWS and for customers) to do some perhaps on-the-spot case studies of looking at how someone has an application built, and how they could take what they have and implement more/other Amazon services to enhance their architecture and improve on what they’ve got. This could even be done using existing web apps as examples, and breaking down how you could build it on AWS. For example, you could take Flickr.com and talk about spinning up instances for handling image uploads, resizing etc, then look at a possible architecture of databases and web servers, fault tolerance at all levels, etc.