{"componentChunkName":"component---src-templates-tag-jsx","path":"/blog/tag/engineering/3/","result":{"data":{"prismic":{"allFeaturedblogs":{"edges":[{"node":{"featured_blogs_enabled":true,"heading":[{"type":"paragraph","text":"Featured posts","spans":[]}],"featured_blog_1":{"__typename":"PRISMIC_Blog","_linkType":"Link.document","blog_header_image":{"dimensions":{"width":790,"height":395},"alt":null,"copyright":null,"url":"https://images.prismic.io/www-static/6d8d81b1-971a-4313-b033-b4e125cb14a0_MondoDB-blog-header-790x395.PNG?auto=compress,format"},"blog_headline":[{"type":"heading1","text":"Introducing DigitalOcean Managed MongoDB – a fully managed, database as a service for modern apps","spans":[]}],"blog_post_date":"2021-06-29","blog_post_content":[{"type":"paragraph","text":"MongoDB is one of the most popular databases, and it’s ideal for apps that evolve rapidly and need to handle huge volumes of data and traffic. It offers advantages like flexible document schemas, code-native data access, change-friendly design, and easy horizontal scale-out.","spans":[{"start":22,"end":44,"type":"hyperlink","data":{"link_type":"Web","url":"https://db-engines.com/en/ranking","target":"_blank"}}]},{"type":"paragraph","text":"However, building and maintaining MongoDB clusters from the ground up can be a huge undertaking. Developers often complain that they have to spend their valuable time and resources on database management. Well, we’ve been listening and have some great news: accessing and managing MongoDB on DigitalOcean just got a lot simpler!","spans":[]},{"type":"paragraph","text":"We are excited to announce that DigitalOcean Managed MongoDB is now in General Availability. Managed MongoDB is a fully managed, database as a service (DBaaS) offering from DigitalOcean, built in partnership with and certified by MongoDB Inc. It provides you all the technical capabilities that make MongoDB so beloved in the developer community. Together we have ensured that you will get access to all the latest releases of the MongoDB document database as they become available.","spans":[{"start":32,"end":91,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.digitalocean.com/products/managed-databases-mongodb/"}},{"start":230,"end":241,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.mongodb.com/","target":"_blank"}}]},{"type":"paragraph","text":"Managed MongoDB simplifies the MongoDB administration. Developers of all skill levels, even those who do not have prior experience in databases, can spin up MongoDB clusters in just a few minutes. We handle the provisioning, managing, scaling, updates, backups, and security of your MongoDB clusters, allowing you to offload the complex, time consuming –yet critical – database administration tasks to us. This empowers you to focus on what really matters: building awesome apps.","spans":[]},{"type":"embed","oembed":{"height":113,"width":200,"embed_url":"https://www.youtube.com/watch?v=NvHQSV7jnKA","type":"video","version":"1.0","title":"Create a MongoDB Database on DigitalOcean","author_name":"DigitalOcean","author_url":"https://www.youtube.com/c/Digitalocean","provider_name":"YouTube","provider_url":"https://www.youtube.com/","cache_age":null,"thumbnail_url":"https://i.ytimg.com/vi/NvHQSV7jnKA/hqdefault.jpg","thumbnail_width":480,"thumbnail_height":360,"html":"<iframe width=\"200\" height=\"113\" src=\"https://www.youtube.com/embed/NvHQSV7jnKA?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>"}},{"type":"heading2","text":"Benefits of Managed MongoDB","spans":[]},{"type":"paragraph","text":"","spans":[]},{"type":"list-item","text":"Easy set up and maintenance: We create the database clusters for you. Simply choose the cluster configuration (e.g., memory, disk size, number of nodes, etc.), and the data center in which you want to host the database. Follow a few simple steps and your database cluster will be up and running in a matter of minutes. You can spin up clusters using the cloud control panel, CLI, or API.\n\n","spans":[{"start":0,"end":28,"type":"strong"}]},{"type":"list-item","text":"Automatic daily backups with point in time recovery: Data is one of the most important assets of an app, so it’s critical to backup your database. We take backups of your entire clusters automatically on a daily basis, for free. We also provide a point in time recovery for 7 days, that way if things go wrong due to human error, machine error, or some combination of both, you can easily restore the database as it was at any point in the previous 7 days. \n\n","spans":[{"start":0,"end":52,"type":"strong"}]},{"type":"list-item","text":"Automatic updates and access to latest MongoDB releases: You get access to MongoDB 4.4. This is the latest release of MongoDB and comes packed with numerous enhancements like hedged reads, rust, and swift drivers. Since we have developed Managed MongoDB in partnership with MongoDB Inc, you will always get access to new releases as they become available. With Managed MongoDB, the updates happen automatically. Just select a date and time for the updates and we take care of the rest. This makes it easy to stay up to date with MongoDB releases without disrupting your business.\n\n","spans":[{"start":0,"end":56,"type":"strong"},{"start":148,"end":169,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.mongodb.com/new","target":"_blank"}}]},{"type":"list-item","text":"High availability with automated failover: If your database goes down, it can take down the entire app, leading to bad customer experiences. With Managed MongoDB, you can easily minimize the downtime for your database and make it highly available with standby nodes. Standby nodes add redundancy, so if for example the primary node fails, the standby node is immediately promoted to primary and begins serving requests while we provision a replacement standby node in the background.\n\n","spans":[{"start":0,"end":42,"type":"strong"}]},{"type":"list-item","text":"Scale up easily to handle traffic spikes: As your app gains traction and the usage grows, it’s important to have a database that can keep up with the increased demand. With Managed MongoDB, you can easily scale up the size of database nodes when needed.\n\n","spans":[{"start":0,"end":41,"type":"strong"}]},{"type":"list-item","text":"Secure by default: Since data is critical, it also needs to be secure. We encrypt data at rest with LUKS and in transit with SSL. When you create a new cluster, it’s placed in a VPC network by default that provides a more secure connection between resources. You can also restrict access to your nodes to prevent brute-force password and denial-of-service attacks.","spans":[{"start":0,"end":18,"type":"strong"},{"start":178,"end":189,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.digitalocean.com/docs/networking/vpc/"}}]},{"type":"heading2","text":"The need for Managed Databases","spans":[]},{"type":"paragraph","text":"DigitalOcean’s mission is to simplify cloud computing so developers, startups, and SMBs can spend more time building software that changes the world. While databases are a critical component to any application, building, maintaining, and scaling them can be complex and time consuming. For developers that are building apps for their business, database administration is often not a core focus area. But it’s quite common to find developers that write the code and then also roll up their sleeves to maintain databases. Such users would rather offload the tedious database administration and focus their limited time and energy on building and enhancing their apps. ","spans":[]},{"type":"paragraph","text":"With this in mind, we introduced Managed Databases a couple of years ago and are excited to add Managed MongoDB to our portfolio. With this release, DigitalOcean Managed Databases now supports the following engines:","spans":[{"start":33,"end":50,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.digitalocean.com/products/managed-databases/"}}]},{"type":"image","url":"https://images.prismic.io/www-static/87745cc1-1c5f-4463-b104-104b7fc30dc7_managed-databases-logos.png?auto=compress,format","alt":null,"copyright":null,"dimensions":{"width":849,"height":104}},{"type":"paragraph","text":"Managed MongoDB launch comes on the heels of DigitalOcean App Platform, a modern, reimagined PaaS (Platform as a Service) that we released a few months ago. App Platform makes it very easy to build, deploy, and scale apps and static sites. You can deploy code by simply pointing to your GitHub and GitLab repos, and App Platform will do all the heavy lifting of managing infrastructure, app runtimes, and dependencies. App Platform, along with Managed Databases, helps fulfill DigitalOcean’s mission by empowering developers, startups, and SMBs to focus more on their apps, and less on the underlying infrastructure and databases.","spans":[{"start":45,"end":70,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.digitalocean.com/products/app-platform/"}}]},{"type":"heading2","text":"How Managed MongoDB works","spans":[]},{"type":"paragraph","text":"DigitalOcean provides you with various compute options to build your apps like:","spans":[]},{"type":"list-item","text":"Droplets: On-demand, Linux virtual machines suitable for production business applications and personal passion projects.","spans":[{"start":0,"end":8,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.digitalocean.com/products/droplets/"}}]},{"type":"list-item","text":"DigitalOcean Kubernetes: Managed Kubernetes with automatic scaling, upgrades, and a free control plane.","spans":[{"start":0,"end":23,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.digitalocean.com/products/kubernetes/"}}]},{"type":"list-item","text":"DigitalOcean App Platform: A fully managed Platform as a Service.","spans":[{"start":0,"end":25,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.digitalocean.com/products/app-platform/"}}]},{"type":"paragraph","text":"No matter which compute option you choose to build your apps, you can easily add Managed MongoDB to it. In addition to this, Managed MongoDB also integrates with the Node.js 1-Click App from DigitalOcean Marketplace making it a lot easier to build Node.js apps.","spans":[{"start":166,"end":215,"type":"hyperlink","data":{"link_type":"Web","url":"https://marketplace.digitalocean.com/apps/nodejs"}}]},{"type":"heading2","text":"Simple, predictable pricing","spans":[]},{"type":"paragraph","text":"Just like all DigitalOcean products, Managed MongoDB provides simple, predictable pricing that allows you to control costs and prevent any surprise bills. You can spin up a database cluster for just $15/month, or a highly available three-node replica set for $45/month. Click here for more information.","spans":[{"start":270,"end":301,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.digitalocean.com/pricing/#managed-databases"}}]},{"type":"heading2","text":"Regional availability","spans":[]},{"type":"paragraph","text":"Managed MongoDB is currently available in the following regions:","spans":[]},{"type":"list-item","text":"NYC3 (New York, USA)","spans":[]},{"type":"list-item","text":"FRA1 (Frankfurt, Germany)","spans":[]},{"type":"list-item","text":"AMS3 (Amsterdam, Netherlands)","spans":[]},{"type":"paragraph","text":"We will be making Managed Mongo available in other regions soon. Please check out the release notes for most up to date information on regional availability.","spans":[{"start":86,"end":99,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.digitalocean.com/docs/release-notes/"}}]},{"type":"heading2","text":"Join us at deploy, DigitalOcean’s virtual user conference","spans":[]},{"type":"paragraph","text":"Today we have deploy, DigitalOcean’s signature user conference, which focuses on celebrating, educating, and connecting awesome builders from all over the world.","spans":[{"start":14,"end":20,"type":"hyperlink","data":{"link_type":"Web","url":"https://deploy.digitalocean.com/home"}}]},{"type":"paragraph","text":"Check out the keynote session from DigitalOcean's CEO, Yancey Spruill, in which he talks about where we're headed as a company and shares some exciting product updates. His keynote will be followed by sessions from community members, engineers, customers, and other experts that are building technologies and businesses powered by the cloud. With live Q&A and an active Discord server, there’s ample opportunity to engage and learn something new. Click here to attend the deploy conference.","spans":[{"start":14,"end":69,"type":"hyperlink","data":{"link_type":"Web","url":"https://deploy.digitalocean.com/agenda/session/552806"}},{"start":347,"end":384,"type":"hyperlink","data":{"link_type":"Web","url":"http://do.co/deploy-discord"}},{"start":461,"end":489,"type":"hyperlink","data":{"link_type":"Web","url":"http://do.co/deploy"}}]},{"type":"paragraph","text":"We are also launching a hackathon for DigitalOcean Managed MongoDB. Learn how you can participate, submit an app and get a t-shirt.","spans":[{"start":24,"end":66,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.digitalocean.com/mongodb-hackathon"}}]},{"type":"paragraph","text":"We hope you will give Managed MongoDB a try. Here are some sample datasets and sample apps that you can use to kick the tires. Check out the docs and let us know what you think!","spans":[{"start":22,"end":43,"type":"hyperlink","data":{"link_type":"Web","url":"https://cloud.digitalocean.com/databases/new?engine=mongodb"}},{"start":59,"end":90,"type":"hyperlink","data":{"link_type":"Web","url":"https://github.com/do-community/mongodb-resources","target":"_blank"}},{"start":141,"end":145,"type":"hyperlink","data":{"link_type":"Web","url":"https://docs.digitalocean.com/products/databases/mongodb/"}}]},{"type":"paragraph","text":"If you’d like to have a conversation about using DigitalOcean and Managed MongoDB in your business, please feel free to contact our sales team.","spans":[{"start":120,"end":142,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.digitalocean.com/company/contact/sales/"}}]},{"type":"paragraph","text":"Happy coding!","spans":[]},{"type":"paragraph","text":"André Bearfield","spans":[]},{"type":"paragraph","text":"Director of Product Management","spans":[]}],"tags":[{"tag1":{"__typename":"PRISMIC_Tag","tag":"Product Updates","_linkType":"Link.document","_meta":{"uid":"product-updates"}}}],"author":{"__typename":"PRISMIC_Author","author_name":"André Bearfield","author_image":{"dimensions":{"width":553,"height":547},"alt":"André Bearfield","copyright":null,"url":"https://images.prismic.io/www-static/fdc7c85186f0a850b04083e1d4306bd1c19772e8_andre-bearfield.png?auto=compress,format"},"_meta":{"uid":"andre-bearfield"}},"_meta":{"uid":"introducing-digitalocean-managed-mongodb"}},"featured_blog_2":{"__typename":"PRISMIC_Blog","_linkType":"Link.document","blog_header_image":{"dimensions":{"width":790,"height":400},"alt":"Droplet Console","copyright":null,"url":"https://images.prismic.io/www-static/710499ae-78cc-4179-afc1-15793637b200_DODX3727-790x400-logo-2.jpg?auto=compress,format"},"blog_headline":[{"type":"heading1","text":"Securely connect to Droplets with SSH key pairs using a new Droplet Console","spans":[]}],"blog_post_date":"2021-08-10","blog_post_content":[{"type":"paragraph","text":"The famous author Ken Blanchard once said, “Feedback is the breakfast of champions.\" This is something we truly believe at DigitalOcean, and we always strive to enhance our products based on customer feedback.","spans":[]},{"type":"paragraph","text":"With this goal in mind, we are excited to introduce a new Droplet Console that will make it much easier to connect to your Droplets securely. The new Droplet Console provides one-click SSH access to your Droplets through a native-like SSH/Terminal experience. It also eliminates the need for a password or manual configuration of SSH keys. Starting today, we’re pleased to announce that the new Droplet Console is now available to all Droplet users.","spans":[]},{"type":"heading2","text":"Why you should be using Secure Shell (SSH) ","spans":[]},{"type":"paragraph","text":"Password-based security is notoriously insecure due to password fatigue and the overuse of passwords such as ‘123456’. Secure Shell or SSH is a network communication protocol that solves this by using passwordless solutions for encryption, enabling two computers to communicate and securely share data. At a high level, SSH works by creating cryptographic key pairs consisting of a public and private key, which are computer generated and stored separately to ensure their security. ","spans":[{"start":80,"end":117,"type":"hyperlink","data":{"link_type":"Web","url":"https://cybernews.com/best-password-managers/most-common-passwords/"}}]},{"type":"paragraph","text":"SSH has become the default encryption protocol for many industries, but it was difficult to use SSH keys with DigitalOcean’s current Recovery (VNC) console, which is why we developed our new Droplet Console. The new Droplet Console is backed by an agent that security supervises the key pair, while also providing one-click SSH access to our users. You can see the full list of features below.","spans":[]},{"type":"heading2","text":"The new Droplet Console: More time saving, less time wasting ","spans":[]},{"type":"paragraph","text":"The new Droplet Console is for everyone who is looking to build fast, secure apps and avoid hassles with SSH access & usability issues.","spans":[]},{"type":"paragraph","text":"In addition to easier SSH access, the new Droplet Console comes with:","spans":[]},{"type":"list-item","text":"Copy/paste text: Instead of typing lengthy key pairs and text manually, you can use copy/paste to save time. ","spans":[{"start":0,"end":17,"type":"strong"}]},{"type":"list-item","text":"Multi-color support: Multi-color support makes the console more useful and intuitive, and breaks the conventional standard appearance which is black text on a white background. ","spans":[{"start":0,"end":41,"type":"strong"}]},{"type":"list-item","text":"Multi-language support: DigitalOcean’s new Droplet Console supports multiple languages, meaning you can now type and view any content in any language that is supported by UTF-8","spans":[{"start":0,"end":24,"type":"strong"}]},{"type":"list-item","text":"OS/images supported: Linux distributions (Ubuntu(16.04 - 20.04), Fedora (32 & 33), Debian (9), CentOS (7.6 & 8.3), CentOS 8 Stream, Rocky Linux and Marketplace images.","spans":[{"start":0,"end":20,"type":"strong"},{"start":148,"end":159,"type":"hyperlink","data":{"link_type":"Web","url":"https://marketplace.digitalocean.com/"}}]},{"type":"paragraph","text":"The new Droplet Console is available by default on any new Droplets you spin up. You can also enable it manually on older Droplets. Click here to learn more!","spans":[{"start":132,"end":157,"type":"hyperlink","data":{"link_type":"Web","url":"https://docs.digitalocean.com/products/droplets/how-to/connect-with-console/"}}]},{"type":"paragraph","text":"Check out this short walkthrough video that shows the new Droplet Console in action: ","spans":[]},{"type":"embed","oembed":{"type":"video","embed_url":"https://www.youtube.com/watch?v=Qt7QihVuxiE","title":"Access Your Droplet Terminal Through the Web Console","provider_name":"YouTube","thumbnail_url":"https://i.ytimg.com/vi/Qt7QihVuxiE/hqdefault.jpg","provider_url":"https://www.youtube.com/","author_name":"DigitalOcean","author_url":"https://www.youtube.com/c/Digitalocean","height":113,"width":200,"version":"1.0","thumbnail_height":360,"thumbnail_width":480,"html":"<iframe width=\"200\" height=\"113\" src=\"https://www.youtube.com/embed/Qt7QihVuxiE?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>"}},{"type":"paragraph","text":"We hope you’re excited about the new Droplet Console. You’re welcome to spin some Droplets up right now, and try out the new Droplet Console – why wait?","spans":[{"start":72,"end":103,"type":"hyperlink","data":{"link_type":"Web","url":"https://cloud.digitalocean.com/droplets/new"}}]},{"type":"paragraph","text":"Happy coding!","spans":[]},{"type":"paragraph","text":"Harsh Banwait, Senior Product Manager","spans":[]}],"tags":[{"tag1":{"__typename":"PRISMIC_Tag","tag":"Product Updates","_linkType":"Link.document","_meta":{"uid":"product-updates"}}}],"author":{"__typename":"PRISMIC_Author","author_name":"Harsh Banwait","author_image":{"dimensions":{"width":600,"height":399},"alt":null,"copyright":null,"url":"https://images.prismic.io/www-static/e83ff690-b20c-4d88-a2b6-57e562558cd6_download.png?auto=compress,format"},"_meta":{"uid":"harsh-banwait"}},"_meta":{"uid":"new-droplet-console-ssh-support"}},"featured_blog_3":{"__typename":"PRISMIC_Blog","_linkType":"Link.document","blog_header_image":{"dimensions":{"width":790,"height":400},"alt":null,"copyright":null,"url":"https://images.prismic.io/www-static/588e28d3-d41e-480b-937b-8c3b19201f6e_DODX3568-790x400-Blog.jpg?auto=compress,format"},"blog_headline":[{"type":"heading1","text":"How to scale your SaaS product without breaking the bank","spans":[]}],"blog_post_date":"2021-06-22","blog_post_content":[{"type":"paragraph","text":"These days, if you are in the business of software, chances are you are delivering or plan to deliver your services using a Software-as-a-Service (SaaS) model. A combination of internet-based delivery, subscription-based pricing, and low-friction product experiences have made SaaS solutions valuable tools for their users, and an excellent vehicle for software builders looking to distribute their products.","spans":[]},{"type":"paragraph","text":"These factors have made SaaS solutions ubiquitous; SaaS is the largest segment in the public cloud market, and is used to provide functionality ranging from personal finance apps for consumers, to productivity software for businesses, and even tools and services for software developers themselves to compose their applications and simplify their workflows. It is also not uncommon to find micro-SaaS applications being built for specific industries such as retail, job functions such as accounting or marketing, or tasks such as event management. ","spans":[]},{"type":"paragraph","text":"The best thing about this SaaS wave has been that it has allowed a new generation of software builders to build and monetize applications and participate in the digital economy. Previously, you had to be a big company with lots of resources, name recognition and distribution networks to successfully sell software products. Now, irrespective of whether you are a single person working on a passion project, a small team of developers in a startup, or a small and medium-sized business (SMB), the SaaS model enables you to express your ideas in the form of software and deliver them to customers anywhere in the world.","spans":[]},{"type":"heading2","text":"The unique challenges of building SaaS solutions","spans":[]},{"type":"paragraph","text":"","spans":[]},{"type":"paragraph","text":"Despite the opportunities that come with the widespread adoption of SaaS products, software builders still have to answer key questions in their journey to building successful SaaS products. Understanding what customers to target, features to prioritize, how to price your product, and how to acquire customers are all critical questions to figure out while you are also doing the important job of actually building and operating the product. ","spans":[]},{"type":"paragraph","text":"Writing the code, testing, deployment, monitoring the usage in production, and ensuring that your apps are able to handle the additional demand when customer base and usage grows are all essential and time-consuming tasks.","spans":[]},{"type":"paragraph","text":"Additionally, being able to test multiple ideas, pivot, and double down on the ideas that actually work is critical in early stages of SaaS development. Once growth comes, it is equally important to scale up without compromising on performance or reliability. Needless to say, all of this needs to be economically viable as well, since not everyone has the resources of large SaaS providers like Salesforce or Adobe.","spans":[]},{"type":"heading2","text":"Cloud Computing enables builders but also poses challenges","spans":[]},{"type":"paragraph","text":"","spans":[]},{"type":"paragraph","text":"Fortunately, for the act of building and operating your apps, cloud computing can help take some load off your shoulders. Unless you have the scale and resources of Facebook, chances are you are not going to set up your own data centers to host the computing infrastructure that powers your SaaS company. Public cloud infrastructure providers can bring great value to SaaS builders by providing on-demand computing services with usage-based pricing. However, just like how the legacy software companies weren't built for the SaaS model, the early (and big) cloud computing services were not optimized for the unique needs of small SaaS building teams. ","spans":[]},{"type":"paragraph","text":"Smaller SaaS teams face challenges with large cloud computing providers, including:","spans":[]},{"type":"heading4","text":"Too many technology options","spans":[]},{"type":"paragraph","text":"There are just too many options for tech stacks on which to build your SaaS - programming languages, application development frameworks, libraries, runtime environments, architectural patterns, and deployment models - and the list is growing by the day.","spans":[]},{"type":"heading4","text":"Complexity of cloud computing services","spans":[]},{"type":"paragraph","text":"Even when you have decided on a technology stack, there is a lot of cloud vendor-specific terminology you need to learn and heavy lifting you need to do to build on the cloud, not all of which contributes to making your SaaS applications successful.","spans":[]},{"type":"heading4","text":"Unpredictable costs","spans":[]},{"type":"paragraph","text":"The experimentation necessary in early stages of SaaS development, as well as the scaling of applications required during the growth phase, call for affordable and predictable pricing from your cloud provider. The last thing SaaS teams want is surprising and indecipherable bills from your cloud provider. Unfortunately, smaller businesses often experience unpredictable costs with cloud providers who are busy serving only the large enterprises.","spans":[]},{"type":"heading2","text":"DigitalOcean provides a simple, cost effective solution for SaaS builders","spans":[]},{"type":"paragraph","text":"Fortunately, at DigitalOcean we have a laser focus on small software development teams, who are trying to build the next generation of applications. Today, DigitalOcean customers are already building SaaS applications which serve all kinds of customers.","spans":[{"start":191,"end":217,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.digitalocean.com/solutions/saas/"}}]},{"type":"paragraph","text":"We believe SaaS builders should focus on building apps that power their business, and not spend their valuable time on managing infrastructure. That is exactly what we have been able to enable through our intuitive products that are built for scale and reliability.","spans":[{"start":205,"end":223,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.digitalocean.com/products/"}}]},{"type":"list-item","text":"Vidazoo is an advertising technology company specializing in video streaming and serving. It serves video ads to thousands of websites and handles close to 10 billion requests per day. \n\n“We are as much a data company as an adtech company. Our business relies on speedy and accurate data processing at massive scale. DigitalOcean provides us the perfect set of tools to operate our SaaS business profitably, while not making us feel the need to become full time system administrators. We plan to move a lot of our apps to DigitalOcean App Platform and other fully managed products.” - Roman Svichar, CTO of Vidazoo","spans":[{"start":0,"end":7,"type":"hyperlink","data":{"link_type":"Web","url":"https://vidazoo.com/"}},{"start":187,"end":583,"type":"em"}]},{"type":"paragraph","text":"We believe in meeting customers where they are. If they already have an understanding of cloud infrastructure technologies, they should be able to leverage that knowledge and get started with our products without any further ramp up.","spans":[]},{"type":"list-item","text":"Whatfix is an enterprise SaaS provider that offers a digital adoption platform to businesses. The company helps enterprises gain the full value of their investments in enterprise applications by providing real-time, interactive, and contextual guidance to users of those applications. \n\n“What we really love about the DigitalOcean platform is the ease of use. We feel like we know infrastructure and can handle most of the configuration and management. What we needed from a cloud was not bells and whistles but efficiency and reliability. DigitalOcean provides us a platform to build our apps and then gets out of the way. Just how we like it.” - Achyuth Krishna, Director of Engineering of Whatfix","spans":[{"start":0,"end":7,"type":"hyperlink","data":{"link_type":"Web","url":"https://whatfix.com/blog/driving-the-future-now-were-excited-to-announce-our-90-million-series-d-funding/"}},{"start":287,"end":648,"type":"em"}]},{"type":"paragraph","text":"We understand that scaling while maintaining reliability of applications and profitability of business is important, so we provide robust solutions which minimize downtime.","spans":[]},{"type":"list-item","text":"Centra is a SaaS-based e-commerce platform for global direct-to-consumer and wholesale e-commerce brands. Centra provides a powerful e-commerce backend that lets brands build pixel-perfect, custom designed, online flagship stores. \n\n“How do we enable our customers to create differentiated online experiences? How do we ensure their e-commerce apps stay up and running at all times? How do we scale on-demand when traffic grows or new customers come in? These are the questions that we ask ourselves every day. Thankfully, we have a partner in DigitalOcean that provides just the platform to answer those questions enabling us to guarantee 99.9% uptime for our clients.” - Martin Jensen, CEO of Centra","spans":[{"start":0,"end":6,"type":"hyperlink","data":{"link_type":"Web","url":"https://centra.com/"}},{"start":233,"end":673,"type":"em"}]},{"type":"paragraph","text":"These are just a few examples of SaaS businesses finding success on DigitalOcean. We are constantly amazed by the creativity and innovation that software builders are utilizing our platform for. If you are interested in learning more about product updates, technical deep-dives and best practices for building SaaS products and businesses, please contact us to learn how we can help you get started. ","spans":[{"start":340,"end":357,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.digitalocean.com/migrate/?utmmedium=blog","target":"_blank"}}]},{"type":"paragraph","text":"Come build with DigitalOcean!","spans":[]},{"type":"paragraph","text":"Looking to migrate your SaaS to DigitalOcean? Leverage free infrastructure credits, robust training, and technical support to ensure a worry-free migration.","spans":[{"start":0,"end":156,"type":"strong"},{"start":0,"end":156,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.digitalocean.com/migrate/?utmmedium=blog","target":"_blank"}}]},{"type":"paragraph","text":"","spans":[]},{"type":"paragraph","text":"Raman Sharma","spans":[]},{"type":"paragraph","text":"Vice President, Product & Programs Marketing","spans":[]}],"tags":[{"tag1":{"__typename":"PRISMIC_Tag","tag":"Developer Relations","_linkType":"Link.document","_meta":{"uid":"developer-relations"}}}],"author":{"__typename":"PRISMIC_Author","author_name":"Raman Sharma","author_image":{"dimensions":{"width":512,"height":512},"alt":null,"copyright":null,"url":"https://images.prismic.io/www-static/497b4b14-d192-493a-8b66-7ae176ba99f3_raman.png?auto=compress,format"},"_meta":{"uid":"raman-sharma"}},"_meta":{"uid":"how-to-scale-your-saas-product-without-breaking-the-bank"}}}}]}}},"pageContext":{"limit":12,"skip":24,"numTagPages":5,"currentPage":3,"uid":"engineering","data":[{"node":{"author":{"_linkType":"Link.document","author_name":"Blake Thorne","author_image":{"dimensions":{"width":532,"height":566},"alt":"Blake Thorne","copyright":null,"url":"https://images.prismic.io/www-static/252c1d86d1456ee5cd008f2c5b542d4d04d73a18_blake.png?auto=compress,format"},"_meta":{"uid":"blake_thorne"}},"blog_header_image":{"dimensions":{"width":784,"height":418},"alt":"Game plan drawing with circles and X layered on top of code illustration","copyright":null,"url":"https://images.prismic.io/www-static/7af7b90903a03d72cbf2f26eb7f3fac06f49bd32_incidentcommssocial_blog.png?auto=compress,format"},"blog_headline":[{"type":"heading1","text":"Getting Started with an Incident Communications Plan","spans":[]}],"blog_post_content":[{"type":"paragraph","text":"At Statuspage, we believe it’s never too early for a team to start thinking about an incident communications plan. When your first big incident happens is way too late. Unplanned downtime can cause customer churn and unmanageable inbound support volume. Just one hour of unplanned downtime can cost organizations more than $100,000—and often much more—according to the latest annual downtime survey from Information Technology Intelligence Consulting Research.","spans":[{"start":3,"end":13,"type":"hyperlink","data":{"link_type":"Web","url":"http://www.statuspage.io"}},{"start":404,"end":459,"type":"hyperlink","data":{"link_type":"Web","url":"http://itic-corp.com/blog/2017/05/hourly-downtime-tops-300k-for-81-of-firms-33-of-enterprises-say-downtime-costs-1m/"}}]},{"type":"paragraph","text":"Some downtime is inevitable, even massive organizations experience outages from time to time. The good news is the harm from downtime can be mitigated by deploying reassuring context and information in a timely fashion. You may hope to never need an incident communications plan but, as any good Site Reliability Engineer (SRE) will tell you, hope is not a strategy.","spans":[]},{"type":"paragraph","text":"Mapping out your team’s first incident communications strategy doesn’t have to be overly complex or resource-draining. In fact, you can accomplish it fairly quickly using these four steps:","spans":[]},{"type":"list-item","text":"Block off two hours (don’t be surprised if you need even less time) with your core team to brainstorm and document a few simple things outlined in this guide. Involve anyone whose work touches an incident, including your ops and customer support leaders.","spans":[]},{"type":"list-item","text":"Use the incident communications planning template to document your strategy.","spans":[{"start":8,"end":49,"type":"hyperlink","data":{"link_type":"Web","url":"https://assets.digitalocean.com/public/IncidentCommsPlan.pdf"}}]},{"type":"list-item","text":"Set a quarterly meeting to review your strategy and make any changes.","spans":[]},{"type":"list-item","text":"Review and adjust your strategy each time you have an incident.","spans":[]},{"type":"heading1","text":"Before the Incident","spans":[]},{"type":"heading2","text":"Know what constitutes an incident","spans":[]},{"type":"paragraph","text":"Sometimes it’s hard to know what exactly to label as an “incident.” Here’s a set of guidelines Google SREs use, where if any one of the following is true the event is considered an incident:","spans":[{"start":75,"end":110,"type":"hyperlink","data":{"link_type":"Web","url":"https://landing.google.com/sre/book/chapters/managing-incidents.html"}}]},{"type":"list-item","text":"Do you need to involve a second team in fixing the problem?","spans":[]},{"type":"list-item","text":"Is the outage visible to customers?","spans":[]},{"type":"list-item","text":"Is the issue unsolved even after an hour’s concentrated analysis?","spans":[]},{"type":"paragraph","text":"Feel free to adopt these exact guidelines, adjust them, or write your own. “If any one of the following is true” is a good format. (Another helpful resource for mapping incident severity is this Severity Definitions guide from VMware.)","spans":[{"start":190,"end":233,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.vmware.com/support/policies/severity.html"}}]},{"type":"paragraph","text":"A note on playing it safe: in our experience it’s better to overcommunicate in situations where you’re uncertain. The inconvenience of closing the loop on an expected incident that never took off far outweighs the downside of playing catch up on incident comms hours into an incident.","spans":[]},{"type":"paragraph","text":"“I’ll just fix this quickly before anyone notices,” is a slippery slope. You might gamble and win the first time you try that, but play the game enough and eventually you’ll lose.","spans":[]},{"type":"heading2","text":"Team Roles","spans":[]},{"type":"paragraph","text":"Define key roles and expectations for incident responders. Clear labels and expectations can prevent a lot of damage in the heat of an incident. While large teams and complex SRE organizations have a web of roles and responsibilities, we see two roles as a good starting point.","spans":[]},{"type":"paragraph","text":"Incident commander","spans":[{"start":0,"end":18,"type":"strong"}]},{"type":"paragraph","text":"The incident commander is in charge of the incident response, making sure everyone is working toward resolution and following through on their tasks. They also are in charge of setting up any communications and documentation channels for the incident. That could be chat rooms, shared pages for documenting the incident, and even physical spaces in the office. This person also drives the post-incident review.","spans":[]},{"type":"paragraph","text":"Communicator","spans":[{"start":0,"end":12,"type":"strong"}]},{"type":"paragraph","text":"The communicator is in charge of translating the technical information into customer communications and getting those communications out via the right channels. They also monitor incoming customer communications and notify the incident commander if new groups of customers become impacted. After the incident, they ensure the post-mortem gets sent out.","spans":[]},{"type":"paragraph","text":"Our recommendation: make it clear from the beginning who has what role in an incident. Even if these people have the bandwidth to help with other areas of the incident, they should respond to these primary objectives first and delegate other tasks where necessary.","spans":[]},{"type":"heading2","text":"Preparation","spans":[]},{"type":"paragraph","text":"With a lean team, any time saved during an incident means a lot. Figuring out the right way to wordsmith an announcement can take up precious time in the heat of an incident.","spans":[]},{"type":"paragraph","text":"Decide on boilerplate language ahead of time and save it in a template somewhere. Use it to plug in the relevant details during an incident when you need it.","spans":[]},{"type":"paragraph","text":"Here is one incident template we use here for our own status page:","spans":[{"start":50,"end":65,"type":"hyperlink","data":{"link_type":"Web","url":"https://metastatuspage.com/"}}]},{"type":"paragraph","text":"\"The site is currently experiencing a higher than normal amount of load, and may be causing pages to be slow or unresponsive. We're investigating the cause and will provide an update as soon as possible.”","spans":[{"start":0,"end":204,"type":"em"}]},{"type":"paragraph","text":"This language is very simple and generic, and can be deployed as-is in a lot of cases where this is all we know. We can also amend the language to add more relevant details if we have them. For example:","spans":[]},{"type":"paragraph","text":"“The site is currently experiencing a higher than normal amount of load due to an incident with one of our larger customers. This is causing about 50% of pages to be unresponsive. We're investigating the cause and will provide an update as soon as possible.”","spans":[{"start":0,"end":258,"type":"em"}]},{"type":"paragraph","text":"You should also define your communications channels during an incident. While we obviously recommend Statuspage, there are a lot of tools you can use: Twitter, email, and company blog, as examples. Just make sure you’re clear where you will be posting messages.","spans":[]},{"type":"heading1","text":"During the incident","spans":[]},{"type":"paragraph","text":"Once the incident begins, we recommend these three “golden rules” which are worth keeping in mind during the incident.","spans":[]},{"type":"heading2","text":"Early","spans":[]},{"type":"paragraph","text":"It’s important to communicate as soon as there is any sign that the incident is impacting customers. Get a message posted as early as possible. It doesn’t have to be perfect. This message serves to reassure users that you’re aware of the issue and actively looking into it. This will also slow down the flood of support tickets and inbound messaging you’re sure to receive during incidents.","spans":[]},{"type":"heading2","text":"Often","spans":[]},{"type":"paragraph","text":"When you’re heads-down working on an incident, it can be easy to let regular updates slide. But these long gaps between updates can cause uncertainty and anxiety for your customers. They can start to expect the worst. Even if you’re just updating to say that you’re still investigating the matter, that’s better than no communication. Bonus points if you give an estimate on when next comms will be (and stick to it).","spans":[]},{"type":"paragraph","text":"Here’s an example a 2016 HipChat incident.","spans":[{"start":0,"end":17,"type":"hyperlink","data":{"link_type":"Web","url":"https://status.hipchat.com/incidents/6zcv7x6rrktk"}}]},{"type":"heading2","text":"Precision","spans":[]},{"type":"paragraph","text":"In your messaging during the incident, be as precise as you can be without guessing or giving non-committal answers.","spans":[]},{"type":"paragraph","text":"Instead of:","spans":[]},{"type":"paragraph","text":"“We think we know what’s going on but we need more time.”","spans":[{"start":0,"end":57,"type":"em"}]},{"type":"paragraph","text":"Try:","spans":[]},{"type":"paragraph","text":"“We’re still working to verify the root cause.”","spans":[{"start":0,"end":47,"type":"em"}]},{"type":"paragraph","text":"Instead of:","spans":[]},{"type":"paragraph","text":"“The problem seems to be database related.”","spans":[{"start":0,"end":43,"type":"em"}]},{"type":"paragraph","text":"Try:","spans":[]},{"type":"paragraph","text":"“We’re continuing to investigate the problem.”","spans":[{"start":0,"end":46,"type":"em"}]},{"type":"paragraph","text":"At first glance this second example may seem counterintuitive. Why leave out the fact that the issue could be database related? Because you aren’t sure yet. Avoiding hedging words like “we think.” Don’t say you “think” you found the root cause. Either you actually have found the cause or you haven’t.","spans":[]},{"type":"paragraph","text":"Once you’ve confirmed the cause, then clearly state as much detail as you’re able to.","spans":[]},{"type":"paragraph","text":"For example:","spans":[]},{"type":"paragraph","text":"“We’ve identified a corruption with our database related to our last deploy. We are currently rolling back that deploy and monitoring results.”","spans":[{"start":0,"end":143,"type":"em"}]},{"type":"heading1","text":"After the Incident","spans":[]},{"type":"paragraph","text":"Some of the biggest opportunities for your team come in the moments after the dust settles from an incident. Your team ideally will run a Post Incident Review session to unpack what happened on the technical side. It’s also a great time to build customer trust by letting them know that you’re taking the incident seriously and taking steps to ensure it doesn’t happen again.","spans":[]},{"type":"paragraph","text":"An incident post-mortem is meant to be written after the incident and give a big picture update of what happened, how it happened, and what steps the team is taking to ensure it isn’t repeated. Here are our post-mortem rules.","spans":[]},{"type":"heading2","text":"Empathize","spans":[]},{"type":"paragraph","text":"Apologize for the inconvenience, thank customers for their patience, and ensure you’re working on a fix.","spans":[]},{"type":"heading2","text":"Be personal","spans":[]},{"type":"paragraph","text":"We see it all the time where teams depersonalize themselves in an effort to seem professional or official. This leads to a cold, distant tone in post-mortems that doesn’t build trust.","spans":[]},{"type":"paragraph","text":"Use active voice and “we” pronouns to tell your story. Steer away from words that are overly academic or corporate sounding when simple ones will do.","spans":[]},{"type":"paragraph","text":"Instead of:","spans":[]},{"type":"paragraph","text":"“Remediation applications on the new load balancer configurations are finalized.”","spans":[{"start":0,"end":81,"type":"em"}]},{"type":"paragraph","text":"Try:","spans":[]},{"type":"paragraph","text":"“We’ve completed the configuration on the new load balancer.”","spans":[{"start":0,"end":61,"type":"em"}]},{"type":"heading2","text":"Details inspire confidence","spans":[]},{"type":"paragraph","text":"People have a good sense for when you’re using a lot of words but not really saying anything. Details are the way to keep your post-mortem from sounding like a lot of hot air.","spans":[]},{"type":"paragraph","text":"Here’s an example from a post-mortem Facebook engineers posted after a 2010 incident.","spans":[{"start":18,"end":84,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919/"}}]},{"type":"paragraph","text":"Consider this paragraph:","spans":[]},{"type":"paragraph","text":"“Today we made a change to the persistent copy of a configuration value that was interpreted as invalid. This meant that every single client saw the invalid value and attempted to fix it. Because the fix involves making a query to a cluster of databases, that cluster was quickly overwhelmed by hundreds of thousands of queries a second.”","spans":[{"start":0,"end":338,"type":"em"}]},{"type":"paragraph","text":"Likely that’s more technical of an explanation than most readers will need. The ones who want this level of detail will appreciate it. The ones who don’t will at least recognize that you’re going above and beyond to explain what happened. A lot of teams worry about being too technical in their messaging and instead wind up sending watered-down communications. Opt for specific details instead.","spans":[]},{"type":"heading2","text":"Close the loop","spans":[]},{"type":"paragraph","text":"The post-mortem is your chance to have the last word in an incident. Leave the reader with a sense of trust and confidence by laying out clearly what you’re doing to keep this from happening again.","spans":[]},{"type":"paragraph","text":"Here’s an example from a Twilio post-mortem:","spans":[{"start":7,"end":43,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.twilio.com/blog/2013/07/billing-incident-post-mortem-breakdown-analysis-and-root-cause.html"}}]},{"type":"paragraph","text":"“In the process of resolving the incident, we replaced the original redis cluster that triggered the incident. The incorrect configuration for redis-master was identified and corrected. As a further preventative measure, Redis restarts on redis-master are disabled and future redis-master recoveries will be accomplished by pivoting a slave.","spans":[{"start":0,"end":341,"type":"em"}]},{"type":"paragraph","text":"The simultaneous loss of in-flight balance data and the ability to update balances also exposed a critical flaw in our auto-recharge system. It failed dangerously, exposing customer accounts to incorrect charges and suspensions. We are now introducing robust fail-safes, so that if billing balances don’t exist or cannot be written, the system will not suspend accounts or charge credit cards. Finally, we will be updating the billing system to validate against our double-bookkeeping databases in real-time.”","spans":[{"start":0,"end":509,"type":"em"}]},{"type":"paragraph","text":"Notice how specific this is with outlining what went wrong and exactly what the team is putting in place to keep the problem from repeating.","spans":[]},{"type":"paragraph","text":"Even though users today expect 24/7 services that are always up, people are tolerant of outages. We’ve heard a lot of stories about outages over the years at Statuspage, nobody ever went out of business by being too transparent or communicative during an incident. Consider the kind of information and transparency you’d like to receive from the products and vendors you use, and try to treat your users the way you’d like to be treated.","spans":[]},{"type":"paragraph","text":"Looking to go even deeper on incident comms exercises? Check out our recent Team Playbook plays on incident response values and incident response communications.","spans":[{"start":0,"end":161,"type":"em"},{"start":99,"end":123,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.atlassian.com/team-playbook/plays/incident-values"}},{"start":128,"end":160,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.atlassian.com/team-playbook/plays/incident-response-communications"}}]},{"type":"paragraph","text":"Blake Thorne is a Product Marketing Manager at Statuspage, which helps teams big and small with incident communications. He can be reached at bthorne@atlassian.com or on Twitter.","spans":[{"start":142,"end":163,"type":"hyperlink","data":{"link_type":"Web","url":"mailto:bthorne@atlassian.com"}},{"start":170,"end":177,"type":"hyperlink","data":{"link_type":"Web","url":"https://twitter.com/blakethorne"}}]}],"blog_post_date":"2018-04-26","tags":[{"tag1":{"tag":"Engineering","_linkType":"Link.document","_meta":{"uid":"engineering"}}}],"_meta":{"uid":"getting-started-with-an-incident-communications-plan"}}},{"node":{"author":{"_linkType":"Link.document","author_name":"Jaime Woo","author_image":{"dimensions":{"width":960,"height":895},"alt":"Jaime Woo","copyright":null,"url":"https://images.prismic.io/www-static/318ff445a77048bed7becaef09b052a6ea607178_digitalocean---copy.jpg?auto=compress,format"},"_meta":{"uid":"jaime_woo"}},"blog_header_image":{"dimensions":{"width":784,"height":418},"alt":"Ruby gem on a rollercoaster ","copyright":null,"url":"https://images.prismic.io/www-static/92392fd7c40c49bbd7a5203031f60f9810055ac7_railsconfsocial_blog.png?auto=compress,format"},"blog_headline":[{"type":"heading1","text":"Pittsburgh, We’ll See Yinz at RailsConf!","spans":[]}],"blog_post_content":[{"type":"paragraph","text":"RailsConf has left the desert and makes its way to Steel City April 17-19, 2018. We’ll have Sam Phippen presenting, and several DO-ers checking out talks and tending our booth. Here’s what you need to know about RailsConf 2018.","spans":[{"start":51,"end":79,"type":"hyperlink","data":{"link_type":"Web","url":"https://railsconf.com/"}}]},{"type":"paragraph","text":"In Sam’s talk, “Quick and easy browser testing using RSpec and Rails 5.1,” you'll learn about the new system specs in RSpec, how to set them up, and what benefits they provide. It’s for anyone wanting to improve their RSpec suite with full-stack testing.","spans":[]},{"type":"paragraph","text":"From the talk description:","spans":[{"start":0,"end":25,"type":"hyperlink","data":{"link_type":"Web","url":"https://railsconf.com/program/sessions#session-577"}}]},{"type":"paragraph","text":"Traditionally doing a full-stack test of a Rails app with RSpec has been problematic. The browser wouldn't automate, capybara configuration would be a nightmare, and cleaning up your DB was difficult. In Rails 5.1 the new 'system test' type was added to address this. With modern RSpec and Rails, testing every part of your stack including Javascript from a browser is now a breeze.","spans":[{"start":0,"end":382,"type":"em"}]},{"type":"paragraph","text":"Make sure you don’t miss it, Thursday, April 19, from 10:50 AM-11:30 AM in the Spirit of Pittsburgh Ballroom. If you’re interested in RSpec, you might dig his talk from 2017, “Teaching RSpec to Play Nice with Rails.”","spans":[{"start":29,"end":108,"type":"strong"},{"start":176,"end":214,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.youtube.com/watch?v=jyPfrK1y1nc"}}]},{"type":"paragraph","text":"You can also catch us in the Exhibit Hall, at booth number 520. The Hall is on Level 2, in Hall A. We’ll be hanging at our booth Wednesday, April 18 from 9:30 AM-6:00 PM, and Thursday, April 19 from 9:30 AM-5:15 PM.","spans":[{"start":46,"end":62,"type":"strong"}]},{"type":"paragraph","text":"See you there, or, as they say in Pittsburgh, meechinsdahnair!","spans":[]}],"blog_post_date":"2018-04-18","tags":[{"tag1":{"tag":"Developer Relations","_linkType":"Link.document","_meta":{"uid":"developer-relations"}}},{"tag1":{"tag":"Engineering","_linkType":"Link.document","_meta":{"uid":"engineering"}}}],"_meta":{"uid":"pittsburgh-well-see-you-at-railsconf"}}},{"node":{"author":{"_linkType":"Link.document","author_name":"Tom Spiegelman","author_image":{"dimensions":{"width":892,"height":772},"alt":"Tom Spiegelman","copyright":null,"url":"https://images.prismic.io/www-static/ae5336e19ec28b1509eb6321c57124f56704d81f_screen-shot-2017-09-25-at-12.18.13-pm.png?auto=compress,format"},"_meta":{"uid":"tom_spiegelman"}},"blog_header_image":{"dimensions":{"width":784,"height":418},"alt":"Woman and Male Developers talking illustration","copyright":null,"url":"https://images.prismic.io/www-static/c46ee9ed2681cc2f82facb789cb95fd7f05de1b0_mentoringengineers_blog-1.png?auto=compress,format"},"blog_headline":[{"type":"heading1","text":"Mentoring Engineers Through An Engineering Fellowship Program","spans":[]}],"blog_post_content":[{"type":"paragraph","text":"For two years, I’ve managed the Infrastructure Engineering (“Infra”) team at DigitalOcean. We’re responsible for managing all servers and machines up to the application layer. This includes hardware spec, firmware, component validation, base OS, configuration management, and hardware management (hardware management database).","spans":[]},{"type":"paragraph","text":"In addition to my core responsibilities managing the Infra team, I wanted to foster an environment where mentorship was possible and worked with colleagues to create the Infrastructure Engineering Fellowship Program. It’s an immersive program where DigitalOcean employees from other teams “join” the Infra team for two weeks. Employees with fundamental Linux knowledge and some configuration management experience are eligible to participate.","spans":[]},{"type":"paragraph","text":"“Fellows”—as they are known—are invited to a private Slack channel with fellowship alum. They work through JIRA tickets assigned to the team (all while pairing with Infra team engineers), attend team stand-ups, and finally, pick a project to work on for the two week duration. Additionally, fellows meet with me at the start and end of each week to discuss what they worked on and to answer questions they have. To date, we’ve had nine people complete the fellowship and we continue to open the fellowship up to other engineers at DO.","spans":[]},{"type":"heading3","text":"How the Fellowship Started","spans":[]},{"type":"paragraph","text":"This program started as a cross-team training experience between my team and the Tier-2 Cloud Operations team (the 24/7 team responsible for uptime on our servers and services), since both of our teams interacted with each other on a daily basis. After a few successful trials with the Cloud Operations team, we realized that there were several other teams that were interested in learning what we do and wanted to take advantage of the fellowship program. We have now had people from five different teams sign up and participate in the program.","spans":[]},{"type":"paragraph","text":"My team gets so much more out of the fellowship than we put in. First, we build comradery between the wider organization and my team. Individuals we only worked with through JIRA and Slack now have a personal relationship with the team and are more eager to engage and work with us. My team gains a better perspective of what other teams go through and work on a daily basis which helps us build better tools and workflows to support them. Finally, it is a great way to recruit. Engineers that have been hired for my team came through the fellowship program.","spans":[]},{"type":"paragraph","text":"Growing people internally is one of the greatest things I have done with my career. I have had three people join my team from inside the company and have been very successful in their new roles. In a perfect world, we would pair every senior engineer on the team with one engineer still early in their career. In my experience, when looking at the “Tuckman's stages of group development” you will have the best performing team when you have mentors and mentees going through the four stages together as a team:","spans":[]},{"type":"image","url":"https://images.prismic.io/www-static/9e553fc24522dfde4c08d234d1e805766b24aca3_tuckman_team_development-large1.png?auto=compress,format","alt":"Tuckman 4-stages of group development","copyright":null,"dimensions":{"width":600,"height":324}},{"type":"paragraph","text":"Tuckman's stages of group development. Photo credit: Tutorials Point","spans":[{"start":0,"end":68,"type":"em"},{"start":53,"end":68,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.tutorialspoint.com/individual_and_group_behavior/five_stage_model_group_development.htm"}}]},{"type":"heading3","text":"Managing the Fellowship Program","spans":[]},{"type":"paragraph","text":"One of the things that we keep top of mind is sustainability. Although two weeks isn’t very long, properly mentoring someone takes a lot of time, and we want to make sure no one feels overwhelmed by the experience. We currently take on just one fellow at a time, and we cater the program to each participant. For example, if a fellow is more interested in hardware than big data, they might pair with our integration team who is charged with managing hardware and firmware, rather than our DevOps-focused team.","spans":[]},{"type":"paragraph","text":"There are a few benefits of managing the fellowship this way. One, we can iterate quickly since the program lasts just two weeks. And two, we can focus our energies on mentoring just one person at a time to limit straining the team’s bandwidth. Based on feedback from past fellows, we’ve changed how we handle our 1:1s with engineers and code pairing sessions. We now conduct 1:1s with specific goals in mind. Each fellow is asked to give feedback at the very end of the program to help us guide future fellows.","spans":[]},{"type":"paragraph","text":"That said, the same benefits are in some ways ongoing challenges. Working with each fellow individually takes up my time, but it also affects the engineers on my team. They need to take time out of their busy schedules to pair with the fellow by breaking their usual workflow and compelling them to walk through projects step by step. This means something that may take them an hour ends up taking most of a day.","spans":[]},{"type":"paragraph","text":"That said, we’re able to make this work because we work on a number of tasks and projects at any given time. If a team is working on one long-term project, the time it takes to explain the project to someone won’t actually yield any benefit in a two-week long program. The fellowship program (and programs like it) really need to be catered to the participant and the team that they are embedding with.","spans":[]},{"type":"heading3","text":"What Makes It Worthwhile","spans":[]},{"type":"paragraph","text":"As I pointed out earlier, pairing engineers with more senior engineers leads to better performing teams. Furthermore, there is an even stronger connection when you pair engineers that have proprietary or historical knowledge from inside the company. I am a firm believer that if strong minded, eager-to-learn engineers exist within the company, you shouldn’t hire from outside the company. Creating infrastructure that supports mentorship leads to strong engineers, strong teams, and a strong company.","spans":[]},{"type":"paragraph","text":"I love seeing people continue to have conversations and work on projects with my team after the fellowship is over. It is simply amazing to see, and I give all the credit to the engineers on my team. Every one of them is eager to pass on knowledge that they have, and they’ve embraced the fellowship and its goals. The fellowship wouldn’t have been successful if my team didn’t share the same beliefs around mentorship and its cross-team benefits that I have.","spans":[]},{"type":"heading3","text":"Future of the Fellowship","spans":[]},{"type":"paragraph","text":"When I started my career in IT, I had an amazing mentor (shout out to Rob Lahnemann) who really took me under his wing and taught me everything he could about programming, Linux, and networking. My manager at the time (shout out to Eric Austin) set this up and put me in a place to succeed as a mentee. This experience really influenced what I believe it means to be a good manager. Pairing engineers eager to learn with senior engineers is huge key factor in any successful team. In the current engineering community, it is not uncommon to find engineers who are not influenced to share their knowledge or are not given the time to be a mentor. But in my opinion, growing as an engineer means being a mentor.","spans":[]},{"type":"paragraph","text":"In the future, I would love to see the program more of a revolving door of people doing more work with the Infrastructure Engineering team and doing the fellowship program multiple times (hopefully sometimes for longer than two weeks). I also would love to influence programs like this more often inside DigitalOcean and outside DigitalOcean. One of my biggest goals and drivers in writing this is to influence similar programs in the industry as a whole. My career and pace of growth was directly influenced by a strong mentor, so my passion here for influencing more mentor/mentee relations in the industry is high.","spans":[]},{"type":"paragraph","text":"Tom Spiegelman is an Infrastructure Engineering Manager at DigitalOcean. He has an awesome dog, a great team, and is married to the amazing Chantal Spiegelman. He is passionate about all things tech, specifically infrastructure. You can find him on LinkedIn or on Twitter.","spans":[{"start":0,"end":272,"type":"em"},{"start":249,"end":257,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.linkedin.com/in/tspiegs/"}},{"start":264,"end":271,"type":"hyperlink","data":{"link_type":"Web","url":"https://twitter.com/tspeegs"}}]}],"blog_post_date":"2018-04-03","tags":[{"tag1":{"tag":"Engineering","_linkType":"Link.document","_meta":{"uid":"engineering"}}},{"tag1":{"tag":"Culture","_linkType":"Link.document","_meta":{"uid":"culture"}}}],"_meta":{"uid":"mentoring-engineers-through-an-engineering-fellowship-program"}}},{"node":{"author":{"_linkType":"Link.document","author_name":"Billie Cleek","author_image":null,"_meta":{"uid":"billie_cleek"}},"blog_header_image":{"dimensions":{"width":784,"height":418},"alt":null,"copyright":null,"url":"https://images.prismic.io/www-static/287bb0dc-2d4f-41ac-b8e7-cc701e3e1d23_CodeReview_blog.png?auto=compress,format"},"blog_headline":[{"type":"heading1","text":"How to Conduct Effective Code Reviews","spans":[]}],"blog_post_content":[{"type":"paragraph","text":"A code review, at its core, is a conversation about a set of proposed changes. Early in my career, I viewed code reviews as a mostly technical exercise that should be devoid of non-technical concerns. I now see them as one of the few opportunities to concurrently learn and teach while also strengthening my relationship with my peers and colleagues. ","spans":[]},{"type":"paragraph","text":"My team, Delivery, has been working together for at least six months (some much longer), but only two members work in the New York City office while the rest are spread across North America. Because of our familiarity with each other, most of our daily interactions take place via text or video chat. Code reviews are often short, but we also go out of our way to communicate when we are stating an opinion or being nit-picky. ","spans":[]},{"type":"paragraph","text":"Most software developers are expected to participate in code reviews, and yet few are offered any training or guidance on conducting and participating in an effective code review. Participants attempt to find the most appropriate solution to a problem given the constraints of time, effort, and skills of all involved. But how do we have that conversation? What does an effective conversation look like? And what are the challenges of participating in a code review, and how can you overcome them?","spans":[]},{"type":"paragraph","text":"Whether your tool of choice is GitHub, GitLab, Gerrit, or another tool, the goal of this article is to help you get as much out of the code review process as possible.","spans":[]},{"type":"heading3","text":"What Are Code Reviews For?","spans":[]},{"type":"paragraph","text":"Code reviews happen in a wide range of contexts, and often the skills and depth of experience of participants vary widely. On open source projects, for example, participants may not have any sort of personal relationship with each other. Indeed, they may never communicate outside of the code review process. At the other end of the spectrum are code reviews where the participants have daily face-to-face interactions, such as when everyone works at the same company. A good participant will adjust how they participate in a code review according to their knowledge of the other participants. ","spans":[]},{"type":"paragraph","text":"While it is important to adjust one's communication style in accordance with the intended recipient, how to adjust is influenced by three primary factors: the purpose of the code review, the intended audience, and one's relationship to the audience.","spans":[]},{"type":"heading3","text":"Identifying the Purpose of a Code Review","spans":[]},{"type":"paragraph","text":"Code reviews serve both technical and cultural purposes: finding bugs before they're integrated, identifying security concerns, ensuring style consistency with the existing codebase, maintaining code quality, training, fostering a greater sense of ownership, and giving other maintainers an opportunity to get familiar with the code before it's integrated are just some of the reasons you may be asked to participate in code reviews. Make sure you know why you're participating in a code review beforehand. ","spans":[]},{"type":"paragraph","text":"Regardless of why you’re conducting a code review, it is important to respect the purposes that code reviews serve for the codebase. If the only purpose of a code review is to check for security concerns, then drop whatever personal concerns you may have about coding style or naming patterns. Unfortunately, it is not uncommon for the purpose of code reviews to be poorly defined or non-existent. In that case, once you've determined that the proposed changes are necessary and add value, I'd suggest reviewing for correctness, bug identification, and security concerns. Secondary to those concerns may be overall quality and long term maintainability of the proposed changes.","spans":[]},{"type":"heading3","text":"Submissions: What to Include","spans":[]},{"type":"paragraph","text":"Code reviews typically start with a contributor submitting a proposed set of changes to the project. The submission should include:","spans":[]},{"type":"list-item","text":"A clear and useful description of the changes and give a general overview of why the change is necessary.","spans":[]},{"type":"list-item","text":"The scope of the change.","spans":[]},{"type":"list-item","text":"Areas where reviewers may want to give special attention.","spans":[]},{"type":"list-item","text":"Subtleties that need clarification.","spans":[]},{"type":"list-item","text":"Details that may help reviewers better understand the patch.","spans":[]},{"type":"paragraph","text":"Depending on the complexity of the changes, reviewers may find an overview of the trade-offs the submitter made in the patch helpful in order to be better understand why the patch is the most appropriate of the possible alternatives.","spans":[]},{"type":"paragraph","text":"Written communication about technical subjects can be difficult: people have limited time, and each of us is on a journey of confronting challenges and personal growth. In code reviews every participant has a role to play, each with its own set of objectives:","spans":[]},{"type":"list-item","text":"As a writer, strive to be as clear as you can. When in doubt, be descriptive. ","spans":[{"start":5,"end":11,"type":"strong"}]},{"type":"list-item","text":"As a reader, ask questions when something is unclear. ","spans":[{"start":5,"end":11,"type":"strong"}]},{"type":"list-item","text":"As a reviewer, be gracious when someone uses their time to submit a patch to your project. ","spans":[{"start":5,"end":13,"type":"strong"}]},{"type":"list-item","text":"As a submitter, be forgiving when your patch is not reviewed in the time frame you would prefer. ","spans":[{"start":5,"end":14,"type":"strong"}]},{"type":"paragraph","text":"Regardless of your role in the review process, respect that others may be at a different place in their journey, and assume that all participants are engaging in the process in good faith and because of shared values and goals. The process is easiest when one assumes that all other participants are doing their utmost to help you succeed and get better. ","spans":[]},{"type":"paragraph","text":"Here's an example of a pull request from our team where I asked for clarification, discussed my concerns, and ultimately landed on a compromise that made the submission better and easier to maintain, all while gaining personal knowledge of the subject at hand:","spans":[]},{"type":"image","url":"https://images.prismic.io/www-static/52f5f421-6510-4b8c-8c92-100cd1272605_pasted-image-1.png?auto=compress,format","alt":"code review example #1","copyright":null,"dimensions":{"width":1318,"height":702}},{"type":"image","url":"https://images.prismic.io/www-static/562cb75e-2021-49e6-951e-07e788b87449_pasted-image-0.png?auto=compress,format","alt":"code review example #2","copyright":null,"dimensions":{"width":1426,"height":1336}},{"type":"paragraph","text":"Example of how my team communicates in our code reviews.","spans":[{"start":0,"end":56,"type":"em"}]},{"type":"heading3","text":"Knowing Your Audience","spans":[]},{"type":"paragraph","text":"Start by reading all the code. As a reviewer, recognize that the submitter gave their time and energy and tried to improve the product in some way. As you read and strive to understand the patch, record your questions and concerns privately so that you understand the full context before providing any feedback. As mentioned previously, make an honest effort to restrict your feedback to the purposes for which the code review is being conducted. ","spans":[{"start":0,"end":29,"type":"strong"}]},{"type":"paragraph","text":"Prepare and submit your feedback after reading and understanding the changes. Be gracious. Try to keep your comments focused on the code and the solution it offers; avoid digressing into unrelated matters. If you see something surprising, ask questions. If you don't have a strong history with a submitter, go the extra mile to communicate your good intentions. It's OK to use emojis to communicate tone. Strive to begin fostering a healthy, productive relationship with this new contributor.","spans":[{"start":0,"end":76,"type":"strong"}]},{"type":"paragraph","text":"Your feedback in code reviews is one of the primary ways to build a community of developers eager to contribute to your project. By nurturing a strong community, you will promote a quality product. Especially for open source maintainers, an authentic, explicit “thank you for the contribution” or other nice words can go a long way towards making people feel appreciated and fostering a supportive community. ","spans":[{"start":0,"end":91,"type":"strong"}]},{"type":"paragraph","text":"Take the feedback, evaluate it, and decide what to do next. For submitters, it can be difficult to read criticism of the code you have written. When a reviewer asks for changes, they are doing so for the same reason a patch author submits a patch: a genuine desire to improve the product. Remind yourself that feedback about code is not personal. You may decide to accept the feedback and change something. Or you may decide that there was a misunderstanding, and that some requested changes are unwarranted or would simply be wrong or add no value. It’s OK to push back.","spans":[{"start":0,"end":58,"type":"strong"}]},{"type":"heading3","text":"Developing a Partnership Through Code Reviews","spans":[]},{"type":"paragraph","text":"When there is an asymmetric level of experience between the submitter and reviewer, use the opportunity to mentor. As a reviewer with more experience than the submitter, you may choose to accept that submitter's patch as-is and then improve upon it, contacting the submitter to let them know about your changes later. In a professional setting, such an approach isn't always feasible. Have the conversation in the open so that observers (i.e. other readers) can learn too, but reach out for a more personal touch if the extent of feedback is becoming overwhelming in written form. In my experience, patches submitted by someone significantly more experienced than the reviewer are usually accepted as-is or with only very minor changes requested.","spans":[]},{"type":"paragraph","text":"When you're thinking out loud, make it clear to the reader so that they do not think you are asking for a change inasmuch as evaluating a possibility. If you're nitpicking, explain your reasons for doing so. On our team, we often preface nit-picky comments with `(nit)`, in order to help contributors recognize these types of comments. This usually serves as a signal that the contributor can ignore that feedback if they want. Without that distinction, the nitpicks are not distinguishable from the feedback that the reviewer feels more strongly about. For all participants: when you're unsure about something, ask, and err on the side of clarity and friendliness.","spans":[]},{"type":"paragraph","text":"A successful code review will result in a higher quality change, strengthen the relationship between reviewer and submitter, and increase the understanding that everyone involved has of the project. Code reviews are not just a formality that require a rubber stamp before being merged; they are an essential aspect of modern software development that provide real value to projects and teams by promoting good software engineering practices.","spans":[]},{"type":"heading3","text":"Conclusion","spans":[]},{"type":"paragraph","text":"Through code reviews, I've learned to be more gracious and more understanding about the personal challenges and technical struggles that everyone experiences. I have learned to more thoughtfully examine the trade-offs that we all make when writing software. I hope the ideas presented here can help you grow your community and increase your effectiveness.","spans":[]},{"type":"paragraph","text":"Billie Cleek is a Senior Software Engineer on the Delivery team where he supports internal tools to provide a consistent deployment surface for DigitalOcean's microservices. In his spare time, Billie is a maintainer of vim-go, infrequent contributor to other open source projects, and can be found working on his 100-year-old house or in the forests of the Pacific Northwest regardless of the weather. You may also find Billie on GitHub and Twitter.","spans":[{"start":0,"end":449,"type":"em"},{"start":430,"end":436,"type":"hyperlink","data":{"link_type":"Web","url":"https://github.com/bhcleek"}},{"start":441,"end":448,"type":"hyperlink","data":{"link_type":"Web","url":"https://twitter.com/bhcleek"}}]}],"blog_post_date":"2018-03-28","tags":[{"tag1":{"tag":"Engineering","_linkType":"Link.document","_meta":{"uid":"engineering"}}}],"_meta":{"uid":"how-to-conduct-effective-code-reviews"}}},{"node":{"author":{"_linkType":"Link.document","author_name":"Geoff Hickey","author_image":null,"_meta":{"uid":"geoff_hickey"}},"blog_header_image":{"dimensions":{"width":784,"height":418},"alt":null,"copyright":null,"url":"https://images.prismic.io/www-static/2aebccc9-23d0-4c4d-9598-460b0a28833a_go-libvirt_blog.png?auto=compress,format"},"blog_headline":[{"type":"heading1","text":"Open Source at DigitalOcean: Extending go-libvirt with Code Generation","spans":[]}],"blog_post_content":[{"type":"paragraph","text":"Back in November 2016, DigitalOcean released go-libvirt, an open source project containing a pure Go interface to libvirt. Using go-libvirt, developers could manage virtual machines leveraging all the power of libvirt’s extensive API without leaving the comfortable environment of Go. But there was a catch.","spans":[{"start":23,"end":55,"type":"hyperlink","data":{"link_type":"Web","url":"https://blog.digitalocean.com/introducing-go-qemu-and-go-libvirt/"}}]},{"type":"paragraph","text":"While the libvirt library has close to 400 API calls, initial versions of go-libvirt implemented only a handful of those calls. But go-libvirt is open source, so you can just add your own implementations for the routines you need, right?","spans":[]},{"type":"paragraph","text":"Well, yes you could. But go-libvirt talks to libvirt by exchanging XDR-encoded buffers using an RPC mechanism based on the venerable ONC RPC (or Sun RPC), so you would first have to familiarize yourself with those RPCs. Then, you would have to locate the argument and return value structures in the libvirt protocol definition file, and write code to marshal and unmarshal them on send and receive. By that time you might be asking yourself, “Why don’t I just give up and use CGO?” But hang on. Tedious, repetitive work; that sounds like what we invented computers for. Maybe they can help?","spans":[]},{"type":"paragraph","text":"This is the tale of how we used code generation to extend go-libvirt to cover every one of the libvirt API calls, and how we made it more resilient to future changes in the libvirt API.","spans":[]},{"type":"heading3","text":"Sun RPC and the Missing Toolchain","spans":[]},{"type":"paragraph","text":"If you’re working with Sun RPC in C, you write a protocol file describing the messages you want to exchange and feed it to a utility called “rpcgen”. The output of rpcgen includes header files and stubs for both client and server. The stubs contain generated code to marshal and unmarshal the message bodies for each of the messages. This is exactly how libvirt works— the protocol files are right there in the libvirt source repo (look for source files ending in .x), and during the build they get processed by rpcgen into .c and .h files.","spans":[{"start":411,"end":430,"type":"hyperlink","data":{"link_type":"Web","url":"https://github.com/libvirt/libvirt"}}]},{"type":"paragraph","text":"If rpcgen could output Go code we’d be all set, but it doesn’t. Sun RPC isn’t a popular option for native Go programs, and although there are libraries for handling its on-the-wire data representation—XDR—there aren’t any libraries around for parsing its protocol files into Go.","spans":[]},{"type":"paragraph","text":"Time to roll up our sleeves!","spans":[]},{"type":"heading3","text":"Learning the Language","spans":[]},{"type":"paragraph","text":"Sun RPC protocol files look a lot like a collection of C declarations. We could throw a parser together with regexes and custom code, but when the source files start getting complex that path often ends in tears. The protocol files we need to parse definitely meet the complexity threshold: like C, data types can be nested inside other data types, and this is exactly the kind of thing that regexes are ill-equipped to handle. To be reliable we’ll want a real stateful parser. We could write one, but there’s a better way.","spans":[]},{"type":"paragraph","text":"Parser generators have been around since the 1970s, and Go includes a port of one of the oldest, yacc, in `golang.org/x/tools/cmd/goyacc`. Using goyacc to generate our parser means we don’t have to write the state machine that makes up the bulk of the parser by hand (and yes, it also means our code generator is itself generated). With a generated parser we’re left with three pieces of code to write: the language grammar, which is consumed by the parser generator to build the parser state machine, the actions, which run when the parser identifies a bit of grammar, and the lexer.","spans":[]},{"type":"paragraph","text":"The grammar definition lives in its own file, `sunrpc.y`, and goyacc uses the same syntax for the contents of this file as yacc did before it. Luckily, some of the documentation for Sun RPC includes grammar definitions in exactly the format goyacc expects, and we used that as a starting point for writing the grammar. ","spans":[{"start":152,"end":218,"type":"hyperlink","data":{"link_type":"Web","url":"https://docs.oracle.com/cd/E26502_01/html/E35597/rpcproto-24229.html#scrolltoc"}}]},{"type":"paragraph","text":"The actions are simply Go code mixed in with the grammar. When the parser identifies an element of the grammar, it will execute any actions defined at that point in the grammar file. In our case, the actions build an internal representation of the protocol file that we’ll use later to output our generated Go code.","spans":[]},{"type":"paragraph","text":"The actions are simply Go code mixed in with the grammar. When the parser identifies an element of the grammar, it will execute any actions defined at that point in the grammar file. In our case, the actions build an internal representation of the protocol file that we’ll use later to output our generated Go code.","spans":[]},{"type":"heading3","text":"Alexa, Where’s My Lexer?","spans":[]},{"type":"paragraph","text":"That leaves the lexer, also called a tokenizer. The lexer is called by the parser, and each time it’s called it returns the next token in the input stream, where a token is a unit of the grammar. For our grammar, if the input stream looks like this:","spans":[]},{"type":"image","url":"https://images.prismic.io/www-static/7d467eb0-0778-4d15-9fce-0cd1e2f7e493_Screen-Shot-2017-12-12-at-6.02.33-PM.png?auto=compress,format","alt":"remote string max","copyright":null,"dimensions":{"width":1600,"height":68}},{"type":"paragraph","text":"The lexer will return the token CONST, then IDENTIFIER, =, CONSTANT, and ;. That matches one of the valid forms of `const_definition` from our grammar file (`const_ident` is elsewhere defined as `IDENTIFIER`):","spans":[]},{"type":"image","url":"https://images.prismic.io/www-static/bd5817d6-a217-4d71-85bb-4033ca81b3c0_Screen-Shot-2017-12-12-at-6.10.29-PM.png?auto=compress,format","alt":"identifier","copyright":null,"dimensions":{"width":1600,"height":660}},{"type":"paragraph","text":"The Go code inside the braces after the grammar is the action the parser will execute when it sees this sequence of tokens. So the parser will call `AddConst()`, passing in the value of the second and fourth tokens, in this case the const_ident and the CONSTANT. The resulting call will be `AddConst(“REMOTE_STRING_MAX”, “4194304”)`, because in our grammar the value of any token is the original string.","spans":[]},{"type":"paragraph","text":"If you’re familiar with yacc, at this point you might be wondering, “Where’s the Go port of lex? Is there a golex?” The answer is no; lex isn’t part of the standard library. (To get an idea of why this might be so, and an excellent introduction to lexers in general, you might want to see this talk by Rob Pike, from back in the early days of Go, in 2011.)","spans":[{"start":289,"end":310,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.youtube.com/watch?v=HxaD_trXwRE"}}]},{"type":"paragraph","text":"So instead we have a handwritten lexer, `lvlexer.go`. It’s pretty straightforward, about 330 lines long, and uses no regular expressions. To work with the parser, the lexer has to satisfy an interface consisting of two functions: `Lex()` and `Error()`.","spans":[]},{"type":"heading3","text":"Generating the Output","spans":[]},{"type":"paragraph","text":"The actions, as well as the code that drives the parser, are found in `generate.go`, which gets compiled together with the lexer and the parser into a standalone binary. The generator calls the parser, and when the parser has finished its work the generator has an internal representation of the protocol file, and we need to tie everything together and output some Go code.","spans":[]},{"type":"paragraph","text":"Up until now we’ve been talking about libvirt and Sun RPC, because libvirt is using many of the pieces that make up Sun RPC. But if you look at `remote_protocol.x` in the libvirt sources, you’ll notice something surprising: the procedure definitions, which would describe the argument and return types for each RPC procedure, are missing. There is an enum containing procedure numbers, but nothing that resembles a function prototype.","spans":[]},{"type":"paragraph","text":"This is where libvirt departs from Sun RPC. Rather than use rpcgen to build the procedure stubs for client and server, they have implemented their own method for calling remote routines (have a look at libvirt’s `callFull()` in `remote_driver.c` if you’re curious). ","spans":[]},{"type":"paragraph","text":"So instead of a procedure definition in the protocol file, the procedure, its arguments, and it’s return values are associated by name. All arguments and return values in libvirt are structures. We can start from the `remote_procedure` enum in the protocol file. For the procedure `REMOTE_NODE_ALLOC_PAGES`, we have a procedure number of 347. To find the arguments structure we convert this to lowercase and add `_args`; for return we add `_ret`. We can apply this pattern to every procedure in the protocol file. If a procedure doesn’t take arguments or return values, the corresponding struct will be missing.","spans":[]},{"type":"paragraph","text":"This gives us enough information to generate the Go client functions for each procedure. We’ll drop the `remote_` prefix, since it’s common to every procedure, and we’ll convert the names to camel case so they look natural in Go. For `REMOTE_NODE_ALLOC_PAGES`, that means our generated Go routine would look like this:","spans":[]},{"type":"paragraph","text":"![NodeAllocPages](https://assets.digitalocean.com/ghost/2018/01/Screen-Shot-2017-12-14-at-5.25.31-PM.png)","spans":[]},{"type":"paragraph","text":"That’s not a bad start, but it forces the caller to construct the arguments structure and decode the return structure. Putting all the arguments into a structure makes the function awkward to use, and doesn’t match the libvirt documentation for `virNodeAllowPages`. We can do better.","spans":[]},{"type":"heading3","text":"API, Deconstructed","spans":[]},{"type":"paragraph","text":"Because we used a real parser to process the protocol definition, our generator has full type information available. In fact for every struct definition in the protocol, we have our own Go struct containing all the type information for the struct’s elements. So the generator can replace the struct in the arguments for our generated function with a list of its contents, simply by doing this:","spans":[]},{"type":"image","url":"https://images.prismic.io/www-static/e6cb83f5-6b71-473e-8fed-48670b8e76f2_Screen-Shot-2017-12-14-at-5.41.42-PM.png?auto=compress,format","alt":"gen.procs","copyright":null,"dimensions":{"width":1518,"height":96}},{"type":"paragraph","text":"Gen.Procs is an array of generated procedures, and Args is an array of arguments. With that statement we set the arguments for a generated function to the array of members in the corresponding arguments structure. We do the same thing for the return values, and then our generated NodeAllocPages looks like this:","spans":[]},{"type":"image","url":"https://images.prismic.io/www-static/013e00f3-805a-42e0-a80a-1dfcfeb5d880_Screen-Shot-2017-12-14-at-5.47.04-PM.png?auto=compress,format","alt":"func (l *Libvirt)","copyright":null,"dimensions":{"width":1460,"height":478}},{"type":"paragraph","text":"This is very close to the libvirt documentation; the Go version just omits two size arguments since slices carry size information with them.","spans":[{"start":8,"end":47,"type":"hyperlink","data":{"link_type":"Web","url":"https://libvirt.org/html/libvirt-libvirt-host.html#virNodeAllocPages"}}]},{"type":"heading3","text":"Templates","spans":[]},{"type":"paragraph","text":"The last step in our code generator is to iterate through all the type information we’ve collected and build the actual Go files. We use Go’s `text/template` library with a couple of fairly simple template files to do this. The generated procedures are output to `libvirt.gen.go`, and some constants to a separate file, `internal/constants/constants.gen.go`. We even generate comments so that godoc has something to work with.","spans":[]},{"type":"heading3","text":"Go, Generate","spans":[]},{"type":"paragraph","text":"Code generation is exactly what the `go generate` command was created for. If you’ve never explored this part of the Go toolchain before, there’s an excellent introduction in the Go blog. The generate command is intended to be run separately from `go build`, and much less often - typically the generated code for a project only needs to be re-created when an external dependency changes. To rebuild the generated files for a particular version of libvirt, you simply set an environment variable `LIBVIRT_SOURCE` with the path to the libvirt sources, and run `go generate ./…` from the go-libvirt directory. That command will descend through the go-libvirt sources executing generate instructions embedded in the source files.","spans":[{"start":149,"end":186,"type":"hyperlink","data":{"link_type":"Web","url":"https://blog.golang.org/generate"}}]},{"type":"heading3","text":"That’s It!","spans":[]},{"type":"paragraph","text":"We hope you enjoyed this brief tour of the code generator for go-libvirt. There’s still work to be done on the project, but with these changes it should be much more useful to a wider audience.","spans":[]},{"type":"paragraph","text":"The go-libvirt project is hosted on GitHub, and pull requests are welcome!","spans":[{"start":4,"end":42,"type":"hyperlink","data":{"link_type":"Web","url":"https://github.com/digitalocean/go-libvirt"}}]},{"type":"paragraph","text":"Geoff Hickey is a Senior Engineer at DigitalOcean, where he works on hypervisors and the code that surrounds them. In his spare time he teaches machines to make things out of wood and then shamelessly takes credit for their work. As a result, he views the rise of machine learning with some trepidation.","spans":[{"start":0,"end":303,"type":"em"}]}],"blog_post_date":"2018-01-23","tags":[{"tag1":{"tag":"Engineering","_linkType":"Link.document","_meta":{"uid":"engineering"}}}],"_meta":{"uid":"extending-go-libvirt-with-code-generation"}}},{"node":{"author":{"_linkType":"Link.document","author_name":"Alejandro (Alex) Jaimes","author_image":{"dimensions":{"width":1600,"height":1067},"alt":"Alejandro (Alex) Jaimes","copyright":null,"url":"https://images.prismic.io/www-static/2ef201f3af242c6dface5c525637e5d45cac7050_ldv_20150519_0485.jpg?auto=compress,format"},"_meta":{"uid":"alejandro_alex_jaimes"}},"blog_header_image":{"dimensions":{"width":785,"height":418},"alt":"AI letters illustration","copyright":null,"url":"https://images.prismic.io/www-static/00c473ae5146ad0b8a9257482741784cfd64d74e_ai_3_blog_kasia.png?auto=compress,format"},"blog_headline":[{"type":"heading1","text":"AI in Practice","spans":[]}],"blog_post_content":[{"type":"paragraph","text":"This is the final installment in a three-part series on artificial intelligence by DigitalOcean’s Head of R&D, Alejandro (Alex) Jaimes. Read the first post about the state of AI, and the second installment about how data and models feed computing.","spans":[{"start":0,"end":247,"type":"em"},{"start":162,"end":177,"type":"hyperlink","data":{"link_type":"Web","url":"https://blog.digitalocean.com/the-state-of-ai/"}},{"start":212,"end":246,"type":"hyperlink","data":{"link_type":"Web","url":"https://blog.digitalocean.com/how-data-and-models-feed-computing/"}}]},{"type":"paragraph","text":"So what does AI as a service mean for hobbyists, professional developers, engineering teams, the open source community, and companies today?","spans":[{"start":134,"end":139,"type":"em"}]},{"type":"paragraph","text":"Starting an AI (or machine learning) project can be a daunting task at any level, and the steps should be different depending on the context. It’s important to note that sophisticated algorithms are not a requirement for AI and more often than not solutions may be simple. Even the most basic machine learning algorithm can do a decent job for some problems and once a process is set up, more sophisticated iterations are possible.","spans":[]},{"type":"paragraph","text":"An alternative is starting with sophisticated algorithms—as long as there’s a good understanding of what those algorithms do and it’s “easy” to get them up and running. You don’t want to start your first iteration setting a large number of parameters you don’t understand.","spans":[]},{"type":"paragraph","text":"There are some exceptions, and arguably, choices that depend on many factors, including level of expertise, but in general, it’s feasible to start small, build, and iterate quickly—you want to build an initial solution that demonstrates value. Even if it’s imperfect, setting up a process, and obtaining data, gets you off the ground. It’s imperative, however, to ask the right questions, focus on the solution, and the needs of who will be using whatever you build, and be resourceful and creative in combining data, models, and open source frameworks. Here’s how that applies to different players in the tech space:","spans":[]},{"type":"list-item","text":"Hobbyists have the most flexibility and can perhaps dream up the wildest ideas, albeit with very limited resources. In many ways, this puts them in the best position to explore; a perfect scenario for an iterative approach, focused initially on a proof of concept, starting with simpler algorithms and existing models and datasets. Start small and experiment—a lot. There are a ton of open source tools and datasets for machine learning. Many city governments, for example (NYC and SF are prime examples) have open data initiatives that can be leveraged.","spans":[{"start":0,"end":9,"type":"strong"}]},{"type":"list-item","text":"Professional developers and engineering teams should focus on solving very specific problems. In many “first” cases, these could evolve around cost saving, speed, efficiency, or specific product features. The “how”, however, can follow the process outlined above for hobbyists. To start, treat the project like any other, by figuring out what is needed in terms of data and other resources, defining clear metrics, working closely with your product team to ask the right questions and focusing strongly on how the solution will be used- that’s the most critical issue because it will determine what algorithms and data are required. In many cases, answers to the questions might point to simple solutions that may not initially need AI, but that will enable it later. A change in an interface, for example, can significantly impact what users do and that could make the AI problem you are trying to solve a lot easier.","spans":[{"start":0,"end":45,"type":"strong"}]},{"type":"list-item","text":"The open source community has never played a more critical role, and there’s no doubt that’s one of the reasons AI is having such an impact. Important initiatives towards the future include working on tools for cleaning, processing, and handling data, as well as tools for exchanging models and repurposing them, as well as packaging task-specific models for specific application domains so they can be easily implemented as services.","spans":[{"start":0,"end":25,"type":"strong"}]},{"type":"list-item","text":"Companies need to focus on processes to enable access to data, constant updating of models, and experimentation. The field is evolving quickly, so making AI part of the cultural fabric of the company is what’s really most critical. Algorithms will change, hardware will evolve, but the processes that enable AI have a clear path. In addition to data and experimentation, the focus should be, on one hand, on improving productivity and using AI as an enabler, and on the other hand, in having a workforce that evolves with it. That requires a strong Human-Centered perspective and a strategy that helps employees be more efficient and focused on customer needs. Internally, this means empowering developers and engineers to have flexibility in choosing the tools they use, and setting up programs to keep them constantly in the loop on product and user needs, not in AI or other “silos”.","spans":[{"start":0,"end":9,"type":"strong"}]},{"type":"heading2","text":"The Future","spans":[]},{"type":"paragraph","text":"The field is evolving extremely quickly and one could argue that most of the research being published consists mainly of experimentation, on either applying known deep learning architectures to new problems or tweaking parameters. It’s clear, however, that efforts—and progress—are being made n areas such as transfer learning, reinforcement learning, and unsupervised learning, among others. In terms of hardware, it’s too early to say, but it’s very positive to see new developments in the space.","spans":[]},{"type":"paragraph","text":"Perhaps more important than advancements in algorithms, we can expect advances in how AI augments human abilities. There will be a much tighter integration between humans and machines than what computing has created thus far. For hobbyists, professional developers, engineering teams, the open source community and companies, this really translates to having a strong human-centered focus.","spans":[]},{"type":"heading2","text":"Conclusion","spans":[]},{"type":"paragraph","text":"I’ve referred to AI throughout this series, but most of my examples relate to machine learning. One of the key differences between the two is that true AI applications will have an even stronger focus on user interaction and experience. At the end of the day, it’s the applications we build that will make a difference, AI or not. How “smart” the system is, or what algorithms it uses, won’t matter.","spans":[]},{"type":"paragraph","text":"Try your hand at Machine Learning with the DigitalOcean Machine Learning One-Click application.","spans":[{"start":43,"end":82,"type":"strong"},{"start":43,"end":82,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.digitalocean.com/products/one-click-apps/machine-learning/"}}]},{"type":"paragraph","text":"*Alejandro (Alex) Jaimes is Head of R&D at DigitalOcean. Alex enjoys scuba diving and started coding in Assembly when he was 12. In spite of his fear of heights, he's climbed a peak or two, gone paragliding, and ridden a bull in a rodeo. He's been a startup CTO and advisor, and has held leadership positions at Yahoo, Telefonica, IDIAP, FujiXerox, and IBM TJ Watson, among others. He holds a Ph.D. from Columbia University.","spans":[]},{"type":"paragraph","text":"Learn more by visiting his personal website or LinkedIn profile. Find him on Twitter: @tinybigdata.*","spans":[{"start":27,"end":43,"type":"hyperlink","data":{"link_type":"Web","url":"http://www.alexjaimes.com/"}},{"start":47,"end":63,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.linkedin.com/in/alexjaimes/"}},{"start":86,"end":98,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.twitter.com/tinybigdata"}}]}],"blog_post_date":"2017-12-05","tags":[{"tag1":{"tag":"Engineering","_linkType":"Link.document","_meta":{"uid":"engineering"}}}],"_meta":{"uid":"ai-in-practice"}}},{"node":{"author":{"_linkType":"Link.document","author_name":"Sneha Inguva","author_image":{"dimensions":{"width":1086,"height":1086},"alt":"Sneha Inguva","copyright":null,"url":"https://images.prismic.io/www-static/7ced102cd11b18e75c8d62a2af51fb429ac9e3fa_10710453_10152371929930870_2991868467032627208_o.jpg?auto=compress,format"},"_meta":{"uid":"sneha_inguva"}},"blog_header_image":{"dimensions":{"width":784,"height":418},"alt":"abstract binoculars graphic with graphs illustration","copyright":null,"url":"https://images.prismic.io/www-static/cc1e4a831dc2079d8cab32ca14ee3fec60cdbb2d_observability_blog.png?auto=compress,format"},"blog_headline":[{"type":"heading1","text":"The Why, How, and What of Metrics and Observability","spans":[]}],"blog_post_content":[{"type":"paragraph","text":"If you are reading this post, you are probably aware that DigitalOcean is an infrastructure company. (And if you weren't aware of that: surprise!) As a cloud platform provider, we have a varied tech stack ranging from the many services that power the cloud, from hardware to virtualization software, and even container orchestration tooling. But with many moving pieces comes a vital need: that of observability. Observability is often defined as consisting of three “pillars”: logging, metrics, and tracing. In this post, however, I will focus on metrics, namely how we at DigitalOcean have leveraged Prometheus and Alertmanager for whitebox monitoring both our services and our container clusters. Whether you are a software engineer writing services or an operations engineer maintaining a container orchestration solution, these monitoring and alerting examples should prove useful.","spans":[]},{"type":"heading2","text":"More to Monitoring Than “Knowing When Things Break”","spans":[]},{"type":"paragraph","text":"Before delving into the what and how of monitoring, however, I’ll first focus on the why: specifically, why you—an engineer—should monitor your applications or infrastructure stack. The immediate answer might be “to know when things break”, and naturally, alerting upon downtime or other metrics issues is a vital reason to set up monitoring. In fact, most applications at DigitalOcean have either a whitebox monitoring or blackbox monitoring setup and some basic downtime alerts (sometimes, we have both). But beyond just alerting, proper monitoring allows one to identify long-term trends, analyze performance issues, and set up visualizations. For example, every containerized application deployed on our Kubernetes clusters at DigitalOcean has an automatic dashboard generated with basic stats such as memory and CPU usage as well as traffic. Our clusters themselves also have dashboards. These are essential in visualizing general trends and for debugging during on call rotations:","spans":[]},{"type":"paragraph","text":"Fig. 1: doccserver application dashboard","spans":[{"start":0,"end":40,"type":"em"}]},{"type":"image","url":"https://images.prismic.io/www-static/937739709938d3c2b30dc359fe7104e4bbd463e2_screen-shot-2017-11-02-at-4.18.22-pm.png?auto=compress,format","alt":"doccserver application dashboard","copyright":null,"dimensions":{"width":1600,"height":715}},{"type":"paragraph","text":"I also mentioned that we currently use Prometheus and Alertmanager for both monitoring and alerting: our hows of monitoring. The journey towards using this tooling is also quite interesting and one that I’ve had the opportunity to bear witness to. DigitalOcean originally leveraged Chef and a hodgepodge of scripts or CI/CD tools for provisioning, updates, and deployments. Nagios was commonly utilized (and still is) to perform blackbox monitoring on hosts. This, however, was not enough. While blackbox monitoring is one piece of the puzzle, it doesn’t provide sufficient introspection into applications or truly help with debugging a variety of issues. As a result, engineers went through a long process of trying out several other solutions, which were always lacking in some element. Some were difficult to set up and operationally maintain while others didn’t provide useful visualizations or UX. Furthermore, actually leveraging the data for analysis proved difficult...but along came Prometheus and Alertmanager.","spans":[]},{"type":"paragraph","text":"Four base metrics types are at the core of Prometheus— counters, gauges, summaries, and histograms—which can be combined alongside a powerful query language with various functions for analysis and debugging. Prometheus proved to be extremely easy to install, either via a simple Go binary or Docker container. Furthermore, the fact that the time-series data was multidimensional proved immensely helpful, as our adoption of Prometheus coincided with our move towards containerization; having the ability to label data made analysis and aggregation all the more simple when using tools such as Kubernetes. As a result, Prometheus swiftly became our go-to tool for whitebox monitoring alongside Alertmanager for alerting.","spans":[]},{"type":"heading2","text":"Metrics: Leveraging The Four Golden Signals","spans":[]},{"type":"paragraph","text":"We’ve established the why and how of monitoring; let us now look into the what. Two categories of metrics we leverage at DigitalOcean are the four golden signals (of Google SRE-book fame) for our request-based microservices and USE metrics, which we heavily utilize to monitor our container orchestration clusters such as Kubernetes or Mesos.","spans":[]},{"type":"paragraph","text":"The four golden signals consist of latency, saturation, traffic, and error. Latency refers to the duration of requests; one important thing to note is that it is vital to consider the distribution of request duration, especially the longtail or 99th percentile. After all, what affects only a few of your users can often be an indication of impending saturation—another golden signal. Saturation itself is defined as the “fullness” of a system; it indicates how long something is waiting to be serviced. As a result, we often carefully track an alert based upon 95th or 99th percentile request latencies:","spans":[]},{"type":"paragraph","text":"Fig. 2: Kube-dns 95th percentile request latency in ms","spans":[{"start":0,"end":54,"type":"em"}]},{"type":"image","url":"https://images.prismic.io/www-static/6bfe92349c292172693a7142c3493812714686c1_screen-shot-2017-11-02-at-5.40.58-pm.png?auto=compress,format","alt":"Kube-dns 95th percentile request latency in ms","copyright":null,"dimensions":{"width":1600,"height":562}},{"type":"paragraph","text":"Note that generating these metrics, and subsequently configuring alerts, is fairly easy using the Prometheus histogram metric type:","spans":[]},{"type":"preformatted","text":"histogram_quantile(0.95, sum(rate(kubedns_probe_kubedns_latency_ms_bucket[1m])) BY (le, kubernetes_pod_name)) > 1000  \n","spans":[]},{"type":"paragraph","text":"As histogram metrics are collected as counts in various buckets, we merely need to specify which percentile measurement we would like to calculate and leverage the histogram_quantile function. It is also possible to calculate latency quantiles using the summary metric by specifying the exact quantile we wish to track. While this may reduce quantile estimation error, it is a more expensive client side calculation.","spans":[]},{"type":"paragraph","text":"Now onto the final two golden signals! Traffic refers to the amount of demand placed on your system. In a request-based system, this is often measured in HTTP requests per second. In order to measure traffic using Prometheus, we often instrument our applications to expose a request count metric that is monotonically increasing and then calculate the per-second rate. Why? Looking at a constantly increasing counts alone would not provide any indication of abnormal traffic increases or decreases:","spans":[]},{"type":"paragraph","text":"Fig. 3: Note that request count constantly increases.","spans":[{"start":0,"end":53,"type":"em"}]},{"type":"image","url":"https://images.prismic.io/www-static/78fdfd1a9cd78b1c8cd78af889093d18d026c51c_screen-shot-2017-10-13-at-11.09.45-am.png?auto=compress,format","alt":"Note that request count constantly increases","copyright":null,"dimensions":{"width":1600,"height":708}},{"type":"paragraph","text":"However, looking at rate of requests gives us a meaningful idea of traffic; we can then subsequently set up alerts for per-second rate exceeding above or dropping below a particular value:","spans":[]},{"type":"paragraph","text":"Fig. 4:  By applying rate() over a 15 minute window, we get a meaningful idea of traffic.","spans":[{"start":0,"end":89,"type":"em"},{"start":21,"end":27,"type":"strong"}]},{"type":"image","url":"https://images.prismic.io/www-static/de3c47c6bdefb7bbcf62e55b6f6c49224abff80f_screen-shot-2017-10-13-at-11.11.22-am.png?auto=compress,format","alt":null,"copyright":null,"dimensions":{"width":1600,"height":711}},{"type":"paragraph","text":"Error rates of failing or succeeding requests—the final golden signal—are calculated similarly. Applications are instrumented to expose error count with labels indicating status code or other information; per-second rate is then calculated to provide a meaningful metric:","spans":[]},{"type":"paragraph","text":"Fig. 5: Per-second error rate for cassandra","spans":[{"start":0,"end":43,"type":"em"}]},{"type":"image","url":"https://images.prismic.io/www-static/a92fa433631daee608f47f82fdfede307b2fb917_screen-shot-2017-11-02-at-7.42.37-pm.png?auto=compress,format","alt":"Per-second error rate for cassandra","copyright":null,"dimensions":{"width":1600,"height":667}},{"type":"paragraph","text":"Note that the per-second rate suddenly spiked but decreased. However, as this was lower than  the alert configured below (at 50 errors per-second), no alert was triggered:","spans":[]},{"type":"preformatted","text":"sum(rate(cassandra_query_latency_count{docc_app=~\"timeseries-ingest-index[0-9]+\",result=\"error\"}[5m])) > 50  \n","spans":[]},{"type":"paragraph","text":"Using these four golden signals, we can gain a basic idea of both the health of our request-based services as well as an idea of our end user’s experiences. We can both visualize these metrics on a dashboard and also set up alerts for any abnormal metrics.","spans":[]},{"type":"paragraph","text":"In addition to monitoring our services, we also monitor our infrastructure. As a former member of the team that maintained our container clusters, I noticed enormous benefits when leveraging the USE method: utilization, saturation, and errors. Coined by Brendan Gregg, the USE method allows one to solve “80% of server issues with 5% of the effort”.","spans":[{"start":269,"end":348,"type":"hyperlink","data":{"link_type":"Web","url":"http://www.brendangregg.com/usemethod.html"}}]},{"type":"paragraph","text":"Let us take a look at how we leveraged these metrics to monitor our Kubernetes clusters. Each cluster consists of multiple worker nodes known as kubelets. Monitoring overall CPU, memory utilization, and reservation on these nodes has proven essential:","spans":[]},{"type":"paragraph","text":"Fig. 6: Kubernetes worker node CPU utilization for a single worker node","spans":[{"start":0,"end":71,"type":"em"}]},{"type":"image","url":"https://images.prismic.io/www-static/1b455355a13fb3166aef62e50ef6bc1c06d3f864_screen-shot-2017-11-22-at-3.03.11-pm.png?auto=compress,format","alt":"Kubernetes worker node CPU utilization for a single worker node","copyright":null,"dimensions":{"width":1600,"height":806}},{"type":"paragraph","text":"Note that CPU utilization is measured in CPU seconds, a constantly increasing counter. As a result, calculating the per-second rate of CPU seconds gives us the number of CPU cores being utilized at a given time. We can further tweak this calculation to craft an alert to detect greater than a given percentage of CPU utilization on a worker node. (If interested in how exactly to do this, be sure to check out this blog post.)","spans":[]},{"type":"paragraph","text":"Another key component of our Kubernetes architecture is our HAProxy ingress load balancers; these are components in our network stack and direct much of outside traffic to appropriate services within the cluster. As you can imagine, load balancer connection utilization and saturation are therefore vital to measure:","spans":[]},{"type":"paragraph","text":"Fig. 7: Ingress load balancer connection utilization as a %","spans":[{"start":0,"end":59,"type":"em"}]},{"type":"image","url":"https://images.prismic.io/www-static/180e998c35e2802132d9300d468b3d88d7edb5dd_screen-shot-2017-11-02-at-8.37.06-pm.png?auto=compress,format","alt":"Ingress load balancer connection utilization as a %","copyright":null,"dimensions":{"width":1600,"height":671}},{"type":"paragraph","text":"If all connections are utilized to an ingress load balancer, no additional connections can be made until some are dropped. As a result, we also elect to alert for greater than 50% load balancer connection utilization:","spans":[]},{"type":"preformatted","text":"max(haproxy_frontend_current_sessions / haproxy_frontend_limit_sessions) BY (kubernetes_node_name, frontend) * 100 > 50  \n","spans":[]},{"type":"heading2","text":"Conclusion","spans":[]},{"type":"paragraph","text":"And there you have it— a small slice of our monitoring and alerting setup at DigitalOcean! With these few examples, hopefully you can see how and why we have elected to leverage the four golden signals and the USE method to monitor and alert on our microservices and container clusters. Doing so has allowed both ops teams and service owners to maintain observability of running applications and key infrastructure. We have also leveraged this data to track long-term trends and look into improving performance. Hopefully you can do the same!","spans":[]},{"type":"heading3","text":"Sources:","spans":[]},{"type":"paragraph","text":"[1] Logs and Metrics, Cindy Sridharan. https://medium.com/@copyconstruct/logs-and-metrics-6d34d3026e38","spans":[{"start":39,"end":102,"type":"hyperlink","data":{"link_type":"Web","url":"https://medium.com/@copyconstruct/logs-and-metrics-6d34d3026e38"}}]},{"type":"paragraph","text":"[2] Chapter 6: Monitoring Distributed Systems. Site Reliability Engineering at Google. https://landing.google.com/sre/book/index.html","spans":[{"start":87,"end":133,"type":"hyperlink","data":{"link_type":"Web","url":"https://landing.google.com/sre/book/index.html"}}]},{"type":"paragraph","text":"[3] USE Method. Brendan Gregg. http://www.brendangregg.com/usemethod.html","spans":[{"start":31,"end":73,"type":"hyperlink","data":{"link_type":"Web","url":"http://www.brendangregg.com/usemethod.html"}}]},{"type":"paragraph","text":"[4] An Appropriate Use of Metrics. Martin Fowler. https://martinfowler.com/articles/useOfMetrics.html","spans":[{"start":50,"end":101,"type":"hyperlink","data":{"link_type":"Web","url":"https://martinfowler.com/articles/useOfMetrics.html"}}]},{"type":"paragraph","text":"[5] Monitoring and Observability. Cindy Sridharan. https://medium.com/@copyconstruct/monitoring-and-observability-8417d1952e1c","spans":[{"start":51,"end":126,"type":"hyperlink","data":{"link_type":"Web","url":"https://medium.com/@copyconstruct/monitoring-and-observability-8417d1952e1c"}}]},{"type":"paragraph","text":"Sneha Inguva is an enthusiastic software engineer working on building developer tooling at DigitalOcean. Previously, Sneha worked at a number of startups. Her experience across an eclectic range of verticals, from education to 3D printing to casinos, has given her a unique perspective on building and deploying software. When she isn't bashing away on a project or reading about the latest emerging technology, Sneha is busy molding the minds of young STEM enthusiasts in local NYC schools.","spans":[{"start":0,"end":491,"type":"em"}]}],"blog_post_date":"2017-11-29","tags":[{"tag1":{"tag":"Engineering","_linkType":"Link.document","_meta":{"uid":"engineering"}}}],"_meta":{"uid":"observability-and-metrics"}}},{"node":{"author":{"_linkType":"Link.document","author_name":"Luca Salvatore","author_image":{"dimensions":{"width":250,"height":250},"alt":"Luca Salvatore","copyright":null,"url":"https://images.prismic.io/www-static/fd8fb2a54e8e54d882c33bccac82b22a684d920e_9bb2be860884302b74920173da25866a.jpg?auto=compress,format"},"_meta":{"uid":"luca_salvatore"}},"blog_header_image":{"dimensions":{"width":784,"height":418},"alt":"DigitalOcean logo illustration","copyright":null,"url":"https://images.prismic.io/www-static/23626aa48384759b9118769a56f86f991bbfd315_whatsnewonthedonetwork_blog_pat.png?auto=compress,format"},"blog_headline":[{"type":"heading1","text":"What's New With the DigitalOcean Network","spans":[]}],"blog_post_content":[{"type":"paragraph","text":"Early this year the network engineering team at DigitalOcean embarked on a fairly ambitious project. We were thinking about areas of our network that needed improvement both for our customers and for our internal systems. One of the key things that we strive for at DO is to provide our customers with a stable and high performing cloud platform. As we continue to grow and release new products, it becomes clear that network infrastructure is a critical component and it must keep up with our customers needs. In order to allow our customers to grow, the network must be able to scale, it must be performant, and above all, must be reliable.","spans":[]},{"type":"paragraph","text":"With those factors in mind, we went to work building out the DigitalOcean global backbone. It’s not finished yet, but we wanted to share what has been done so far, what is in progress, and what the end state will be.","spans":[]},{"type":"heading2","text":"Creating a Backbone Network","spans":[]},{"type":"paragraph","text":"DigitalOcean currently operates 12 datacenter regions (DCs) all around the world. Up until recently, these datacenters have functioned as independent “island” networks. This means that if you have Droplets in multiple locations and they need to communicate with each other, that communication goes across the public internet. For the most part, that “just works”, but the internet is susceptible to a multitude of potential problems: ISPs can have technical problems, congestion is common, and there are malicious attacks that can cause widespread issues. If you have an application that requires communication between multiple regions, the factors mentioned above could throw a wrench in even the most well designed system. To mitigate this risk, we are building our own backbone network.","spans":[]},{"type":"paragraph","text":"A backbone network allows us to interconnect our DCs using a variety of technologies such as dark fiber and wavelengths. This means that communication between DO locations no longer needs to traverse the public internet. Instead, traffic between locations runs over dedicated links that DigitalOcean manages. This gives our customers predictable and reliable transport between regions. Predictable and reliable are the key words here, and this is immensely important for anyone who is building mission critical applications. It allows developers and engineers to know exactly how their application will perform, and feel safe in the fact that their traffic is running over dedicated and redundant infrastructure.","spans":[]},{"type":"paragraph","text":"Our customers have probably noticed a number of “Network Maintenance Notifications” that we’ve sent out. In order to build out our backbone and ensure that it is scalable, reliable, and performant, we’ve had to make a number of changes to our existing network infrastructure. This includes software upgrades, new hardware, and a number of complex configuration changes. The end result will ensure that our current and future customers will benefit from all of this work.","spans":[]},{"type":"paragraph","text":"Now, onto the details. This is what we have built so far, and what we'll  build in the future.","spans":[]},{"type":"heading2","text":"Networking Through DO-Owned Links","spans":[]},{"type":"paragraph","text":"We’ve interconnected our three NYC locations; All Droplet-to-Droplet traffic between NYC1, NYC2, and NYC3 now traverses DO-owned links. Latency is predictable and stable, and packet loss is nonexistent.","spans":[]},{"type":"paragraph","text":"We’ve done the same thing around all of our European locations: LON1, AMS2, AMS3, and FRA1 are all now interconnected together. Again, all traffic between Droplets within the EU now stays within the DO network. Here is how it looks:","spans":[]},{"type":"image","url":"https://images.prismic.io/www-static/9f23c1291d8e12155deb9a29e7dcecd72fdbf5c4_pasted-image-0.png?auto=compress,format","alt":"Active and provisioning EU networks","copyright":null,"dimensions":{"width":1161,"height":817}},{"type":"paragraph","text":"We’ve also provisioned transatlantic links connecting our NYC regions to our European regions. This means that your communication between NYC and any datacenter in Europe also stays within our own network:","spans":[]},{"type":"image","url":"https://images.prismic.io/www-static/ff5e1d0a1bb2fbfc15fccc84b2022342863d5c9f_na---eu-fiber.jpg?auto=compress,format","alt":"Transatlantic links","copyright":null,"dimensions":{"width":1570,"height":660}},{"type":"paragraph","text":"Adding more to the mix, we’ve connected our NYC locations to our two facilities in California, SFO1 and SFO2.  All communication around North America as well as communication within and to Europe now stays within the DO backbone:","spans":[]},{"type":"image","url":"https://images.prismic.io/www-static/ea41f20b50d226c02409586fee4f62f8187fe39f_na-fiber-1.jpg?auto=compress,format","alt":"North American network","copyright":null,"dimensions":{"width":1427,"height":743}},{"type":"paragraph","text":"Next up will be connectivity from the SFO region to SGP1. We also have plans to link Singapore to Europe which is slated for Q1 2018 as well as TOR1 to NYC. Once fully completed, the DO global backbone will look like this:","spans":[]},{"type":"image","url":"https://images.prismic.io/www-static/fce7e20463050252ad6de03b08e808e467dedf6f_do-global-network-1.jpg?auto=compress,format","alt":"DO global network","copyright":null,"dimensions":{"width":1569,"height":792}},{"type":"paragraph","text":"We are very excited about what these upgrades mean for DO and for you, our users. We’re continually striving to create better performing and more reliable infrastructure, and I feel that these upgrades to the network will set the stage for some really awesome things to be built on top of the DO platform.","spans":[]},{"type":"paragraph","text":"Luca Salvatore is currently the manager of the Networking Engineering Team at DigitalOcean. Over the past decade Luca has held various network engineering roles both in Australia and the USA.  He has designed and built large enterprise and datacenter networks and has first hand experience dealing with massively scalable networks such as DigitalOcean.  He has been working in the cloud networking space for the past 5 years and is committed to peering and an open internet for all.","spans":[{"start":0,"end":482,"type":"em"},{"start":445,"end":452,"type":"hyperlink","data":{"link_type":"Web","url":"https://blog.digitalocean.com/connect-the-dots-with-internet-peering-at-ix-points/"}}]}],"blog_post_date":"2017-10-18","tags":[{"tag1":{"tag":"Engineering","_linkType":"Link.document","_meta":{"uid":"engineering"}}}],"_meta":{"uid":"whats-new-with-the-digitalocean-network"}}},{"node":{"author":{"_linkType":"Link.document","author_name":"Matt Layher","author_image":null,"_meta":{"uid":"matt_layher"}},"blog_header_image":{"dimensions":{"width":1200,"height":640},"alt":null,"copyright":null,"url":"https://images.prismic.io/www-static/e29e9451-61cf-4541-9722-ddb1d5c78b6f_DivingIntoTheDepths_blog_pat.png?auto=compress,format"},"blog_headline":[{"type":"heading1","text":"Cthulhu: Organizing Go Code in a Scalable Repo","spans":[]}],"blog_post_content":[{"type":"paragraph","text":"At DigitalOcean, we’ve used a “mono repo” called cthulhu to organize our Go code for nearly three years. A mono repo is a monolithic code repository which contains many different projects and libraries. Bryan Liles first wrote about cthulhu in early 2015, and I authored a follow-up post in late 2015.","spans":[{"start":221,"end":240,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.digitalocean.com/company/blog/taming-your-go-dependencies/"}},{"start":273,"end":287,"type":"hyperlink","data":{"link_type":"Web","url":"https://blog.gopheracademy.com/advent-2015/go-in-a-monorepo/"}}]},{"type":"paragraph","text":"A lot has changed over the past two years. As our organization has scaled, we have faced a variety of challenges while scaling cthulhu, including troubles with vendoring, CI build times, and code ownership. This post will cover the state of cthulhu as it is today, and dive into some of the benefits and challenges of using a mono repo for all of our Go code at DigitalOcean.","spans":[]},{"type":"heading2","text":"Cthulhu Overview","spans":[]},{"type":"paragraph","text":"Our journey using Go with a mono repo began in late 2014. Since then, the repository, called \"cthulhu\", has grown exponentially in many ways. As of October 6th, 2017, cthulhu has:","spans":[]},{"type":"list-item","text":"28,639 commits","spans":[]},{"type":"list-item","text":"824 branches","spans":[]},{"type":"list-item","text":"142 contributors","spans":[]},{"type":"list-item","text":"830,434 lines of DigitalOcean-authored Go code","spans":[]},{"type":"list-item","text":"2,136,373 lines of vendored Go code","spans":[]},{"type":"paragraph","text":"As the scale of the repository has grown over the past three years, it has introduced some significant tooling and organizational challenges.","spans":[]},{"type":"paragraph","text":"Before we dive into some of these challenges, let’s discuss how cthulhu is structured today (some files and directories have been omitted for brevity):","spans":[]},{"type":"preformatted","text":"```[php]{`","spans":[]},{"type":"paragraph","text":"    cthulhu  ","spans":[]},{"type":"paragraph","text":"    ├── docode","spans":[]},{"type":"paragraph","text":"    │   └── src","spans":[]},{"type":"paragraph","text":"    │       └── do","spans":[]},{"type":"paragraph","text":"    │           ├── doge","spans":[]},{"type":"paragraph","text":"    │           ├── exp","spans":[]},{"type":"paragraph","text":"    │           ├── services","spans":[]},{"type":"paragraph","text":"    │           ├── teams","spans":[]},{"type":"paragraph","text":"    │           ├── tools","spans":[]},{"type":"paragraph","text":"    │           └── vendor","spans":[]},{"type":"paragraph","text":"    └── script","spans":[]},{"type":"paragraph","text":"    `}```","spans":[]},{"type":"paragraph","text":"`docode/` is the root of our `GOPATH`. Readers of our previous posts may notice that `third_party` no longer exists, and `do/` is now the prefix for all internal code.","spans":[]},{"type":"paragraph","text":"## Code Structure","spans":[]},{"type":"paragraph","text":"All Go code lives within our `GOPATH`, which starts at `cthulhu/docode`.  Each directory within the `do/` folder has a unique purpose, although we have deprecated the use of `services/` and `tools/` for the majority of new work.","spans":[]},{"type":"paragraph","text":"`doge/` stands for “DigitalOcean Go Environment”, our internal “standard library”. A fair amount of code has been added and removed from `doge/` over time, but it still remains home to a great deal of code shared across most DigitalOcean services. Some examples include our internal logging, metrics, and gRPC interaction packages.","spans":[]},{"type":"paragraph","text":"`exp/` is used to store experimental code: projects which are in a work-in-progress state and may never reach production. Use of `exp/` has declined over time, but it remains a useful place to check in prototype code which may be useful in the future.","spans":[]},{"type":"paragraph","text":"`services/` was once used as a root for all long-running services at DO. Over time, it became difficult to keep track of ownership of code within this directory, and it was replaced by the `teams/` directory.","spans":[]},{"type":"paragraph","text":"`teams/` stores code owned by specific teams. As an example, a project called “hypervisor” owned by team “compute” would reside in `do/teams/compute/hypervisor`. This is currently the preferred method for organizing new projects, but it has its drawbacks as well. More on this later on.","spans":[]},{"type":"paragraph","text":"`tools/` was once used to store short-lived programs used for various purposes. These days, it is mostly unused except for CI build tooling, internal static analysis tools, etc. The majority of team-specific code that once resided in `tools/` has been moved to `teams/`.","spans":[]},{"type":"paragraph","text":"Finally, `vendor/` is used to store third-party code which is vendored into cthulhu and shared across all projects. We recently added the prefix `do/` to all of our Go code because existing Go vendoring solutions did not work well when `vendor/` lived at the root of the `GOPATH` (as was the case with our old `third_party/` approach).","spans":[]},{"type":"paragraph","text":"`script/` contains shell scripts which assist with our CI build process. These scripts perform tasks such as static analysis, code compilation and testing, and publishing newly built binaries.","spans":[]},{"type":"heading2","text":"CI Build Tooling","spans":[]},{"type":"paragraph","text":"One of the biggest advantages of using a mono repo is being able to effectively make large, cross-cutting changes to the entire repository without fear of breaking any “downstream” repositories. However, as the amount of code within cthulhu has grown, our CI build times have grown exponentially.","spans":[]},{"type":"paragraph","text":"Even though Go code builds rather quickly, in early 2016, CI builds took an average of 20 minutes to complete. This resulted in extremely slow development cycles. If a poorly written test caused a spurious failure elsewhere in the repo, the entire build could fail, causing a great deal of frustration for our developers.","spans":[]},{"type":"paragraph","text":"After experiencing a great deal of pain because of slow and unreliable builds, one of our engineers, Justin Hines, set out to solve the problem once and for all. After a few hours of work, he authored a build tool called `gta`, which stands for “Go Test Auto”. `gta` inspects the git history to determine which files changed between master and a feature branch, and uses this information to determine which packages must be tested for a given build (including packages that import the changed package).","spans":[]},{"type":"paragraph","text":"As an example, suppose a change is committed which modifies a package, `do/teams/example/droplet`. Suppose this package is imported by another package, `do/teams/example/hypervisor`. `gta` is used to inspect the git history and determine that both of these packages must be tested, although only the first package was changed.","spans":[]},{"type":"paragraph","text":"For very large changes, it can occasionally be useful to test the entire repository, regardless of which files have actually changed. Adding “force-test” anywhere in a branch name disables the use of `gta` in CI builds, restoring the old default behavior of “build everything for every change”.","spans":[]},{"type":"paragraph","text":"The introduction of `gta` into our CI build process dramatically reduced the amount of time taken by builds. An average build now takes approximately 2-3 minutes—a dramatic improvement over the 20 minute builds of early 2016. This tool is used almost everywhere in our build pipeline, including static analysis checks, code compilation and testing, and artifact builds and deployment.","spans":[]},{"type":"heading2","text":"Static Analysis Tooling","spans":[]},{"type":"paragraph","text":"Every change committed to cthulhu is run through a bevy of static analysis checks, including tools such as `gofmt`, `go vet`, `golint`, and others. This ensures a high level of quality and consistency between all of our Go code. Some teams have even introduced additional tools such as `staticcheck` for code that resides within their `teams/` folder.","spans":[]},{"type":"paragraph","text":"We have also experimented with the creation of custom linting tools that resolve common problems found in our Go code. One example is a tool called `buildlint` that checks for a blessed set of build tags, ensuring that tags such as `!race` (exclude this file from race detector builds) cannot be used.","spans":[]},{"type":"paragraph","text":"Static analysis tools are incredibly valuable, but it can be tricky to introduce a new tool into the repository.  Before we decided to run `golint` in CI, there were nearly 1,500 errors generated by the tool for the entirety of cthulhu.  It took a concerted effort and several Friday afternoons to fix all of these errors, but it was well worth the effort. Our internal `godoc` instance now provides a vast amount of high quality documentation for every package that resides within cthulhu.","spans":[]},{"type":"heading2","text":"Challenges","spans":[]},{"type":"paragraph","text":"While there are many advantages to the mono repo approach, it can be challenging to maintain as well.","spans":[]},{"type":"paragraph","text":"Though many different teams contribute to the repository, it can be difficult to establish overall ownership of the repository, its tooling, and its build pipelines. In the past, we tried several different approaches, but most were unsuccessful due to the fact that customer-facing project work typically takes priority over internal tooling improvements. However, this has recently changed, and we now have engineers working specifically to improve cthulhu and our build pipelines, alongside regular project work. Time will tell if this approach suits our needs.","spans":[]},{"type":"paragraph","text":"The issue of code vendoring remains unsolved, though we have made efforts to improve the situation. As of now, we use the tool “govendor” to manage our third-party dependencies.  The tool works well on Linux, but many of our engineers who run macOS have reported daunting issues while running the tool locally. In some cases, the tool will run for a very long time before completion. In others, the tool will eventually fail and require deleting and re-importing a dependency to succeed.  In the future, we’d also like to try out “dep”, the “official experiment” vendoring tool for the Go project. At this time, GitHub Enterprise does not support Go vanity imports, which we would need to make use of dep.","spans":[{"start":701,"end":704,"type":"hyperlink","data":{"link_type":"Web","url":"https://github.com/golang/dep/issues/174"}}]},{"type":"paragraph","text":"As with most companies, our organizational structure has also evolved over time. Because we typically work in the `teams/` directory in cthulhu, this presents a problem. As of now, our code structure is somewhat reliant on our organizational structure. Because of this, code in `teams/` can become out of sync with the organizational structure, causing issues with orphaned code, or stale references to teams that no longer exist. We don’t have a concrete solution to this problem yet, but we are considering creating a discoverable “project directory service” of sorts so that our code structure need not be tied to our organizational structure.","spans":[]},{"type":"paragraph","text":"Finally, as mentioned previously, scaling our CI build process has been a challenge over time. One problem in particular is that non-deterministic or “flappy” tests can cause spurious failures in unrelated areas of the repository.  A test typically flaps when it relies on some assumption which cannot be guaranteed, such as timing or ordering of concurrent operations. This problem is compounded when interacting with a service such as MySQL in an integration test. For this reason, we encourage our engineers to do everything in their power to make their tests as deterministic as possible. ","spans":[]},{"type":"heading2","text":"Summary","spans":[]},{"type":"paragraph","text":"We’ve been using cthulhu for three years at DigitalOcean, and while we’ve faced some significant hurdles along the way, the mono repo approach has been a huge benefit to our organization as a whole. Over time, we’d like to continue sharing our knowledge and tools, so that others can reap the benefits of a mono repo just as we have. ","spans":[]},{"type":"paragraph","text":"*Matt Layher is a software engineer on the Network Services team, and a regular contributor to a wide variety of open source networking applications and libraries written in Go. You can find Matt on Twitter and GitHub.*","spans":[{"start":199,"end":206,"type":"hyperlink","data":{"link_type":"Web","url":"https://twitter.com/mdlayher"}},{"start":211,"end":217,"type":"hyperlink","data":{"link_type":"Web","url":"https://github.com/mdlayher"}}]}],"blog_post_date":"2017-10-10","tags":[{"tag1":{"tag":"Engineering","_linkType":"Link.document","_meta":{"uid":"engineering"}}}],"_meta":{"uid":"cthulhu-organizing-go-code-in-a-scalable-repo"}}},{"node":{"author":{"_linkType":"Link.document","author_name":"Alejandro (Alex) Jaimes","author_image":{"dimensions":{"width":1600,"height":1067},"alt":"Alejandro (Alex) Jaimes","copyright":null,"url":"https://images.prismic.io/www-static/2ef201f3af242c6dface5c525637e5d45cac7050_ldv_20150519_0485.jpg?auto=compress,format"},"_meta":{"uid":"alejandro_alex_jaimes"}},"blog_header_image":{"dimensions":{"width":785,"height":418},"alt":"AI letter illustration","copyright":null,"url":"https://images.prismic.io/www-static/3dc3d453e2c54f4c2ef5be82c48a6b38ca07a854_ai_2_blog_kasia.png?auto=compress,format"},"blog_headline":[{"type":"heading1","text":"How Data and Models Feed Computing","spans":[]}],"blog_post_content":[{"type":"paragraph","text":"This post is the second in a three-part series on artificial intelligence by DigitalOcean’s Head of R&D, Alejandro (Alex) Jaimes. (Click here to read the first installment.)","spans":[{"start":0,"end":173,"type":"em"},{"start":131,"end":171,"type":"hyperlink","data":{"link_type":"Web","url":"https://blog.digitalocean.com/the-state-of-ai/"}}]},{"type":"paragraph","text":"Not every company, nor every developer will have the resources or the time to collect vast amounts of data to create models from scratch. Fortunately, the same repetition that I described in my last post occurs within and across industries. Because of this, particularly with deep learning, we’ve seen two very important trends:","spans":[]},{"type":"list-item","text":"(1) creation and sharing of public data to build models; and","spans":[]},{"type":"list-item","text":"(2) sharing of the models themselves even when the data is not released.","spans":[]},{"type":"paragraph","text":"While the companies that have the most data may never release it, such data is not a requirement for every problem. It’s clear, however, that teams that leverage existing public models and combine public and proprietary datasets will have a competitive advantage. They must be “smart” about how they use and leverage the data they are able to collect, again with an AI mindset and strategy in mind.","spans":[]},{"type":"heading2","text":"Supervised and Unsupervised Learning","spans":[]},{"type":"paragraph","text":"The majority of successes in AI so far have been based on supervised learning, in which machine learning algorithms are fed with labeled data—labeled data refers to a sample group that can be identified with a meaningful label or tag—versus unlabeled data. Labeling data is expensive, time consuming, and difficult (e.g., maintaining the desired quality, dealing with subjectivity, etc). For this reason, the ideal algorithms will be “unsupervised”—in other words, learning from unlabeled data. While promising, those algorithms have not shown the success levels needed to have the desired impact. Teams should then rely on creative strategies to leverage existing datasets, and combine supervised and unsupervised methods for now.","spans":[]},{"type":"paragraph","text":"A number of companies offer labeling and data collection services. But there are ways to use algorithms to simplify the manual labeling process (e.g., with a “small” dataset one can create an algorithm that labels a much larger unlabeled dataset, so that humans have to correct errors made by the algorithm instead of labeling all of the data from scratch), or to create synthetic datasets (e.g., by using algorithms to generate “fake” data that looks like the original data). The bottom line is that no matter what size the project is, there are almost always alternatives to either obtain new data or augment existing datasets.","spans":[]},{"type":"heading2","text":"AI as a Service","spans":[]},{"type":"paragraph","text":"Generally, significant efforts are required in developing models to perform tasks in accurate, efficient ways. For that reason, many companies and teams focus on specific verticals—building functionalities that are limited, but that work well in practice (versus the ideal of building a “human-like” AI capable of doing many things at once).","spans":[]},{"type":"paragraph","text":"In some cases, those functionalities can be applied across domains. Developing a speech recognition system from scratch, for example, is a major effort, and most companies and teams that need it would be better off using a service than building it from scratch.","spans":[]},{"type":"paragraph","text":"As the AI industry advances, we can expect to see more and more of those functionalities coming from specific vendors and open source initiatives, similar to the way software is built today: combinations of libraries, APIs, and open source and commercial components, coupled with custom software for specific applications.","spans":[]},{"type":"paragraph","text":"In addition, given the nature of AI, building an infrastructure that quickly scales as needs shift is a major challenge. This implies that running AI will mostly happen on the cloud. Note that in the new AI computing paradigm, growing datasets, experimentation, and constant “tweaking” of models is a critical component.","spans":[]},{"type":"paragraph","text":"Therefore, AI will be used as a cloud-based service for many applications. That’s a natural progression and in many ways leads to the commoditization of AI, which will lead to greater efficiency, opportunities, innovation, and positive economic impact. In our next installment, we’ll explore what all of this means for today’s developers.","spans":[]},{"type":"paragraph","text":"In line with the trends we’re seeing in research and industry, we’re releasing a powerful set of tools that allow developers to easily re-use existing models, work with large quantities of data, and easily scale, on the cloud. We encourage you to take a look at our machine learning one-click. What other tools or functionalities would you be interested in having us provide? Feel free to leave feedback in the comments section below.","spans":[{"start":266,"end":292,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.digitalocean.com/community/tutorials/how-to-use-the-machine-learning-one-click-install-image-on-digitalocean"}}]},{"type":"paragraph","text":"*Alejandro (Alex) Jaimes is Head of R&D at DigitalOcean. Alex enjoys scuba diving and started coding in Assembly when he was 12. In spite of his fear of heights, he's climbed a peak or two, gone paragliding, and ridden a bull in a rodeo. He's been a startup CTO and advisor, and has held leadership positions at Yahoo, Telefonica, IDIAP, FujiXerox, and IBM TJ Watson, among others. He holds a Ph.D. from Columbia University.","spans":[]},{"type":"paragraph","text":"Learn more by visiting his personal website or LinkedIn profile. Find him on Twitter: @tinybigdata.*","spans":[{"start":86,"end":98,"type":"hyperlink","data":{"link_type":"Web","url":"twitter.com/tinybigdata"}}]}],"blog_post_date":"2017-08-31","tags":[{"tag1":{"tag":"Engineering","_linkType":"Link.document","_meta":{"uid":"engineering"}}}],"_meta":{"uid":"how-data-and-models-feed-computing"}}},{"node":{"author":{"_linkType":"Link.document","author_name":"Dave (Dizzy) Smith","author_image":{"dimensions":{"width":400,"height":400},"alt":"Dave (Dizzy) Smith","copyright":null,"url":"https://images.prismic.io/www-static/bf8843f291543bd23476b7d50ce843257205a220_wpymhhwq_400x400.jpg?auto=compress,format"},"_meta":{"uid":"dave_dizzy_smith"}},"blog_header_image":{"dimensions":{"width":784,"height":418},"alt":"abstract illustration of sending chat messages","copyright":null,"url":"https://images.prismic.io/www-static/03a714fa1cf1fd8b0f53e7e928818e9ed996e640_distributed-teams_blog.png?auto=compress,format"},"blog_headline":[{"type":"heading1","text":"How to Manage, Build, and Nurture Distributed Teams","spans":[]}],"blog_post_content":[{"type":"paragraph","text":"This blog post was adapted from Dizzy’s OSCON 2017 talk, “Managing, Nurturing, and Building Distributed Teams”.","spans":[{"start":0,"end":111,"type":"em"},{"start":58,"end":109,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.youtube.com/watch?v=1qI0tJ6Edhw&amp;feature=youtu.be"}}]},{"type":"paragraph","text":"There’s a lot of talk out there about what worked for one company or another when it comes to distributed teams. The challenge we all face is how to make successful, distributed teams reproducible. Lots of companies have made it work—by luck or by force of personality. I, however, want to engineer it using effective communication.","spans":[]},{"type":"paragraph","text":"And at the end of my life, I want to look back and know I made the world a tiny bit better for people who create amazing things. This is the work of a manager—to create environments in which people do the best work of their lives. It is especially important in a distributed team since you can’t rely on personality or stage presence; you must be disciplined and focused, and work deliberately to construct these environments. Most importantly, you have to understand the forces you face and how to counteract them.","spans":[]},{"type":"heading2","text":"How Distributed Teams Differ From Other Teams","spans":[]},{"type":"paragraph","text":"A distributed team can be succinctly summed up as: “People + Work - Location; United by purpose”.","spans":[]},{"type":"paragraph","text":"There are lots of implications related to the removal of location. Without location, managers have to find something more fundamental to bind people together. As managers, we must consider how to unite distributed teams by rallying them around a collective purpose. There are forces at work making it difficult for distributed teams to unite—and stay united—around any purpose. Thankfully, there are ways to counteract these forces, too.","spans":[]},{"type":"list-item","text":"The Challenge: Navigating Space and Time. When we say “space”, we really mean location, and “time” refers to the various time zones people live in. Dealing with a distributed team means keeping these forces in mind, since every action you take as the manager of a distributed team will be impacted by these forces. We must artificially introduce supporting structures that provide the same benefits as common-location rituals, such as lunch or clocking in at the same time.","spans":[{"start":0,"end":40,"type":"strong"}]},{"type":"list-item","text":"The Counterbalance: Effective Communication. Effective communication provides a framework for teams to bind around purpose. It’s like a trellis that enables vines to grow. Communication does not solve the purpose problem; it just provides structure for the team to grow in a unified direction with purpose. It is also the one tool managers have to get their distributed teams to work together more effectively across space and time.","spans":[{"start":0,"end":43,"type":"strong"}]},{"type":"heading2","text":"Dimensions of Communication","spans":[]},{"type":"paragraph","text":"Communication always takes the easiest path. Being cognizant of this will save you a lot of pain as a manager (this is also a reason why many distributed teams fail).","spans":[]},{"type":"paragraph","text":"Consider how having a team mostly in office with a few dispersed remote team members affects communication. It might be easier for people in-office to walk over to someone—or yell across the table—to discuss something versus typing away in Slack. Or if it’s difficult to set up a conference call, the group of people may choose to run a meeting without waiting for the remotes to log in. At DigitalOcean, every conference room has a Hangout-equipped computer, so it’s never hard to get a meeting going with everyone.","spans":[]},{"type":"paragraph","text":"When running a distributed team, it’s helpful to think about communication in three dimensions:","spans":[]},{"type":"list-item","text":"Latency: How frequently do exchanges occur?","spans":[{"start":0,"end":7,"type":"strong"}]},{"type":"list-item","text":"Throughput: How deep are the exchanges?","spans":[{"start":0,"end":10,"type":"strong"}]},{"type":"list-item","text":"Cost: What’s the overhead of the exchange?","spans":[{"start":0,"end":4,"type":"strong"}]},{"type":"paragraph","text":"With these dimensions, we can begin characterizing the common forms or modalities of communication:","spans":[]},{"type":"image","url":"https://images.prismic.io/www-static/18bc9f67bdf51749452b5b1299e5c5f07f60f4b4_screen-shot-2017-05-12-at-9.36.28-am.png?auto=compress,format","alt":"modalities of communication graphic","copyright":null,"dimensions":{"width":1062,"height":492}},{"type":"paragraph","text":"It’s important to remember that all modalities matter, and choosing one over the other means making explicit tradeoffs. Where possible, seek balance (e.g., choosing only IM as a form of communication but no email might mean deeper thinking is discouraged). All of this framework is important to consider as you manage and nurture a team, since conversations may need to span some or all of them, so choose your modalities wisely.","spans":[]},{"type":"paragraph","text":"Make sure your team is using the right modalities for the communication they need to have. Guide conversations to the correct forums. Set guidelines, like knowing when to start a Hangout. Moderate the amount of face to face to give people time to think and ensure everyone who needs to be a part of the conversation is present. If you don’t, communication will be slower and far less efficient, taking you off the rails.","spans":[]},{"type":"heading2","text":"How to Keep Distributed Teams Moving","spans":[]},{"type":"paragraph","text":"There are (at least) three things you need to be doing in a distributed to keep things moving:","spans":[]},{"type":"list-item","text":"Stay aligned with what’s happening","spans":[]},{"type":"list-item","text":"Stay in touch with people","spans":[]},{"type":"list-item","text":"Keep an eye on the horizon","spans":[]},{"type":"paragraph","text":"Stay aligned. Alignment means that you and your team(s) have a shared contextual view of the (business/operating) world that allows them to function despite being out of time sync or in different locations. Staying aligned means paying attention to the following: progress (what, why, blockers), priorities (what’s most important), and people (offering praise and feedback, and addressing challenges).","spans":[{"start":0,"end":12,"type":"strong"}]},{"type":"paragraph","text":"Stay in touch. Where alignment ensures the team can operate without physical presence, staying in touch ensures the team can remain human in the face of space/time differences. Staying in touch includes scheduling regular 1:1s to connect with a person individually, schedule leadership syncs to connect with the leaders in your business, and holding office hours.","spans":[{"start":0,"end":13,"type":"strong"}]},{"type":"paragraph","text":"Keep an eye on the horizon. If you’re paying attention to the people and the work they do, the final major step is making sure you’re not overwhelmed by the details. You can’t lose track of the purpose when the details threaten to overwhelm you. A few simple things you can do to that end are asking when a task will be done (to drive urgency and encourage an environment open to discussing problems) and carving out time for thinking and planning.","spans":[{"start":0,"end":26,"type":"strong"}]},{"type":"paragraph","text":"In conclusion, remember the forces at work against all distributed teams—space and time—and focus on creative effective communication practices to counteract those forces. Staying aligned and in touch will help you foster communication, and keeping an eye on the horizon will get you in the habit of talking through problems and setting aside time to think ahead.","spans":[]},{"type":"paragraph","text":"Dave “Dizzy” Smith is a senior director of engineering at DigitalOcean. A software industry veteran with over 21 years of experience, he has a broad range of experience across real-time messaging systems, identity federation and authentication, and low-latency peer-to-peer data stores and has been an active contributor to many open source projects. Follow him on Twitter at @dizzyd.","spans":[{"start":0,"end":384,"type":"em"},{"start":376,"end":383,"type":"hyperlink","data":{"link_type":"Web","url":"http://www.twitter.com/dizzyd"}}]}],"blog_post_date":"2017-08-22","tags":[{"tag1":{"tag":"Engineering","_linkType":"Link.document","_meta":{"uid":"engineering"}}}],"_meta":{"uid":"how-to-manage-build-and-nurture-distributed-teams"}}},{"node":{"author":{"_linkType":"Link.document","author_name":"Alejandro (Alex) Jaimes","author_image":{"dimensions":{"width":1600,"height":1067},"alt":"Alejandro (Alex) Jaimes","copyright":null,"url":"https://images.prismic.io/www-static/2ef201f3af242c6dface5c525637e5d45cac7050_ldv_20150519_0485.jpg?auto=compress,format"},"_meta":{"uid":"alejandro_alex_jaimes"}},"blog_header_image":{"dimensions":{"width":785,"height":418},"alt":"AI letter illustration","copyright":null,"url":"https://images.prismic.io/www-static/4192a269dfed282ea3298d24ab19081824e1277d_ai_blog_kasia.png?auto=compress,format"},"blog_headline":[{"type":"heading1","text":"The State of AI","spans":[]}],"blog_post_content":[{"type":"paragraph","text":"This post is the first in a three-part series we're publishing this year on artificial intelligence, written by DigitalOcean’s Head of R&D, Alejandro (Alex) Jaimes.","spans":[{"start":0,"end":164,"type":"em"}]},{"type":"paragraph","text":"In recent months, the amount of media coverage on AI has increased so significantly that a day doesn’t go by without news about it. Whether it’s an acquisition, a funding round, a new application, a technical innovation, or an opinion piece on ethical and philosophical issues (“AI will replace humans, take over the world, eat software, eat the world”), the content just keeps coming.","spans":[]},{"type":"paragraph","text":"The field is progressing at amazing speeds and there’s a lot of experimentation. But with so much noise, it’s hard to distinguish hype from reality, and while everyone seems to be rushing into AI in one way or another, it’s fair to say there is a good amount of confusion on what AI really is, what sort of value it can bring and where things will go next.","spans":[]},{"type":"paragraph","text":"While the reality is that AI has the potential to impact just about everything and be embedded in just about anything—just like software already is—getting started can be daunting, depending on who you ask.","spans":[]},{"type":"paragraph","text":"In this post, I will first explain why computing is now AI. Then, in future posts, I’ll describe the most significant trends, outline steps to be taken in actually implementing AI in practice, and say a few words about the future.","spans":[]},{"type":"heading2","text":"Computing Is Now AI","spans":[]},{"type":"paragraph","text":"AI is already embedded, in some form, in most of the computing services we use on a daily basis: when we search the web, visit a webpage, read our email, use social media, use our phone, etc. Most of those applications use some form of machine learning to perform “basic” tasks, like spam detection, personalization, and advertising. But like computing itself, penetration of AI doesn’t stop there. Our transportation systems, security, cargo shipping, banking, dating, and just about everything else is likely “touched” by algorithms that use machine learning.","spans":[]},{"type":"image","url":"https://images.prismic.io/www-static/c7357ac1b776146a21e76fd1c46a7091c7e75b9a_ai-umbrella-1.png?auto=compress,format","alt":"Ai umbrella","copyright":null,"dimensions":{"width":282,"height":264}},{"type":"preformatted","text":"AI is really an umbrella term that encompasses many subfields. For the sake of simplicity, most of what people think of as AI currently has machine learning, and/or deep learning. The ideas behind the three concepts are rather straightforward: AI aims to \"emulate or supercede human intelligence,\" machine learning is concerned with algorithms that learn models from data, and deep learning is \"simply\" a subset of machine learning algorithms that learn from data with less human intervention. In building \"traditional\" machine learning algorithms an engineer has to design features, but in a deep learning framework the features themselves are learned by the algorithm—those algorithms, however, need significantly greater amounts of data.","spans":[]},{"type":"paragraph","text":"Some industries use computing technology in more advanced ways than others. Tech companies, in particular, have taken the lead in developing products and services around data and AI (in various forms), and scaling to millions and billions of users. This has led to significant advances in some areas where having large, diverse datasets can improve performance to the point where problems that seemed out of reach now seem solvable. Other industries, such as healthcare and education, have been slower to adapt, but we're beginning to see significant progress with very promising prospects.","spans":[]},{"type":"paragraph","text":"If we look closely at trends, and technical requirements (for AI to deliver in products and services), it's easy to see that AI can already be applied everywhere. More specifically, where repetitive patterns occur, and those patterns can be recorded, whether the data is individual or aggregated. One could easily argue that everything in life—and business—consists of cycles, and what's changed significantly in recent years is our ability to record, store, and process behavioral patterns at every level. AI adds prediction, which is extremely valuable.","spans":[]},{"type":"paragraph","text":"The power of AI comes at multiple granularities. There are a plethora of decisions made every day based on simple, repetitive patterns—and those apply to businesses as much as they do to individuals. It's no surprise then, that most companies are using AI today to cut costs and improve efficiency. As more processes become digital, AI, then, becomes not just a critical part of the ecosystem, but the driving force, in large part because its main benefit is efficiency. And if we look at things from this perspective, it's easy to see why computing and AI are already converging to the point where there's no distinction. It the very near future, it will be assumed that AI is part of computing, just as networking and other technical components are.","spans":[]},{"type":"paragraph","text":"This is not a minor shift, however. It is massive because it emphasizes processes that leverage data, and evolving models (vs. \"fixed\" algorithms), impacting how software is developed. This has several ripple effects that I'll describe in future posts, including pushing the hardware boundaries. I would argue that the companies and teams that understand this and think and operate with this mindset will now have a significant advantage over others that try to \"add\" AI at a later stage.","spans":[]},{"type":"paragraph","text":"On one hand, this means that individuals and teams must constantly learn and grow, remain up to date, and rely on the larger community for the exchange of models, ideas, code, and knowledge. It also means that applications will be increasingly built by layering components and data—nothing will be built from scratch. For hobbyists, \"professional\" developers, engineering teams, the open source community, and companies, this translates into significant synergies—an ecosystem that relies on the cloud, which is the perfect platform to combine multiple resources and scale with a single click. Ultimately, this implies that AI skills will be as critical to individuals as they are to companies, and they will form the basis of economic progress for decades to come.","spans":[]},{"type":"paragraph","text":"We'd love to get your thoughts on AI. How it has impacted the way you build software? What do you think you need to make AI part of your workflow? What opportunities and barriers do you see? What are the topics you'd like to learn more about or the tools you'd like to use? Let us know in the comments below!","spans":[]},{"type":"paragraph","text":"*Alejandro (Alex) Jaimes is Head of R&D at DigitalOcean. Alex enjoys scuba diving and started coding in Assembly when he was 12. In spite of his fear of heights, he's climbed a peak or two, gone paragliding, and ridden a bull in a rodeo. He's been a startup CTO and advisor, and has held leadership positions at Yahoo, Telefonica, IDIAP, FujiXerox, and IBM TJ Watson, among others. He holds a Ph.D. from Columbia University.","spans":[]},{"type":"paragraph","text":"Learn more by visiting his personal website or LinkedIn profile. Find him on Twitter: @tinybigdata.*","spans":[{"start":27,"end":43,"type":"hyperlink","data":{"link_type":"Web","url":"http://www.alexjaimes.com"}},{"start":47,"end":63,"type":"hyperlink","data":{"link_type":"Web","url":"https://www.linkedin.com/in/alexjaimes/"}},{"start":86,"end":98,"type":"hyperlink","data":{"link_type":"Web","url":"http://www.twitter.com/tinybigdata"}}]}],"blog_post_date":"2017-05-31","tags":[{"tag1":{"tag":"Engineering","_linkType":"Link.document","_meta":{"uid":"engineering"}}}],"_meta":{"uid":"the-state-of-ai"}}}]}}}