{"id":6901,"date":"2019-12-18T09:42:00","date_gmt":"2019-12-18T09:42:00","guid":{"rendered":"https:\/\/gtechbooster.com\/?p=6901"},"modified":"2023-04-01T01:36:47","modified_gmt":"2023-04-01T01:36:47","slug":"netflix-opens-metaflow","status":"publish","type":"post","link":"https:\/\/gtechbooster.com\/netflix-opens-metaflow\/","title":{"rendered":"Netflix opens Metaflow"},"content":{"rendered":"\n<p>Netflix has open sourced Metaflow, which the developers describe as a\n human-centric framework for data science. The framework has been used \ninternally at Netflix for the last two years to build and manage \nhundreds of data-science projects.<\/p>\n\n\n\n<div class=\"gtech-migrated-from-ad-inserter-placement-2 gtech-entity-placement\" style=\"text-align: center;\" id=\"gtech-2600282662\"><div id=\"gtech-695459579\" style=\"margin-right: auto;margin-left: auto;text-align: center;\"><a data-bid=\"1\" data-no-instant=\"1\" href=\"https:\/\/gtechbooster.com\/linkout\/17207\" rel=\"noopener\" class=\"notrack\" aria-label=\"26001\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/gtechbooster.com\/media\/2023\/01\/26001.jpeg\" alt=\"\"  srcset=\"https:\/\/gtechbooster.com\/media\/2023\/01\/26001.jpeg 1024w, https:\/\/gtechbooster.com\/media\/2023\/01\/26001-768x960.jpeg 768w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" width=\"500\" height=\"625\"  style=\"display: inline-block;\" \/><\/a><\/div><\/div><p>Metaflow is a Python library that provides a unified API to the \ninfrastructure stack that is required to execute data science projects, \nfrom prototype to production. It lets data scientists structure their \nworkflow as a Directed Acyclic Graph of steps.<\/p>\n\n\n\n<p>Netflix developed Metaflow to boost the productivity of data  scientists working on a wide variety of projects from classical  statistics to state-of-the-art deep learning. The aim was to make  problems in model prototypes visible earlier in the process. In February  Netflix said that Metaflow had cut model deployment times from four  months to seven days.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-embed-youtube wp-block-embed is-type-video is-provider-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"Human-Centric Machine Learning Infrastructure @Netflix\" width=\"1290\" height=\"726\" src=\"https:\/\/www.youtube.com\/embed\/XV5VGddmP24?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe>\n<\/div><\/figure>\n\n\n\n<p>The steps can be arbitrary Python code, with data and models stored \nas normal Python instance variables. They work even if the code is \nexecuted on a distributed compute platform, which Metaflow supports by \ndefault, thanks to its built-in content-addressed artifact store.<\/p>\n\n\n\n<p>The Metaflow team says that using code is important to make the \nframework useful. They say that while many machine learning \ninfrastructures rely on GUIs, configuration, DSLs, or automation, they \ntend to fail to scale when the problems become more complex. Instead, \nMetaflow relies on code. They say:<\/p>\n\n\n\n<p><em>&#8220;We rely on code. General-purpose programming languages are \narguably the most effective tool in building functional systems for \ncomplex use cases. The main downside of code is a steeper learning \ncurve: for the simplest use cases, almost any other modality would \nprovide a simpler solution. To minimize this cost, we rely on easy to \nlearn, popular languages like Python or R.&#8221;<\/em><\/p>\n\n\n\n<p>Metaflow uses existing infrastructure when feasible. In particular, \nit is tightly integrated with Amazon Web Services. You can use Metaflow \nwith a variety of machine learning or data science libraries, such as \nPyTorch, Tensorflow, or SciKit Learn. Metaflow allows you to write your \nmodels and business logic as idiomatic Python code with not much new to \nlearn.<\/p>\n\n\n\n<p>There&#8217;s more information on the Metaflow website and on Github.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">More Information<\/h2>\n\n\n\n<ul class=\"wp-block-list\"><li><a href=\"https:\/\/metaflow.org\/\">Metaflow Website<\/a><\/li><li><a href=\"https:\/\/github.com\/Netflix\/metaflow\">Metaflow On GitHub<\/a><\/li><\/ul>\n<div class=\"gtech-end-cont gtech-entity-placement\" id=\"gtech-1897807926\"><div id=\"gtech-3269051086\" style=\"margin-right: auto;margin-left: auto;text-align: center;\"><a data-bid=\"1\" data-no-instant=\"1\" href=\"https:\/\/gtechbooster.com\/linkout\/17207\" rel=\"noopener\" class=\"notrack\" aria-label=\"26001\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/gtechbooster.com\/media\/2023\/01\/26001.jpeg\" alt=\"\"  srcset=\"https:\/\/gtechbooster.com\/media\/2023\/01\/26001.jpeg 1024w, https:\/\/gtechbooster.com\/media\/2023\/01\/26001-768x960.jpeg 768w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" width=\"500\" height=\"625\"  style=\"display: inline-block;\" \/><\/a><\/div><\/div>","protected":false},"excerpt":{"rendered":"<p>Netflix has open sourced Metaflow, which the developers describe as a human-centric framework for data science. The framework has been used internally at Netflix for the last two years to build and manage hundreds of data-science projects. Metaflow is a Python library that provides a unified API to the infrastructure stack that is required to [&hellip;]<\/p>\n","protected":false},"author":7,"featured_media":6910,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_gspb_post_css":"","footnotes":""},"categories":[1915],"tags":[1132,1131,620,6],"class_list":["post-6901","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ndocs","tag-metaflow","tag-netflix","tag-open-source","tag-programming"],"blocksy_meta":{"styles_descriptor":{"styles":{"desktop":"","tablet":"","mobile":""},"google_fonts":[],"version":6}},"_links":{"self":[{"href":"https:\/\/gtechbooster.com\/api-json\/wp\/v2\/posts\/6901","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/gtechbooster.com\/api-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/gtechbooster.com\/api-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/gtechbooster.com\/api-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/gtechbooster.com\/api-json\/wp\/v2\/comments?post=6901"}],"version-history":[{"count":0,"href":"https:\/\/gtechbooster.com\/api-json\/wp\/v2\/posts\/6901\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/gtechbooster.com\/api-json\/wp\/v2\/media\/6910"}],"wp:attachment":[{"href":"https:\/\/gtechbooster.com\/api-json\/wp\/v2\/media?parent=6901"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/gtechbooster.com\/api-json\/wp\/v2\/categories?post=6901"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/gtechbooster.com\/api-json\/wp\/v2\/tags?post=6901"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}