{"id":6760,"date":"2019-11-22T08:34:00","date_gmt":"2019-11-22T08:34:00","guid":{"rendered":"https:\/\/gtechbooster.com\/?p=6760"},"modified":"2022-11-30T22:01:19","modified_gmt":"2022-11-30T22:01:19","slug":"tiledb-promises-faster-discoveries-for-data-scientists","status":"publish","type":"post","link":"https:\/\/gtechbooster.com\/tiledb-promises-faster-discoveries-for-data-scientists\/","title":{"rendered":"Faster discoveries for Data Scientists"},"content":{"rendered":"\n

A new database designed to help data science teams make faster \ndiscoveries by giving them a more powerful way to store, update, \nanalyze, and share large sets of diverse data has been released.<\/p>\n\n\n\n

\"\"<\/a><\/div><\/div>

TileDB consists of a new multi-dimensional array data format, a fast, embeddable, open-source C++ storage engine with data science tooling integrations, and a cloud service for easy data management and serverless computations.<\/p>\n\n\n\n

The developers say traditional databases aren’t ideal for data \nscience use as they’re not cloud-optimized, while cloud object stores \nsuffer from object immutability, eventual consistency, and IO request \nlimiting. A second problem is that some formats lack sufficient support \nfor efficient data updates. They give the example of updating a Parquet \nfile requiring the creation of a new file, pushing the entire update \nlogic to the user\u2019s higher-level application, and say similar problems \narise whenever the update logic is not built into the format and storage\n engine, but it is rather delegated to higher-level applications.<\/p>\n\n\n\n

Finally, the developers cite limited scope as a problem, on the basis\n that most data science applications require at least two separate file \nformats to handle both array data and dataframes; multi-dimensional \narrays for uses such as linear algebra; and dataframes for OLAP \noperations.<\/p>\n\n\n\n

The team started with the storage layer when creating TileDB, and say\n it has the only format and storage engine that handles both dense and \nsparse multi-dimensional arrays. It supports efficient array IO on \nmultiple storage backends, including cloud object stores like AWS S3. It\n also offers rapid, highly parallel, lock-free, batch updates that are \ndesigned to work particularly well on the cloud with immutable objects. \nAll update logic and functionality (like time traveling) is built into \nthe format and storage engine.<\/p>\n\n\n\n

TileDB offers a standalone, embeddable C++ library that ships with \nAPIs in C, C++, Python, R, Java and Go, and has direct access to TileDB \narrays. The library is integrated with Spark, Dask, PrestoDB, MariaDB, \nArrow and geospatial libraries like PDAL, GDAL and Rasterio. TileDB \npushes down as much computation as possible to storage, such as filter \nconditions from the SQL engines and  dataframe computations from Dask \nand Spark.<\/p>\n\n\n\n

Alongside the database is TileDB Cloud, a pay as you use priced service that you can use to share TileDB arrays on the cloud with other users and perform serverless computations on them. Both TileDB and TileDB Cloud are available to try now.<\/p>\n\n\n\n

More Information<\/h2>\n\n\n\n