Remote Blog Storage (RBS) Best Practices in SharePoint 2010 - EPC Group

Editor's Notes

1. BLOBs and relational database data are very different entities. Relational data usually consists of text or numbers and tends to be small. In contrast, BLOB data is most often pictures in .jpg, .tiff, or .bmp format—such as product images on a Web site—which can be quite large.
2. There are three (3) approaches to storing unstructured data with SQL Server, RBS, SQL BLOB, and FILESTREAM.Remote BLOB Storage (RBS), SharePoint relies on a new layer in SQL Server to read or update BLOB data stored outside of the database on separate BLOB Stores (File System or dedicated BLOB Stored)SQL BLOB traditional BLOB storage with SharePoint, BLOB data is stored along side the structured metadata in the Content DatabaseFILESTREAM Storage: Improving SQL BLOBGeneral guidance can be summarized as:Storing records that are on average smaller than 256KB is optimized with traditional SQL BLOBStoring large BLOB data like large video files will benefit from FILESTREAM - FILESTREAM will provide better streaming performance on these filesStoring the TIFF images or large static files on remote BLOBs, you should consider to leveraging the new Remote BLOB API layer, i.e. RBS
3. varbinary is the binary data type designation for binary large objects stored in SharePoint 2010 Content Databases and refers to variable-length binary data. (MAX) refers to a value that max indicates that the maximum storage size is 2^31-1 bytes or otherwise 2GB.
4. RBS is a SQL Server 2008 add-on that uses auxiliary tables, stored procedures and a managed client library to provide its services. A reference to the blob (provided by the Blob Store) is stored in RBS auxiliary tables and an RBS Blob ID is generated.ISVs create RBS Provider Libraries to enable custom BLOB stores using the RBS API set. The externalization of binary large objects in SharePoint 2010 is often referred to as RBS.
5. FILESTREAM storage is implemented as a *varbinary(max) column in which the data is stored as BLOBs in the file system.Transact-SQL statements can insert, update, query, search, and back up FILESTREAM data. Win32 file system interfaces provide streaming access to the data.A FILESTREAM filegroup is a special filegroup that contains file system directories instead of the files themselves. These file system directories are called data containers. Data containers are the interface between Database Engine storage and file system storage.FILESTREAM uses the NT system cache for caching file data. This helps reduce any effect that FILESTREAM data might have on Database Engine performance. The SQL Server buffer pool is not used; therefore, this memory is available for query processing.Performance: The way the data is going to be used is a critical factor. If streaming access is needed, storing the data inside a SQL Server database may be slower than storing it externally in a location such as the NTFS file system. Using file system storage, the data is read from the file and passed to the client application (either directly or with additional buffering). When the BLOB is stored in a SQL Server database, the data must first be read into SQL Server’s memory (the buffer pool) and then passed back out through a client connection to the client application. Not only does this mean the data goes through an extra processing step, it also means that SQL Server’s memory is unnecessarily “polluted” with BLOB data, which can cause further performance problems for SQL Server operations.FILESTREAM is most appropriate when the following conditions are true:Objects that are being stored are, on average, larger than 1 MB.Fast read access is important.http://technet.microsoft.com/en-us/library/bb933993.aspxhttp://msdn.microsoft.com/en-us/library/cc949109(SQL.100).aspx*See slide
6. Storage AttributeA FILESTREAM BLOB is a SQL BLOB, varbinary(MAX) Dual Programming ModelAll the existing code you have on BLOBs can work for a filestream BLOB.We leverage the Windows file systems layer to enable the win32 rich streaming programming model with the same transactional semantics as T-SQL. You open can obtain a real win32 handle to the filestream BLOB and call win32 read/write APIs on it.This is main advantage over SQL BLOB. With your large BLOB data stored now on the filesystem, the streaming performance can be as good as the filesystem’s. You can leverage the file system streaming optimizations options that fit your app’s access patterns.Data ConsistencySQL Server is now on the I/O stack of the streaming that you do on the filestream unstructured data Integrated ManageabilityThe filestream BLOB is still a SQL BLOB; FTS, Indexing, Replication, Log shipping etc.. Works for filestream BLOBs.
7. BLOBs smaller than 256 kilobytes SQL BLOB suitable, and BLOBs larger than 1 megabyte (MB) are best stored outside the database. For those sized between 256 KB and 1 MB, the more efficient storage solution depends on the read vs. write ratio of the data, and the rate of “overwrite”. Storing BLOB data solely within the database (e.g., using the varbinary(max) data type) is limited to 2 gigabytes (GB) per BLOB and enforced through SharePoint 2010.
8. Write Once, Read Many (alternatively Write One, Read Multiple or WORM) refers to computer data storage systems, data storage devices, and data storage media that can be written to once, but read from multiple times. DVD/CDRLook through the application tables and find blobs that are no longer referenced by the application.GC time window to define when BLOBs are deleted from the blob store.blobs are "orphans" and can be caused due to aborted transactions, application misbehavior or other failures. Orphan blobs created before the GC time window are deleted from the blob store.Get more info here http://blogs.msdn.com/b/sqlrbs/archive/2008/08/08/rbs-garbage-collection-settings-and-rationale.aspx.
9. Enterprises have 3 options. Each option has advantages & disadvantages based on your scenario.What are the main tradeoffs? When I choose to store my Jpeg images or video files on Windows file system/file server configuration, I get great streaming performance. But my development cost is high because SharePoint needs to have the logic to keep the structured data in sync with my Jpegs/MP3 files stored outside the db.When I choose to store my Jpeg images or video files on a dedicated Blob store, I can scale and expand my storage space easily. The more I expand it the cheaper it gets to add to it. In this case streaming performance depends on your store and keeping structured and unstructured data in sync requires a complex implementation at the app level. When I choose to store my Jpeg images as SQL VARBINAY(MAX) BLOBs in SQL Database, my Jpeg files are constantly in sync with the metadata. I can also use all the SQL Server database management features on my Jpeg files. What’s the tradeoff in this case? SharePoint has to manage the file size limitation and poor streaming performance if my jpegs grow larger (a paper by Jim Grey points out the SQL BLOBs streaming performance is better than file system streaming perf for files smaller than 250KB, while when the unstructured data is around 1MB and larger the File system streaming performs much better than SQL BLOB Streaming).I’ve also seen large enterprises, designing more than one of these options in the same app and some central module decides in which option to store the new BLOB depending on the storage purpose (how large the BLOB is, how it will be used)Show of handsWho store some or all BLOB data in SQL Server BLOBs (Varbinary(max))?Who store some of the BLOB data on file system?Who store some of the BLOB data on remote dedicated store?
10. The Office Web Applications cache is used by Word and PowerPoint Web Applications to create a version of a document requested for viewing through the browser improving performance and reducing resource consumption on server machines by making cached versions of a document or presentation available in cases where there are multiple requests for the same document.The Office Web Applications cache occurs in two (2) distinct tiers, on the server file system and within a “specialized” site collection hosted on a per Web application basis. Document or presentation requests made through the Office Web Applications are served through both caches as the images are rendered for client consumption. Both cache locations are used by all site collections within a Web application where the Office Web Applications features activated.For example, when a client requests a document, the document is rendered through an AppServerHost.exe process and the document subsequently is cached to the server file system cache located on each server from which the content is propagated to the site collection cache which exists on a per Web application basis. Subsequent requests for the document are rendered from the site collection cache.
11. The maintenance tasks associated with Remote BLOB Storage (RBS) are mainly performed through the RBS Maintainer. The RBS Maintainer performs periodic garbage collection and other maintenance tasks for an RBS deployment.GCGarbage collection is how unreferenced or deleted data is removed from the remote BLOB store. Garbage collection in RBS is performed passively. References to BLOBs are counted by looking at the list of BLOB IDs stored by the application in its RBS table columns at the time of garbage collection.RSLook through the application tables and find blobs that are no longer referenced by the application. The list of registered RBS columns is used for this purpose. BlobIds must not be stored in any place other than the registered columns. The blobs that are no longer referenced by the application are marked to be deleted.DPBlobs marked for deletion are actually deleted from the blob store. There is a gap between when the blobs get marked for deletion and when they are actually deleted. This gap duration can be configured using the "garbage_collection_time_window" config item and defaults to 30 days.