The allocated storage is append only. For updates, just allocate another blob. The deleted blobs would be garbage collected later. So it is not really mmap.
> Also what is the difference between a file, an object, a blob, a filesystem and an object store?
The answer would be too long to fit here. Maybe chatgpt can help. :)
I, too, am interested in your views on the last 2 questions, since your views, not chatGPT's, are what informed the design. Part of learning from others' designs [0] is understanding what the designers think about their own design, and how they came about it.
Would you mind elaborating on them? HN gives a lot of space, and I'm confident you can find a way to summarize without running out, or sounding dismissive (which is what the response kind of sounds like now).
The blob storage is what SeaweedFS built on. All blob access has O(1) network and disk operation.
Files and S3 are higher layers above the blob storage. They require metadata to manage to the blobs, and other metadata for directories, S3 access, etc.
These metadata usually sit together with the disks containing the files. But in highly scalable systems, the metadata has dedicated stores, e.g., Google's Colossus, Facebook's Techtonics, etc. SeaweedFS file system layer is built as a web application of managing the metadata of blobs.
Actually SeaweedFS file system implementation is just one way to manage the metadata. There are other possible variations, depending on requirements.
There are a couple of slides on the SeaweedFS github README page. You may get more details there.
Thank you, that was very informative. I appreciate your succinct, information dense writing style, and appreciate it in the documentation, too, after reviewing that.
what makes it different is a new way of programming for the cloud era.
but you aren't even explaining how anything is different from what a normal file system can do, let alone what makes it a "new way of programming for the cloud era".
Sorry, everybody has different background of knowledge. Hard to understand where the question comes from.
They were straightforward questions. The paper you linked talks about blobs as a term for appending to files. Mostly it seems to be about wrapping and replicating XFS.
Is that why you are avoiding talking about specifics? Are you wrapping XFS?
I'm little confused why people are being so weird with the OP, asking what the difference between a blob and a file aren't something for seaweedfs lol, Blobs and Files, and other terms are terms used to describe different layers of data allocation in almost every modern object storage solution.
Blobs are what lie under files, you can have a file split into multiple blobs spread across different drives, or different servers etc, and then can put it back together into a file when requested, thats how i understand it at a basic level
I think they are being weird. According to the facebook pdf they linked I think that would fall under chunks, but either way, why would someone advertise a filesystem for 'blob' storage when users don't interact with that? According to the paper 'blobs' are sent to append to files, but that isn't really 'blob storage', it's just giving a different name for an operation that's been done since the 70s - appending to a networked file. No one would say 'this filesystem can store all your file appends' and no one is storing discreet serialized data without a name and once you do that, you have a file.
They also seem like they are being vague and patronizing to avoid admitting that their product is not a unique filesystem, but just something to distribute XFS.
> Why does a user need that? Filesystems already break up files into blocks / sectors. Why wouldn't a user just deal with files and let the filesystem handle it?
A blob has its own storage, which can be replicated to other hosts in case current host is not available. It can scale up independently of the file metadata.
Why does a user need that? Filesystems already break up files into blocks / sectors. Why wouldn't a user just deal with files and let the filesystem handle it?
I really don't understand why you aren't eager to explain the differences and what problems are being solved.
just fallocate some space to write to, and a file_id is returned. Use the file_id similar to a pointer to a memory block.
How is that not mmap?
Also what is the difference between a file, an object, a blob, a filesystem and an object store? Is all this just files indexed with sql?