Solving the problem of multiple writes to the database

This is what sessions are for, maintaining volatile state that may or may not be written to a more permanent storage.

If your application stack does not have a session implementation that 'Just Works!'(tm) for you, you will have to implement one yourself.

Generally it will look something like a cross between your solutions 1 and 3, you create an opaque session key which is stored as a cookie and is used to retrieve the data from the session store. What you use to store session data is going to vary depending on how valuable the data and on how tragic failure is.

Avoid hidden fields in the html, they are a pain to maintain and will cause you problems because the temptation is to trust that their contents are what you put into them... it also means that you are repeatedly processing the same data. In sum; hidden form values make baby jesus cry.

Thanks. It seemed to me in the beginning that HIDDEN fields were a very clumsy solution. Now I understand why they indeed are.

We use PHP as our server side scripting language. I was thinking of using the $_SESSION variable but attributes and pointer information for 30 image uploads per user * no of concurrent users seems like a huge drag on server memory.

I'll probably just write the pointer information to the database for now after each upload. It might make the web app slower by a few milliseconds but it reduces unnecessary complications

PHP uses files for storing sessions; you do want to restrict where on your site you are using sessions as they do have side-effects. Most notably any page served in a path configured for sessions will have an expiry time in the past and a bunch of no-cache headers; for a dynamic page that is the behaviour you want, make sure it's not being applied where it should not be.

this is what i did:

I have a form that: user can upload as many as s/he wants user clicks submit the browser then sends a stream of data one time

on the server: regex find .multipart/form-data;.boundary=.* from CONTENT_TYPE to make sure it's the right stream save the stream into temp-file, read the temp-file

split file-stream based on .filename=\"(.)\".* into chunks afterward, "calibrate" the chunk (determine start & end write positions) then loop and write the chunks into specific filenames provided in the stream

so there are 1 upload stream and 1 load n+1 write operations

I wonder if it helps since I don't use php (hint: newlisp)

Thanks for the suggestion. So I would basically write the pointer information (that would have been written to the database) to a temporary file instead? Do you think this improves performance significantly?

Why not write to the database and just not COMMIT until ll the images have arrived?

Not sure what you meant. Could you elaborate? The form basically accepts input until a user reaches the 30 image limit. A user can then hit the confirm button if he/she is satisfied and wants to commit all the changes.

Also, I didn't want to write to the database each time someone uploads an image. I just wanted to queue all these writes and do it at once in the end when a user commits to make all the changes.

That would save some processing time right? Thanks.

Would it save processing time? Why do you think it would? How do you plan to manage the queue - by writing your own "database"? If you queue them somewhere, you'll end up having to do two writes, one to your queue and one eventually to the database. So that's adding more processing.

It's a serious question. Have you benchmarked this? Writing BLOB data to any reasonable database (or storing it as BFILE if you must) is imperceptibly slower than a filesystem write. In the case of MySQL the database is just syntactic sugar sprinkled on the filesystem. In the case of Oracle on raw volumes it's going to be faster to write to than a cooked filesystem. If you're worried about huge transactions, just COMMIT each one and add a column for approved_yn that you update in a single transaction when the user is happy.

Sorry if I am not being clear.

I am not uploading each image as a BLOB. Rather, I am storing a bunch of strings in the database that will later help me point to the actual image (which is first stored in the local webserver but is pushed into Amazon S3 after the user has finished with the form).

I want to avoid these multiple writes to the database (which stores various pointers to the image) and just do one large write operation at the end. Does this not matter? Would you know any benchmarking tools that will help me measure this?

Thanks

It doesn't matter. Just trust the database :-)

Great. Thanks

That would save some processing time right?

If premature optimization is the root of all evil, I'm afraid you are halfway to hell.

This is not a good reason to make your app more complicated than it needs to be. For one thing, it isn't even easy to tell if your statement is true, let alone how true it is. (i.e. it may be true that you save a few milliseconds, but is that even going to be significant?)

If there is a UI reason why you have to have the user specify 25 uploads on one HTML page and then do them all at once, that's one thing. (I'd use the big-pile-of-hidden-HTML-fields approach; that has the fewest moving parts, it's simple to conceptualize, and I don't think the other methods will make a big difference from the user's perspective.) But don't try to work around your DB until you absolutely have to. Use it. That's what it's there for.

That's helpful. Thanks. Maybe I am overthinking too much about optimization. It doesn't seem like updating a bunch of columns with text will result in significant time savings.